belijzajac.dev

Proto-Danksharding: Speeding Up Blobs Verification

blog@belijzajac.dev (belijzajac) — Sat, 27 May 2023 00:00:00 +0000

Introduction

The Ethereum Foundation proposed EIP-4844 on February 25, 2022, with the objective of reducing gas fees. It introduces a new transaction type called “blob”, which is temporarily stored and committed using the KZG commitment scheme. In addition, the Ethereum Foundation developed a project called c-kzg-4844, which provides a minimal implementation of the polynomial commitments API written in C. This project does not use parallelization and exposes its C API for bindings in different programming languages. Another project, called go-kzg-4844, which uses parallelism, has been practically implemented into the Ethereum code and is rumored to be the fastest implementation of EIP-4844 thus far.

Next week, I will be defending my thesis titled “Parallelization of the KZG10 scheme”. In my thesis, I parallelized the KZG commitment scheme and BLS12-381 elliptic curve operations, along with a subset of the EIP-4844 proposal that uses these KZG commitments. My changes were incorporated into the rust-kzg project, where we exported C functions through Rust to bind the parallelized functions of rust-kzg backends to those of c-kzg-4844. Fortunately, we were presented with a unique opportunity due to the go binding included in the c-kzg-4844 project. We then used this binding to benchmark our rust-kzg’s highly parallelized blst backend against their go-kzg-4844 project and assess its speed in comparison.

How c-kzg-4844 does things

C-kzg-4844 leaves the implementation of parallelism to higher-level programming languages that use its bindings. This approach is not only simpler but also safer. The focus of c-kzg-4844 is on single-core performance, which is great for a low-latency environment.

How go-kzg-4844 does things

Go-kzg-4844 offers the function VerifyBlobKZGProofBatch, which is designed for single-core execution similar to c-kzg-4844. However, they also provide a parallelized version of this function called VerifyBlobKZGProofBatchPar. This parallelized version uses go-routines to process each proof in parallel. Although not perfect, this parallel implementation is considerably faster than the sequential one.

How we do things in rust-kzg

The general idea behind our approach is as follows: if the number of blobs exceeds the number of physical CPU cores, we divide the blobs into subgroups of equal size. Each CPU core then independently runs the batched algorithm. For example, consider the illustration below. If there are 64 blobs and 4 CPU cores, we create 4 groups, each containing 16 blobs. Each group is assigned to its dedicated CPU core, which handles the execution of the blob verification process. By utilizing this approach, we effectively distribute the workload across multiple CPU cores, optimizing performance and ensuring efficient verification of the blobs.

However, one could argue that the performance of batched blob KZG proof verification depends on how Ethereum protocol execution clients choose to utilize this approach. If clients choose to verify blobs as soon as they receive them, they would likely opt for an approach that performs single blob verification faster. However, if they decide to wait and accumulate a fixed amount of blobs before performing the verification, this approach will yield much better performance.

Code example

In the code snippet, there is more to the implementation, but let’s focus on illustrating the main concept of this approach:

 1#[cfg(feature = "parallel")]
 2{
 3 let num_blobs = blobs.len();
 4 let num_cores = num_cpus::get_physical();
 5
 6 return if num_blobs > num_cores {
 7 // Process blobs in parallel subgroups
 8 let blobs_per_group = num_blobs / num_cores;
 9
10 blobs
11 .par_chunks(blobs_per_group)
12 .enumerate()
13 .all(|(i, blob_group)| {
14 let num_blobs_in_group = blob_group.len();
15 let commitment_group = &commitments_g1
16 [blobs_per_group * i..blobs_per_group * i + num_blobs_in_group];
17 let proof_group =
18 &proofs_g1[blobs_per_group * i..blobs_per_group * i + num_blobs_in_group];
19 let (evaluation_challenges_fr, ys_fr) =
20 compute_challenges_and_evaluate_polynomial(
21 blob_group,
22 commitment_group,
23 ts,
24 );
25
26 verify_kzg_proof_batch(
27 commitment_group,
28 &evaluation_challenges_fr,
29 &ys_fr,
30 proof_group,
31 ts,
32 )
33 })
34 } else {
35 // Each group contains either one or zero blobs, so iterate
36 // over the single blob verification function in parallel
37 (blobs, commitments_g1, proofs_g1)
38 .into_par_iter()
39 .all(|(blob, commitment, proof)| {
40 verify_blob_kzg_proof(blob, commitment, proof, ts)
41 })
42 };
43}

When num_blobs > num_cores, the code divides the blobs into parallel subgroups. The number of blobs per group is calculated based on the division. The code then iterates over each subgroup, performing various operations such as retrieving the corresponding commitment and proof groups. It also computes evaluation challenges and evaluates a polynomial using the provided data. Finally, it verifies a batch of KZG proofs using the obtained information.

In the else statement, when the number of blobs is not greater than the number of cores, the code handles each blob individually or in groups with only one blob. It uses parallel iteration to execute the blob verification function concurrently, similar to how go-kzg-4844 handles parallelism using go-routines.

Results

Rust and Go bindings, using the rust-kzg with blst backend, verified 64 blobs on 16 cores in 29.82 ms and 30.164 ms, respectively. In comparison, the native rust-kzg accomplished this task in 18.397 ms, while the parallelized implementation of go-kzg-4844 took 48.037 ms. It’s important to note that we only perform full error checking through the exported C API when we convert bytes to our internal types. Therefore, the performance of the native rust-kzg code is probably better because we omit those checks here, assuming we receive correct data from the byte conversion functions. With this in mind, the rust-kzg with blst backend outperformed go-kzg-4844 by approximately 161.11% in terms of speed, while its bindings were approximately 59.25% faster.

Summary

We potentially outperform go-kzg-4844 by approximately 59.25% within Go in batched blob KZG proof verification

WisniaLang: Compiler Project

blog@belijzajac.dev (belijzajac) — Mon, 17 Oct 2022 00:00:00 +0000

Introduction

For the past 3 years, I have been working on the WisniaLang compiler for my own programming language that compiles to native machine code and packs it into an executable by itself. Unlike many others, I rolled out my own compiler backend from scratch that does fast but naive code generation. While it’s admittedly a more old-fashioned approach to compiler engineering, it’s the path I chose to take when developing my compiler.

Architecture

My compiler’s architecture is divided into several main phases that work together to complete this translation. These phases include lexical analysis, which breaks the source code down into smaller pieces called tokens; syntactic analysis, which builds a representation of the structure of the source code called an abstract syntax tree (AST); semantic analysis, which checks the AST for semantic errors while traversing the tree; intermediate representation (IR), which represents the code in a lower-level form close to the target architecture; code generation, which allocates registers and generates machine code from the said IRs; and, lastly, packing the resulting machine code into an executable program in ELF format.

Programming languages and LLVM

Before going further, let me get straight to the point:

Writing compilers is easy
Optimizing the machine code is hard
Supporting arbitrary architectures / operating systems is hard

This is where LLVM comes in handy. LLVM uses an intermediate representation language, which is kind of similar to assembly, but with a few higher level constructs. LLVM is good at optimizing this IR language, as well as compiling into different architecture and binary formats. So as a language author using LLVM, I’m really writing a transpiler from my language to LLVM IR, and letting the LLVM compiler do the hard work.

You talk about LLVM so much, why’s that? Let me begin with this illustration:

I’m not sure if it’s a positive thing, but the LLVM project has achieved such widespread adoption that it’s almost reached a monopoly status, much like the Chromium project, for instance. Apart from Google Chrome, numerous other browsers are built upon the Chromium codebase. From Electron web apps to Arc, Microsoft Edge, Opera, Vivaldi, Brave, and beyond, the list just goes on. Firefox and Safari are perhaps the only web browsers that stand out from this copy-paste crowd.

I just wanted to point out that while 99.9% of compiler developers opt for LLVM, the remaining few explore alternative compiler backends like QBE, develop interpreters (like Python), or create virtual machines (such as the JVM for Java and Kotlin). Some even write transpilers that convert high-level languages into something low-level like C, which is then compiled with gcc. If you recall the dragon compiler book appearing at the top of this page, these and similar compiler books are gradually losing relevance because they don’t teach how to use LLVM, the industry’s compiler standard.

Benchmark No. 1: Fibonacci sequence

To benchmark different compilers, I chose the Fibonacci sequence without recursion problem and computed the 46th Fibonacci number with each compiler under test. This number was chosen because it conveniently fits within 32 bits. Compile-time and runtime benchmarks were performed using the hyperfine command-line benchmarking tool, which closely resembles Rust’s Criterion benchmarking library. Binary size benchmarks were carried out using standard Linux tools like strip to remove debug symbols from binaries and wc to display byte counts for each binary file.

WisniaLang benchmark

 1fn fibonacci(n: int) -> int {
 2 if (n <= 1) {
 3 return n;
 4 }
 5 int prev = 0;
 6 int current = 1;
 7 for (int i = 2; i <= n; i = i + 1) {
 8 int next = prev + current;
 9 prev = current;
10 current = next;
11 }
12 return current;
13}
14
15fn main() {
16 print(fibonacci(46));
17}

Compile time

1hyperfine --runs 1000 --warmup 10 --shell=none './wisnia fibonacci.wsn'
2Benchmark 1: ./wisnia fibonacci.wsn
3 Time (mean ± σ): 1.6 ms ± 0.3 ms [User: 0.8 ms, System: 0.5 ms]
4 Range (min … max): 1.3 ms … 8.7 ms 1000 runs

Runtime

1hyperfine --runs 1000 --warmup 10 --shell=none './a.out'
2Benchmark 1: ./a.out
3 Time (mean ± σ): 109.6 µs ± 36.8 µs [User: 58.2 µs, System: 4.7 µs]
4 Range (min … max): 84.0 µs … 736.3 µs 1000 runs

Binary size

1wc -c a.out
2421 a.out

C++ (gcc) benchmark

 1#include <iostream>
 2
 3constexpr auto fibonacci(u_int32_t n) {
 4 if (n <= 1) {
 5 return n;
 6 }
 7 u_int32_t prev = 0, current = 1;
 8 for (size_t i = 2; i <= n; i++) {
 9 u_int32_t next = prev + current;
10 prev = current;
11 current = next;
12 }
13 return current;
14}
15
16int main() {
17 std::printf("%d", fibonacci(46));
18}

Compile time

1hyperfine --runs 100 --warmup 10 --shell=none 'g++ -std=c++23 -O3 fibonacci.cpp'
2Benchmark 1: g++ -std=c++23 -O3 fibonacci.cpp
3 Time (mean ± σ): 456.4 ms ± 4.5 ms [User: 415.8 ms, System: 35.2 ms]
4 Range (min … max): 448.9 ms … 472.1 ms 100 runs

Runtime

1hyperfine --runs 1000 --warmup 10 --shell=none './a.out'
2Benchmark 1: ./a.out
3 Time (mean ± σ): 347.1 µs ± 62.8 µs [User: 206.4 µs, System: 67.2 µs]
4 Range (min … max): 271.9 µs … 926.4 µs 1000 runs

Binary size

1strip a.out
2wc -c a.out
314472 a.out

C++ (clang) benchmark

Same program as before, just different compiler.

Compile time

1hyperfine --runs 100 --warmup 10 --shell=none 'clang++ -std=c++2b -O3 fibonacci.cpp'
2Benchmark 1: clang++ -std=c++2b -O3 fibonacci.cpp
3 Time (mean ± σ): 538.2 ms ± 16.9 ms [User: 481.7 ms, System: 45.7 ms]
4 Range (min … max): 524.3 ms … 657.9 ms 100 runs

Runtime

1hyperfine --runs 1000 --warmup 10 --shell=none './a.out'
2Benchmark 1: ./a.out
3 Time (mean ± σ): 351.4 µs ± 67.7 µs [User: 203.2 µs, System: 72.2 µs]
4 Range (min … max): 267.1 µs … 984.8 µs 1000 runs

Binary size

1strip a.out
2wc -c a.out
314504 a.out

Rust benchmark

 1fn fibonacci(n: u32) -> u32 {
 2 if n <= 1 {
 3 return n;
 4 }
 5 let (mut prev, mut current) = (0, 1);
 6 for _ in 2..=n {
 7 let next = prev + current;
 8 prev = current;
 9 current = next;
10 }
11 current
12}
13
14fn main() {
15 println!("{}", fibonacci(46));
16}

Compile time

1hyperfine --runs 100 --warmup 10 --shell=none 'rustc -C opt-level=3 fibonacci.rs'
2Benchmark 1: rustc -C opt-level=3 fibonacci.rs
3 Time (mean ± σ): 173.4 ms ± 3.0 ms [User: 130.5 ms, System: 51.2 ms]
4 Range (min … max): 168.6 ms … 183.8 ms 100 runs

Runtime

1hyperfine --runs 1000 --warmup 10 --shell=none './fibonacci'
2Benchmark 1: ./fibonacci
3 Time (mean ± σ): 490.4 µs ± 82.8 µs [User: 264.9 µs, System: 129.3 µs]
4 Range (min … max): 375.1 µs … 1092.6 µs 1000 runs

Binary size

1strip fibonacci
2wc -c fibonacci
3321920 fibonacci

Benchmark No. 2: 29'988 lines of code

I wrote a Python script that generates program code for WisniaLang, C++, and Rust. It generates similar calls to a function named calculate_1997, such as calculate_1, calculate_2, and calculate_1999, for over 2000 times:

 1...
 2void calculate_1997() {
 3 int i = 0;
 4 int a = 0;
 5 int b = 0;
 6 while (b < 1997) {
 7 a = a + b + i;
 8 b = a - b - i;
 9 int c = a + b;
10 int d = a + b + c;
11 int e = a + b + c + d;
12 int f = a + b + c + d + e;
13 i = f - e - d - c + 1;
14 }
15}
16...
17int main() {
18 ...
19 calculate_1997();
20 ...
21}

You can run this script with python main.py --wisnia --cpp --rust 2000.

WisniaLang benchmark

The program can be found at post-data/calculate.wsn.

Compile time

1hyperfine --runs 20 --warmup 1 --shell=none './wisnia calculate.wsn'
2Benchmark 1: ./wisnia calculate.wsn
3 Time (mean ± σ): 2.367 s ± 0.054 s [User: 2.328 s, System: 0.036 s]
4 Range (min … max): 2.282 s … 2.466 s 20 runs

Binary size

1wc -c a.out
2336025 a.out

C++ (gcc) benchmark

The program can be found at post-data/calculate.cpp.

Compile time

1hyperfine --runs 20 --warmup 1 --shell=none 'g++ -std=c++23 -O3 calculate.cpp'
2Benchmark 1: g++ -std=c++23 -O3 calculate.cpp
3 Time (mean ± σ): 2.177 s ± 0.009 s [User: 2.110 s, System: 0.064 s]
4 Range (min … max): 2.156 s … 2.193 s 20 runs

Binary size

1strip a.out
2wc -c a.out
396304 a.out

C++ (clang) benchmark

Same program as before, just different compiler.

Compile time

1hyperfine --runs 20 --warmup 1 --shell=none 'clang++ -std=c++2b -O3 calculate.cpp'
2Benchmark 1: clang++ -std=c++2b -O3 calculate.cpp
3 Time (mean ± σ): 2.179 s ± 0.025 s [User: 2.125 s, System: 0.048 s]
4 Range (min … max): 2.156 s … 2.252 s 20 runs

Binary size

1strip a.out
2wc -c a.out
396336 a.out

Rust benchmark

The program can be found at post-data/calculate.rs.

Compile time

1hyperfine --runs 20 --warmup 1 --shell=none 'rustc -C opt-level=3 calculate.rs'
2Benchmark 1: rustc -C opt-level=3 calculate.rs
3 Time (mean ± σ): 2.353 s ± 0.027 s [User: 2.268 s, System: 0.095 s]
4 Range (min … max): 2.324 s … 2.436 s 20 runs

Binary size

1strip calculate
2wc -c calculate
3317824 calculate

Results

Combining mean compile time, runtime, and binary sizes from benchmark results, we obtain the following graphs.

The runtime range for WisniaLang was from 84.0 µs to 736.3 µs over 1000 program runs, indicating ambiguous results due to benchmarking a 17-line program that executes 3 lines of code 45 times. However, this does demonstrate the speed at which we can compile small programs. In the future, I plan to report on the recursive Fibonacci sequence.

WisniaLang generates code as fast as established compilers, but this may be because it doesn’t perform many static code analysis or optimization steps. This has resulted in my binary being quite large. In contrast, C++ optimizes out redundant code, simplifying the while loop to use at most three variables. This is the while loop in question:

1while (b < 1997) {
2 a = a + b + i;
3 b = a - b - i;
4 int c = a + b;
5 int d = a + b + c;
6 int e = a + b + c + d;
7 int f = a + b + c + d + e;
8 i = f - e - d - c + 1;
9}

What I mean is that C++ likely optimized the code to use only three variables – a, b, and i – by substituting the values of c, d, e, and f directly, thereby reducing redundancy. This is something I’ll fix in the future releases of WisniaLang.

Summary

If compilation speed and binary size are important, dropping the LLVM toolchain can have a positive impact
However, doing so means missing out on LLVM optimizations as well as support for arbitrary OSes and architectures

Outperforming Rayon with OpenMP

blog@belijzajac.dev (belijzajac) — Tue, 16 Nov 2021 00:00:00 +0000

Introduction

For the Blockchain Technologies course, students were paired into groups and assigned to produce the fastest Rust library implementing the KZG10 cryptographic scheme. Two teams used the blst backend, which is implemented in assembly and has direct bindings for Rust and C. The first team, blst-from-scratch, used the Rust bindings provided by the blst library to produce an interface closer to c-kzg. The second team, which I was part of, worked on the ckzg library in C. We were responsible for producing an implementation that could integrate into Rust via the C bindings provided by my team.

Choosing the right tool for the job

It’s a no-brainer for Rust programmers to choose Rayon when it comes to writing parallel code, as there aren’t many other viable and easy-to-use options available. While Rust does offer alternatives like std::thread, which provides access to native OS threads, the manual creation and management of threads can be cumbersome.

When I was working on my C code, I had to decide on the best approach to parallelize it. My options included:

pthread: A POSIX standard for thread creation and management.
A popular third-party threadpool library from GitHub.
OpenMP: Parallel programming library for C and C++ without manual thread management.

I chose OpenMP because, during experimentation, I discovered it yielded the best results and was relatively straightforward to use. However, I encountered a challenge in integrating it with Rust to ensure compatibility across multiple platforms, starting with Linux and possibly macOS. Eventually, I came up with the following Bash script to automate the entire process of building and packaging shared libraries. Fortunately, OpenMP was integrated into Rust by either:

exporting the RUSTFLAGS environment variable pointing to the correct libomp LLVM runtime

 1# Linux
 2apt install libomp-dev
 3export LIBOMP_PATH=$(find /usr/lib/llvm* -name libiomp5.so | head -n 1)
 4
 5# MacOS
 6brew install libomp
 7ln -s /usr/local/opt/libomp/lib/libomp.dylib /usr/local/lib
 8ln -s /usr/local/opt/libomp/include/omp.h /usr/local/include
 9export LIBOMP_PATH=/usr/local/lib/libomp.dylib
10
11# And finally
12export RUSTFLAGS="-C link-arg=$LIBOMP_PATH"

or creating a .cargo/config.toml file inside the project directory and mentioning it there

1[build]
2rustflags = [
3 "-C", "link-arg=LIBOMP_PATH"
4]

Well, that was simple.

Searching for bottlenecks

In order to optimize a program’s performance, CPU profiling tools like Perf play a crucial role by providing detailed insights into where computational resources are being used. One powerful visualization tool generated by these profilers is the flamegraph, which offers a clear representation of a program’s CPU usage over time.

The flamegraph displayed above illustrates the CPU time distribution of the c-kzg library’s fft_g1 benchmark. Upon analysis, it became evident that a significant portion of the execution time was spent in assembly code, highlighting potential areas for optimization. Further investigation on go-kzg revealed that the fft_g1 benchmark was indeed a performance bottleneck and stood out as a prime candidate for parallelization. By parallelizing this specific operation, we can improving the overall performance of the library.

Parallelizing fft_g1

The fft_g1 function calls the fft_g1_fast function, which applies the divide-and-conquer principle to divide a large problem into smaller subproblems, recursively solving each of them. The general procedure here is to distribute work (fft_f1_fasts) among worker threads.

The blst-from-scratch team implemented it as follows:

 1let (lo, hi) = ret.split_at_mut(half);
 2rayon::join(
 3 || fft_g1_fast(lo, data, stride * 2, roots, roots_stride * 2),
 4 || fft_g1_fast(hi, &data[stride..], stride * 2, roots, roots_stride * 2)
 5);
 6
 7for i in 0..half {
 8 let y_times_root = ret[i + half].mul(&roots[i * roots_stride]);
 9 ret[i + half] = ret[i].sub(&y_times_root);
10 ret[i] = ret[i].add_or_dbl(&y_times_root);
11}

As a side note, rayon::join spawns two threads, one executing each of the two closures.

The C equivalent, on the other hand, was as follows:

 1#pragma omp parallel sections
 2{
 3 #pragma omp section
 4 {
 5 fft_g1_fast(out, in, stride * 2, roots, roots_stride * 2, half);
 6 }
 7 #pragma omp section
 8 {
 9 fft_g1_fast(out + half, in + stride, stride * 2, roots, roots_stride * 2, half);
10 }
11}
12#pragma omp parallel
13#pragma omp for
14for (uint64_t i = 0; i < half; i++) {
15 g1_t y_times_root;
16 g1_mul(&y_times_root, &out[i + half], &roots[i * roots_stride]);
17 g1_sub(&out[i + half], &out[i], &y_times_root);
18 g1_add_or_dbl(&out[i], &out[i], &y_times_root);
19}

In addition to parallel sections, I also used OpenMP’s parallel for-loop, because I noticed it yielded a 5% greater performance on my personal machine. Considering the ubuntu-latest runner in GitHub Actions CI had only two available cores, the halves of the problem were shared among two threads where each ran the for-loop to do arithmetic operations on polynomial G1 points.

In the above code snippets, fft_g1 calls fft_g1_fast, which up to scale 16 should at most 1 << 15 times call itself recursively, where each such call will be distributed among the 2 threads. Since we’re computing fft_g1 up to scale 8, there should be (1 << 7) + 1 tasks (not to be confused by OpenMP’s task pragma directive!) for fft_g1_fast or 129 such tasks that will be run in parallel!

Local c-kzg benchmark

Running on my personal computer with i5-7300HQ (4 threads overclocked at 3.50GHz), all mitigations turned off, and a custom Liquorix kernel, I was able to achieve the following results:

Original c-kzg library Parallelized c-kzg library

Original c-kzg library	Parallelized c-kzg library
`1$ ./fft_g1_bench 2*** Benchmarking FFT_g1, 1 second per test. 3fft_g1/scale_4 1729769 ns/op 4fft_g1/scale_5 4935085 ns/op 5fft_g1/scale_6 12897731 ns/op 6fft_g1/scale_7 32022026 ns/op 7fft_g1/scale_8 76552852 ns/op 8fft_g1/scale_9 184970057 ns/op 9fft_g1/scale_10 418273808 ns/op 10fft_g1/scale_11 919499032 ns/op 11fft_g1/scale_12 2025633037 ns/op 12fft_g1/scale_13 4479830518 ns/op 13fft_g1/scale_14 9754557496 ns/op 14fft_g1/scale_15 21125613058 ns/op`	`1$ OMP_NUM_THREADS=4 ./fft_g1_bench 2*** Benchmarking FFT_g1, 1 second per test. 3fft_g1/scale_4 839454 ns/op 4fft_g1/scale_5 2378457 ns/op 5fft_g1/scale_6 6404191 ns/op 6fft_g1/scale_7 16325966 ns/op 7fft_g1/scale_8 38141754 ns/op 8fft_g1/scale_9 90948810 ns/op 9fft_g1/scale_10 204757690 ns/op 10fft_g1/scale_11 457509973 ns/op 11fft_g1/scale_12 1006089135 ns/op 12fft_g1/scale_13 2240095284 ns/op 13fft_g1/scale_14 4879448286 ns/op 14fft_g1/scale_15 10650876381 ns/op`

 1$ ./fft_g1_bench
 2*** Benchmarking FFT_g1, 1 second per test.
 3fft_g1/scale_4 1729769 ns/op
 4fft_g1/scale_5 4935085 ns/op
 5fft_g1/scale_6 12897731 ns/op
 6fft_g1/scale_7 32022026 ns/op
 7fft_g1/scale_8 76552852 ns/op
 8fft_g1/scale_9 184970057 ns/op
 9fft_g1/scale_10 418273808 ns/op
10fft_g1/scale_11 919499032 ns/op
11fft_g1/scale_12 2025633037 ns/op
12fft_g1/scale_13 4479830518 ns/op
13fft_g1/scale_14 9754557496 ns/op
14fft_g1/scale_15 21125613058 ns/op

 1$ OMP_NUM_THREADS=4 ./fft_g1_bench
 2*** Benchmarking FFT_g1, 1 second per test.
 3fft_g1/scale_4 839454 ns/op
 4fft_g1/scale_5 2378457 ns/op
 5fft_g1/scale_6 6404191 ns/op
 6fft_g1/scale_7 16325966 ns/op
 7fft_g1/scale_8 38141754 ns/op
 8fft_g1/scale_9 90948810 ns/op
 9fft_g1/scale_10 204757690 ns/op
10fft_g1/scale_11 457509973 ns/op
11fft_g1/scale_12 1006089135 ns/op
12fft_g1/scale_13 2240095284 ns/op
13fft_g1/scale_14 4879448286 ns/op
14fft_g1/scale_15 10650876381 ns/op

That’s twice as fast with as little effort as putting in a few pragmas!

GitHub Actions CI benchmarks

The fft_g1 benchmark was limited to scale 7 because the overall run time for the job exceeds the 6 hour limit if I were to benchmark it up to scale 16, as Criterion runs each iteration a couple of hundred times to produce more accurate results, and that used to automatically cancel other running CI jobs as jobs submitted to GitHub Actions are limited to 360 minutes.

Benchmarking blst-from-scratch

From the above screenshot we can see that the parallelized version of the library ran 1m 28s shorter than its sequential version, and below are the results of sequential fft_g1 algorithm:

1Benchmarking bench_fft_g1 scale: '7'
2Benchmarking bench_fft_g1 scale: '7': Warming up for 3.0000 s
3Benchmarking bench_fft_g1 scale: '7': Collecting 100 samples in estimated 6.6364 s (200 iterations)
4Benchmarking bench_fft_g1 scale: '7': Analyzing
5bench_fft_g1 scale: '7' time: [33.423 ms 33.785 ms 34.150 ms]

of which the average run time for scale 7 was cut down by 38.926% by its parallel counterpart:

1Benchmarking bench_fft_g1 scale: '7'
2Benchmarking bench_fft_g1 scale: '7': Warming up for 3.0000 s
3Benchmarking bench_fft_g1 scale: '7': Collecting 100 samples in estimated 6.3282 s (300 iterations)
4Benchmarking bench_fft_g1 scale: '7': Analyzing
5bench_fft_g1 scale: '7' time: [20.432 ms 20.634 ms 20.843 ms]
6 change: [-39.822% -38.926% -38.001%] (p = 0.00 < 0.05)
7 Performance has improved.

Benchmarking ckzg

The sequential version of the ckzg library ran 2m 7s faster than the same version of blst-from-scratch because it had other benchmarks that performed faster, though the parallelized version ran 1m 2s faster than its sequential version. Below are the results of the sequantial fft_g1 algorithm:

1Benchmarking bench_fft_g1 scale: '7'
2Benchmarking bench_fft_g1 scale: '7': Warming up for 3.0000 s
3Benchmarking bench_fft_g1 scale: '7': Collecting 100 samples in estimated 6.8313 s (200 iterations)
4Benchmarking bench_fft_g1 scale: '7': Analyzing
5bench_fft_g1 scale: '7' time: [32.194 ms 32.471 ms 32.760 ms]

Yet the parallel version of the fft_g1 algorithm performed much faster than it did for blst-from-scratch, even though both unparallelized versions for both teams performed evenly:

1Benchmarking bench_fft_g1 scale: '7'
2Benchmarking bench_fft_g1 scale: '7': Warming up for 3.0000 s
3Benchmarking bench_fft_g1 scale: '7': Collecting 100 samples in estimated 5.0701 s (300 iterations)
4Benchmarking bench_fft_g1 scale: '7': Analyzing
5bench_fft_g1 scale: '7' time: [16.854 ms 17.107 ms 17.439 ms]
6 change: [-48.216% -47.318% -46.306%] (p = 0.00 < 0.05)
7 Performance has improved.

Summary

OpenMP lets you quickly prototype what is possible to parallelize with the help of CPU profiling tools like Perf
Criterion is actually a really nice benchmarking tool to measure performance, especially when integrated into CI