Benchmark #

For profiling and benchmarking (individual) steps, mbf-fastq-processor has a special benchmark mode.

This mode focuses on benchmarking the steps, and avoids (most) input and output runtime.

Enable it by adding this TOML section. The output section becomes optional (and ignored) when benchmarking is enabled.

[benchmark]
    enable = true # required to enable benchmark mode
    quiet = false # default. If true, don't output timing information
    molecule_count = 1_000_000

Benchmark mode:

• Disables (regular) output
• runs in a temp directory,
• repeats the first molecule ‘block’ of Options.block_size reads until molecule_count has been exceeded.

The last point means that we will spent very little time in reading & decompression (without rapidgzip / parallel BAM processing the largest runtime parts), and focus on the steps. The drawback here is that your pipeline sees the same reads over and over, which of course will lead to a different ‘hit’ profile for set based tests such as duplication counting, TagOtherFileByName,
and Demultiplex

Results #

Parsing, Ryzen Ai Max+ 395, 2025 NVME #

Benchmarking the parsing performance with a no-output, report read count only configuration, using one file of about 44 million 75bp reads, we observe:

• FASTQ, gzip, rapidgzip, 5 threads: 11.2 million reads / second

• FASTQ, gzip, single thread / flate2 : 4.7 million reads / second

• FASTQ, uncompressed: about 6.4 million reads / second

• BAM, single-threaded: 4.7 million reads / second

• BAM, multi-threaded, 5 threads: 4.3 million reads / second (slower than single-thread. We have a bottleneck in read-to-fastq-adaptation)

• FASTA, uncompressed: 3.4 million reads / second

• FASTA, gzip, single thread, flate2 : 2.8 million reads / second

• FASTA, gzip, rapidgzip: 4 million reads / second

Per step micro benchmarks #

This section is mostly useful to estiamte which steps are fast and which are not.

On Ryzen AI Max+ 395 using benchmark mode, 12 threads, 10 million (10k repeated) single end reads.