Reference on fastqrab documentation

LLM Configuration Guide

Mon, 01 Jan 0001 00:00:00 +0000

LLM Configuration Generation Guide #

This guide is optimized for Large Language Models to generate valid fastqrab configurations. It provides structured information with explicit types, constraints, and patterns.

Configuration Structure #

Every configuration has 3 required sections and 2 optional sections:

# example-only - structure overview, not valid TOML
[input] # REQUIRED: Define input files
[[step]] # OPTIONAL: Processing steps (0 or more, order matters)
[output] # REQUIRED: Output configuration
[barcodes.*] # OPTIONAL: Barcode definitions for demultiplexing
[options] # OPTIONAL: Global processing options

Quick Start Patterns #

Pattern 1: Basic Quality Report #

Generate reports without modifying sequences.

Mon, 01 Jan 0001 00:00:00 +0000

Command line interface #

fastqrab is configured exclusively through a TOML document. The CLI is therefore intentionally minimal and focuses on selecting the configuration and the working directory.

Usage #

fastqrab process [config.toml|-] [--allow-overwrite]
fastqrab template
fastqrab verify [config.toml] [--output-dir <OUTPUT_DIR>]
fastqrab interactive [config.toml]
fastqrab completions <SHELL>

Process #

Process FASTQ as described in <config.toml>.(see the TOML format reference). Relative paths are resolved against the current shell directory.

Mon, 01 Jan 0001 00:00:00 +0000

Input section #

The [input] table enumerates all read sources that make up a fragment. At least one segment must be declared.

[input]
 read1 = ['fileA_1.fastq', 'fileB_1.fastq.gz', 'fileC_1.fastq.zst'] # required: one or more paths
 read2 = "fileA_2.fastq.gz" # optional
 index1 = ['index1_A.fastq.gz'] # optional
 # interleaved = [...] # optional, see below

Key	Required	Value type	Notes
segment name (e.g. `read1`)	Yes (at least one)	string or array of strings	Each unique key defines a segment; arrays concatenate multiple files in order.
`interleaved`	No	array of strings	Enables interleaved reading; must list segment names in their in-file order.

Additional points:

Mon, 01 Jan 0001 00:00:00 +0000

Output section #

The [output] table controls how transformed reads and reporting artefacts are written.

[output]
 prefix = "output" # required.
 format = "Fastq", # (optional) output format, defaults to 'Fastq'
					 # Valid values are: Fastq, Fasta, BAM and None (for no sequence output)
 compression = "Gzip" # Raw | Uncompressed | Gzip | Zstd | None (default: Raw)
 compression_threads = 5 # (optional) number of threads to use for compressing gzip data
 suffix = ".fq.gz" # optional override; inferred from format when omitted
 compression_level = 6 # gzip: 0-9, zstd: 1-22, bam: 0-9 (BGZF); defaults are gzip=6, zstd=5
 ix_separator = "_" # optional separator between prefix, infixes, and segments. Defaults to '_'

 report_json = false # write prefix.json (default: false)
 report_html = true # write prefix.html (default: false)
 report_timing = false # write prefix.timing.json (default: false)

 output = ["read1", "read2"] # limit which segments become FASTQ files
 interleave = false # emit a single interleaved FASTQ
 stdout = false # stream to stdout instead of files
 chunk_size = 100000 # Write multiple, numbered output files, each a maximum of chunk_size reads/molecules.

 output_hash_uncompressed = false
 output_hash_compressed = false


[output.bam] # only valid when format = bam

 comment_separation_char = ' ' # read parts after this get put into a 'CO' tag (free-text comment, null terminated in BAM)
 tag_to_bam_tag = {'a-tag-label': 'XY'} # write a-tag-label to bam tag 'xz'
 tag_to_reference = {
 tag = 'assigned-barcode-name',
 # needs one of 
 # references_from_bam = "template.bam", # take reference information from here
 # or
 references_from_barcodes = 'barcode-section-name' # take references from this barcode section.
 }

Key	Default	Description
`prefix`	`"output"`	Base name for all files produced by the run.
`format`	`"Fastq"`	Output format. Valid values are: `Fastq`, `Fasta`, `Bam`, and `None` (for no sequence output).
`compression`	`"Uncompressed"`	Compression format for read outputs. Valid values are: `Gzip`, `Zstd`, `Uncompressed` (alias: `"Raw"`). Must not be set for BAM
`compression_threads`	auto	if using gzip compression, how many thread should be used for compression. See threading
`suffix`	derived from format	Override file extension when interop with other tooling demands a specific suffix.
`compression_level`	gzip: 6, zstd: 5	Fine-tune compression effort. Ignored for `Raw`/`None`. `Bam` maps directly to the BGZF level (0–9).
`report_json` / `report_html`	`false`	Toggle structured or interactive reports.
`report_timing`	`false`	Emit a JSON file with detailed timing information for all steps.
`output`	all input segments	Restrict the subset of segments written to disk. Use an empty list to suppress FASTQs while still running steps that depend on fragment data.
`interleave`	`false`	Generate a single interleaved FASTQ (`{prefix}_interleaved.fq*`).
`stdout`	`false`	Write to stdout. Forces `format = "Raw"`. `Sets interleave=true` if more than one fragment is listed in `output`
`output_hash_uncompressed` / `output_hash_compressed`	`false`	Emit SHA-256 checksums.
`ix_separator`	`"_"`	Separator inserted between `prefix`, any infix (demultiplex labels, inspect names, etc.), and segment names.
`chunk_size`	(unlimited)	Split outputs into multiple files, each containing at most `chunk_size` reads/molecules. For non-interleaved output files, it’s `chunk_size` reads, for interleaved files it’s molecules. This means when mixing interleaved and non-interleaved output, you get the same number of files. Files are numbered sequentially, e.g. `output_read1_0.fq.gz`, …, Numbers start at 0 and use the minimum number of (base 10) digits necessary for alphabetical sorting (by renaming already produced files whenever an extension is needed).

Generated filenames join these components with ix_separator (default _), e.g. {prefix}_{segment}{suffix}. Interleaving replaces segment with interleaved; demultiplexing adds per-barcode infixes before the segment. Checksums use .uncompressed.sha256 or .compressed.sha256 suffixes.

Barcodes section

Mon, 01 Jan 0001 00:00:00 +0000

`[barcodes.*]` section #

Barcode tables supply the sequence-to-sample-name mappings used by Demultiplex, [HammingCorrect](/fastqrab/main/docs/reference/tag-steps/using/HammingCorrect/ and AssignToReference ).

Each table is an independent named dictionary. The name is chosen by the user and referenced from the step that consumes it.

# ignore_in_test
[barcodes.my_barcodes]
AAAAAA = "sample-1"
CCCCCC = "sample-2"
GGGGGG = "sample-3"

The table may appear anywhere in the TOML file; forward and backward references are both valid.

Key format #

Keys are DNA sequences using uppercase IUPAC nucleotide codes. All standard IUPAC ambiguity codes are accepted (e.g. N, R, Y, W, …).

Mon, 01 Jan 0001 00:00:00 +0000

Demultiplexed output #

Demultiplex is a ‘magic’ transformation that forks the output.

You receive one set of output files per barcode (combination) defined.

Transformations downstream are (virtually) duplicated, so you can for example filter to the head reads in each barcode, and get reports for both: all reads and each separate barcode.

Demultiplexing can be done on barcodes, or on boolean tags, and can happen multiple times.

Based on barcodes #

[[step]]
 action = "Demultiplex"
 in_label = "mytag"
 barcodes = "mybarcodes"
 output_unmatched = true # if set, write reads not matching any barcode
 # to a file like ouput_prefix_no-barcode_1.fq

[barcodes.mybarcodes] # can be before and after.
# separate multiple regions with a _
# a Mapping of barcode -> output name.
AAAAAA_CCCCCC = "sample-1" # output files are named prefix{ix_separator}barcode_prefix{ix_separator}segment.suffix
 # with the separator defaulting to '_', e.g. output_sample-1_1.fq.gz
 # or output_sample-1_report.fq.gz

Based on boolean tags #

[[step]]
 action = "TagOtherFile"
 source = "name:read1"
 out_label = "a_bool_tag"
 filename = "path/to/boolean_tags.tsv"
 false_positive_rate = 0

[[step]]
 action = "Demultiplex"
 in_label = "a_bool_tag"
 # output_unmatched = is not valid for boolean tags

Note that this example does not extract the barcodes from the read (use an extract step, such as ExtractRegion).

Mon, 01 Jan 0001 00:00:00 +0000

Options #

There is a small set of runtime knobs exposed under [options].

Most workflows can rely on the defaults.

[options]
 threads = 10
 max_blocks_in_flight = 100
 block_size = 10000
 buffer_size = 102400
 accept_duplicate_files = false
 spot_check_read_pairing = true

Key	Default	Description
`threads`	(auto)	Worker threads for transformations. See threading.
`max_blocks_in_flight`	`100`	How many blocks may be concurrently being processed. Lowering this limits RAM usage.
`block_size`	`10000`	Number of fragments pulled per batch. Increase for very large runs when IO is abundant; decrease to reduce peak memory use.
`buffer_size`	`102400`	Initial bytes reserved per block. The allocator grows buffers on demand, so tuning is rarely necessary.
`accept_duplicate_files`	`false`	Permit the same path to appear multiple times across segments. Useful for fixtures or synthetic tests; keep disabled to catch accidental copy/paste errors.
`spot_check_read_pairing`	`true`	Sample every 1000th fragment to ensure paired reads still share a name prefix; disable when names are intentionally divergent or rely on `ValidateName` to customise the separator.

Changing these knobs can affect memory pressure and concurrency behavior.

Mon, 01 Jan 0001 00:00:00 +0000

Multithreading considerations #

Mbf-fastq-processor is inherently multi-threaded, and strives to make full use of your machine’s cores.

Usually, you should not need to influence the thread counts.

If you want to limit CPU usage, we suggest either systemd based resource control or tools like [cpulimit] (https://github.com/opsengine/cpulimit) instead of coarsely changing thread counts.

Threading architecture #

Mbf-fastq-processor runs the following thread stack for a (non-interleaved configuration):

[decompression / reader threads]
 ↓
[parsers]
 ↓
combining thread
 ↓
[workpool handling steps]
 ↓
[output threads]

Decompression / reader threads #

The number of decompression / reading threads is (in order of precedence)

Mon, 01 Jan 0001 00:00:00 +0000

Benchmark #

For profiling and benchmarking (individual) steps, fastqrab has a special benchmark mode.

This mode focuses on benchmarking the steps, and avoids (most) input and output runtime.

Enable it by adding this TOML section. The output section becomes optional (and ignored) when benchmarking is enabled.

[benchmark]
 enable = true # required to enable benchmark mode
 quiet = false # default. If true, don't output timing information
 molecule_count = 1_000_000

Benchmark mode:

Disables (regular) output
runs in a temp directory,
repeats the first molecule ‘block’ of Options.block_size reads until molecule_count has been exceeded.

The last point means that we will spent very little time in reading & decompression (without rapidgzip / parallel BAM processing the largest runtime parts), and focus on the steps. The drawback here is that your pipeline sees the same reads over and over, which of course will lead to a different ‘hit’ profile for set based tests such as duplication counting, TagOtherFile,
and Demultiplex

Mon, 01 Jan 0001 00:00:00 +0000

Sequencing Adapters #

Paste this count_oligos block into a Report step to identify which adapter is present in your data. See Cookbook 10: Adapter Identification for a complete example.

count_oligos performs exact, full-sequence matching — no mismatches, no IUPAC wildcards.

[[step]] 
 action = 'Report'
 # ...
 count_oligos = {
 # Illumina TruSeq / standard adapters
 # https://support-docs.illumina.com/SHARE/AdapterSequences/adapter-sequences.htm
 "Illumina Nextera/AmpliSeq" = "CTGTCTCTTATACACATCT",
 "Illumina TruSeq R1/miRNA" = "AGATCGGAAGAGCACACGTCTGAACTCCAGTCA",
 "Illumina TruSeq R2" = "AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT",
 "Illumina Small RNA R2" = "GATCGTCGGACTGTAGAACTCTGAACGTGTAGA",
 "Illumina Single End Adapter 1"= "GATCGGAAGAGCTCGTATGCCGTCTTCTGCTTG",
 "Illumina Paired End Adapter 2"= "GATCGGAAGAGCGGTTCAGCAGGAATGCCGAG",

 # MGI/BGI adapters
 # http://seqanswers.com/forums/showthread.php?t=87647 (2nd post)
 "BGI Forward" = "AAGTCGGAGGCCAAGCGGTCTTAGGAAGACAA",
 "BGI Reverse" = "AAGTCGGATCGTAGCCATGTCGTTCTGTGAGCCAAGGAGTTG",

 # QIASeq
 "QIASeq miRNA" = "AACTGTAGGCACCATCAAT",

 # ABI SOLiD
 "ABI SOLiD3 Adapter A" = "CTGCCCCGGGTTCCTCATTCTCTCAGCAGCATG",
 "ABI SOLiD3 Adapter B" = "CCACTACGCCTCCGCTTTCCTCTCTATGGGCAGTCGGTGAT",

 # Clontech
 "Clontech Universal Primer Mix"= "CTAATACGACTCACTATAGGGCAAGCAGTGGTATCAACGCAGAGT",
 }

More adapters may be found in the fastp source

Mon, 01 Jan 0001 00:00:00 +0000

Out of scope #

Things fastqrab will explicitly not do and that won’t be implemented.

Anything based on averaging phred scores #

Based on the average quality in a sliding window. Arithmetic averaging of phred scores is wrong.

see ExtractMeanQuality

Corresponding options in other software #

Trimmomatic SLIDINGWINDOW
fastp –cut_front
fastp –cut_tail
fastp –cut_right

Fast5 #

https://medium.com/@shiansu/a-look-at-the-nanopore-fast5-format-f711999e2ff6 Oxford Nanopore squiggle data. Apparently no formal spec.

kallisto BUS format #

- a brief barcode/umi format for single cell RNA-seq
- needs an 'equivalance class' - i.e. at least pseudo alignment
- weird length restrictions on barcodes and umis (1(!)-32), 
 but stores the length in an uint32...

Alignment #

While it’s tempting to leverage the fastq parsing for an aligner, aligning molecules to references is out of scope for the 1.0 target.

Reference on fastqrab documentation

LLM Configuration Guide

LLM Configuration Generation Guide #

Configuration Structure #

Quick Start Patterns #

Pattern 1: Basic Quality Report #

Command line interface #

Usage #

Process #

Input section #

Output section #

Barcodes section

[barcodes.*] section #

Key format #

Demultiplexed output #

Based on barcodes #

Based on boolean tags #

Options #

Multithreading considerations #

Threading architecture #

Decompression / reader threads #

Benchmark #

Sequencing Adapters #

Out of scope #

Anything based on averaging phred scores #

Corresponding options in other software #

Fast5 #

kallisto BUS format #

Alignment #

`[barcodes.*]` section #