Reference on mbf-fastq-processor documentation

Mon, 01 Jan 0001 00:00:00 +0000

Command line interface #

mbf-fastq-processor is configured exclusively through a TOML document. The CLI is therefore intentionally minimal and focuses on selecting the configuration and the working directory.

Usage #

mbf-fastq-processor <config.toml> [working_directory]

config.toml – required path to the pipeline description (see the TOML format reference). Relative paths are resolved against the current shell directory.
working_directory – optional override for the process working directory. Use it to anchor relative paths inside the configuration without changing your shell location.

When working_directory is omitted, the binary inherits the environment’s current directory.

TOML format

Mon, 01 Jan 0001 00:00:00 +0000

TOML file format #

mbf-fastq-processor pipelines are defined in a single TOML document. The format favours explicitness: every field is named, order is preserved, and unknown keys are rejected with a descriptive error.

Structure overview #

Section	Required	Purpose
`[input]`	Yes	Declare the FastQ segments and associated source files
`[output]`	Yes	Configure how processed reads and reports are written
`[[step]]`	No*	Define transformations, filters, tag operations, reports
`[options]`	No	Tune runtime knobs such as buffer sizes
`[barcodes.*]`	Conditional	Supply barcode tables for demultiplexing

[[step]] entries are optional in the technical sense—an empty pipeline simply copies data between input and output—but in practice most configurations contain at least one transformation or report.

Mon, 01 Jan 0001 00:00:00 +0000

Input section #

The [input] table enumerates all FastQ sources that make up a fragment. At least one segment must be declared.

[input]
 read1 = ['fileA_1.fastq', 'fileB_1.fastq.gz', 'fileC_1.fastq.zst'] # required: one or more paths
 read2 = "fileA_2.fastq.gz" # optional
 index1 = ['index1_A.fastq.gz'] # optional
 # interleaved = [...] # optional, see below

Key	Required	Value type	Notes
segment name (e.g. `read1`)	Yes (at least one)	string or array of strings	Each unique key defines a segment; arrays concatenate multiple files in order.
`interleaved`	No	array of strings	Enables interleaved reading; must list segment names in their in-file order.

Additional points:

Mon, 01 Jan 0001 00:00:00 +0000

Output section #

The [output] table controls how transformed reads and reporting artefacts are written.

[output]
 prefix = "output" # required.
 format = "Gzip" # Raw | Gzip | Zstd | None (default: Raw)
 suffix = ".fq.gz" # optional override; inferred from format when omitted
 compression_level = 6 # gzip: 0-9, zstd: 1-22; defaults are gzip=6, zstd=5

 report_json = false # write prefix.json
 report_html = true # write prefix.html

 output = ["read1", "read2"] # limit which segments become FastQ files
 interleave = false # emit a single interleaved FastQ
 stdout = false # stream to stdout instead of files

 output_hash_uncompressed = false
 output_hash_compressed = false

Key	Default	Description
`prefix`	`"output"`	Base name for all files produced by the run.
`format`	`"Raw"`	Compression applied to FastQ outputs. `None` disables FastQ emission but still allows reports.
`suffix`	derived from format	Override file extension when interop with other tooling demands a specific suffix.
`compression_level`	gzip: 6, zstd: 5	Fine-tune compression effort. Ignored for `Raw`/`None`.
`report_json` / `report_html`	`false`	Toggle structured or interactive reports.
`output`	all input segments	Restrict the subset of segments written to disk. Use an empty list to suppress FastQs while still running steps that depend on fragment data.
`interleave`	`false`	Generate a single interleaved FastQ (`{prefix}_interleaved.fq*`).
`stdout`	`false`	Write to stdout. Forces `format = "Raw"`. `Sets interleave=true` if more than one fragment is listed in `output`
`output_hash_uncompressed` / `output_hash_compressed`	`false`	Emit SHA-256 checksums.

Generated filenames follow {prefix}_{segment}{suffix}. Interleaving replaces segment with ‘interleaved’. Demultiplexing adds additional infixes. Checksums use .uncompressed.sha256 or .compressed.sha256 suffixes.

Mon, 01 Jan 0001 00:00:00 +0000

Demultiplexed output #

[[step]]
 action = "Demultiplex"
 label = "mytag"
 barcodes = "mybarcodes"
 output_unmatched = true # if set, write reads not matching any barcode
 # to a file like ouput_prefix_no-barcode_1.fq

[barcodes.mybarcodes] # can be before and after.
# separate multiple regions with a _
# a Mapping of barcode -> output name.
AAAAAA_CCCCCC = "sample-1" # output files will be named prefix.barcode_prefix.infix.suffix
 # e.g. output_sample-1_1.fq.gz
 # e.g. output_sample-1_report.fq.gz

Demultiplex is a ‘magic’ transformation that ‘forks’ the output.

Mon, 01 Jan 0001 00:00:00 +0000

Options #

There is a small set of runtime knobs exposed under [options]. Most workflows can rely on the defaults.

[options]
 thread_count = -1
 block_size = 10000
 buffer_size = 102400
 accept_duplicate_files = false

Key	Default	Description
`thread_count`	`-1`	Worker threads for transformations. `-1` autotunes per CPU; most runtime is still dominated by decompression threads, so gains are modest.
`block_size`	`10000`	Number of fragments pulled per batch. Increase for very large runs when IO is abundant; decrease to reduce peak memory use.
`buffer_size`	`102400`	Initial bytes reserved per block. The allocator grows buffers on demand, so tuning is rarely necessary.
`accept_duplicate_files`	`false`	Permit the same path to appear multiple times across segments. Useful for fixtures or synthetic tests; keep disabled to catch accidental copy/paste errors.

Changing these knobs can affect memory pressure and concurrency behaviour. Measure before and after if you deviate from defaults.

Mon, 01 Jan 0001 00:00:00 +0000

Out of scope #

Things mbf-fastq-processor will explicitly not do and that won’t be implemented.

Anything based on averaging phred scores #

Based on the average quality in a sliding window. Arithmetic averaging of phred scores is wrong.

see ExtractMeanQuality

Corresponding options in other software #

Trimmomatic SLIDINGWINDOW
fastp –cut_front
fastp –cut_tail
fastp –cut_right

Fast5 #

https://medium.com/@shiansu/a-look-at-the-nanopore-fast5-format-f711999e2ff6 Oxford Nanopore squiggle data. Apparently no formal spec.

kallisto BUS format #

- a brief barcode/umi format for single cell RNA-seq
- needs an 'equivalance class' - i.e. at least pseudo alignment
- weird length restrictions on barcodes and umis (1(!)-32), 
 but stores the length in an uint32...