Introduction on mbf-fastq-processor documentation

Mon, 01 Jan 0001 00:00:00 +0000

Command line interface #

mbf-fastq-processor is configured exclusively through a TOML document. The CLI is therefore intentionally minimal and focuses on selecting the configuration and the working directory.

Usage #

mbf-fastq-processor <config.toml> [working_directory]

config.toml – required path to the pipeline description (see the TOML format reference). Relative paths are resolved against the current shell directory.
working_directory – optional override for the process working directory. Use it to anchor relative paths inside the configuration without changing your shell location.

When working_directory is omitted, the binary inherits the environment’s current directory.

TOML format

Mon, 01 Jan 0001 00:00:00 +0000

TOML file format #

mbf-fastq-processor pipelines are defined in a single TOML document. The format favours explicitness: every field is named, order is preserved, and unknown keys are rejected with a descriptive error.

Structure overview #

Section	Required	Purpose
`[input]`	Yes	Declare the FastQ segments and associated source files
`[output]`	Yes	Configure how processed reads and reports are written
`[[step]]`	No*	Define transformations, filters, tag operations, reports
`[options]`	No	Tune runtime knobs such as buffer sizes
`[barcodes.*]`	Conditional	Supply barcode tables for demultiplexing

[[step]] entries are optional in the technical sense—an empty pipeline simply copies data between input and output—but in practice most configurations contain at least one transformation or report.

Mon, 01 Jan 0001 00:00:00 +0000

Input section #

The [input] table enumerates all FastQ sources that make up a fragment. At least one segment must be declared.

[input]
 read1 = ['fileA_1.fastq', 'fileB_1.fastq.gz', 'fileC_1.fastq.zst'] # required: one or more paths
 read2 = "fileA_2.fastq.gz" # optional
 index1 = ['index1_A.fastq.gz'] # optional
 # interleaved = [...] # optional, see below

Key	Required	Value type	Notes
segment name (e.g. `read1`)	Yes (at least one)	string or array of strings	Each unique key defines a segment; arrays concatenate multiple files in order.
`interleaved`	No	array of strings	Enables interleaved reading; must list segment names in their in-file order.

Additional points:

Mon, 01 Jan 0001 00:00:00 +0000

Output section #

The [output] table controls how transformed reads and reporting artefacts are written.

[output]
 prefix = "output" # required.
 format = "Gzip" # Raw | Gzip | Zstd | None (default: Raw)
 suffix = ".fq.gz" # optional override; inferred from format when omitted
 compression_level = 6 # gzip: 0-9, zstd: 1-22; defaults are gzip=6, zstd=5

 report_json = false # write prefix.json
 report_html = true # write prefix.html

 output = ["read1", "read2"] # limit which segments become FastQ files
 interleave = false # emit a single interleaved FastQ
 stdout = false # stream to stdout instead of files

 output_hash_uncompressed = false
 output_hash_compressed = false

Key	Default	Description
`prefix`	`"output"`	Base name for all files produced by the run.
`format`	`"Raw"`	Compression applied to FastQ outputs. `None` disables FastQ emission but still allows reports.
`suffix`	derived from format	Override file extension when interop with other tooling demands a specific suffix.
`compression_level`	gzip: 6, zstd: 5	Fine-tune compression effort. Ignored for `Raw`/`None`.
`report_json` / `report_html`	`false`	Toggle structured or interactive reports.
`output`	all input segments	Restrict the subset of segments written to disk. Use an empty list to suppress FastQs while still running steps that depend on fragment data.
`interleave`	`false`	Generate a single interleaved FastQ (`{prefix}_interleaved.fq*`).
`stdout`	`false`	Write to stdout. Forces `format = "Raw"`. `Sets interleave=true` if more than one fragment is listed in `output`
`output_hash_uncompressed` / `output_hash_compressed`	`false`	Emit SHA-256 checksums.

Generated filenames follow {prefix}_{segment}{suffix}. Interleaving replaces segment with ‘interleaved’. Demultiplexing adds additional infixes. Checksums use .uncompressed.sha256 or .compressed.sha256 suffixes.

Mon, 01 Jan 0001 00:00:00 +0000

FilterByTag #

Remove sequences that have (or don’t have) a tag.

[[step]]
 action = "FilterByTag"
 label = "mytag"
 keep_or_remove = "Keep" # or "Remove"

This transformation filters reads based on the presence or absence of a specified tag. Use “Keep” to retain reads that have the tag, or “Remove” to discard reads that have the tag.

Mon, 01 Jan 0001 00:00:00 +0000

Report #

Capture data for the final report (see the output section).

You can add multiple reports, at any stage of your transformation chain to get e.g. before/after filtering reports.

Example report

[[step]]
 action = 'Report'
 label = "report" # Key that the report will be listed under. Must be distinct
 count = true # count reads at this position
 base_statistics = false # include base distribution at each read position, q20, q30, total, gc bases
 length_distribution = false # capture read length distribution
 duplicate_count_per_read = false # count duplicates using Cukoo filter on each read1/read2/index1/index2
 duplicate_count_per_fragment = false # count duplicates using Cukoo filter, on concatenated read1/read2/index1/index2
 count_oligos = [] # if set, count these oligos in the segment. Full match only, no iupac
 count_oligos_segment = "all" # segment to count oligos in, can be 'all', 'read1', ...

Statistics available (for each ‘segment’. If demultiplexd, per barcode combination):

Mon, 01 Jan 0001 00:00:00 +0000

Demultiplexed output #

[[step]]
 action = "Demultiplex"
 label = "mytag"
 barcodes = "mybarcodes"
 output_unmatched = true # if set, write reads not matching any barcode
 # to a file like ouput_prefix_no-barcode_1.fq

[barcodes.mybarcodes] # can be before and after.
# separate multiple regions with a _
# a Mapping of barcode -> output name.
AAAAAA_CCCCCC = "sample-1" # output files will be named prefix.barcode_prefix.infix.suffix
 # e.g. output_sample-1_1.fq.gz
 # e.g. output_sample-1_report.fq.gz

Demultiplex is a ‘magic’ transformation that ‘forks’ the output.

Mon, 01 Jan 0001 00:00:00 +0000

FilterByNumericTag #

Remove sequences that exceed thresholds on a numeric tag.


[[step]]
 action = "ExtractLength"
 label = "mytag"
 segment = "read1"

[[step]]
 action = "FilterByNumericTag"
 label = "mytag"
 keep_or_remove = "Keep" # or "Remove"
 min_value = 5 # >= this, optional
 max_value = 21 # < this, optional

The example only keeps reads that are between 5 and 20 bases long.

Mon, 01 Jan 0001 00:00:00 +0000

FilterByBoolTag #

Filter for sequences that have a boolean tag.

[[step]]
 action = "FilterByBoolTag"
 label = "mytag"
 keep_or_remove = "Keep" # or "Remove"

Mon, 01 Jan 0001 00:00:00 +0000

FilterEmpty #

[[step]]
 action = "FilterEmpty"
 segment = "read1" # Any of your input segments, or 'All'

Drop the molecule if the read has length 0. (Use after other processing.)

This get’s expanded to an internal tag and a FilteryByNumeric.

This is necessary if your modification can produce ’empty' reads - downstream aligners like STAR tend to dislike these in their input.

On segment=‘All’, only filters reads that are empty in all parts. Use multiple FilterEmpty steps to filter if any part is empty.

Mon, 01 Jan 0001 00:00:00 +0000

Postfix #

Add DNA to the end of read sequences.

[[step]]
 action = "Postfix"
 seq = "agtc" # DNA sequence to add at end of reads. Checked to be agtcn
 qual = "IIII" # same length as seq. Your responsibility to have valid phred values
 segment = "read1" # Any of your input segments

This transformation adds a specified sequence and corresponding quality scores to the end of reads.

Mon, 01 Jan 0001 00:00:00 +0000

Prefix #

Add text to the beginning of read sequences.

[[step]]
 action = "Prefix"
 seq = "agtTCAa" # DNA sequence to add at beginning of reads. Checked to be agtcn
 qual = "IIIBIII" # same length as seq. Your responsibility to have valid phred values
 segment = "read1" # Any of your input segments

This transformation adds a specified sequence and corresponding quality scores to the beginning of reads.

Mon, 01 Jan 0001 00:00:00 +0000

ReplaceTagWithLetter #

Replace sequence bases in tagged regions with a specified letter.

[[step]]
 action = "ReplaceTagWithLetter"
 label = "mytag" # Tag containing regions to replace
 letter = "N" # Replacement character (defaults to 'N')

This transformation replaces all sequence bases within the regions defined by a tag with a specified replacement character. Quality scores are preserved unchanged. This is commonly used to mask low-quality regions as ‘N’ characters.

Parameters #

label: Name of the tag containing regions to be replaced
letter: Single character to replace bases with (defaults to ‘N’ if not specified)

Example Use Cases #

Mask low-quality bases identified by ExtractRegionsOfLowQuality
Replace specific sequence motifs identified by other extraction steps
Convert tagged regions to ambiguous bases for downstream analysis

The tag must have been created by a previous extraction step and must contain location information.

Mon, 01 Jan 0001 00:00:00 +0000

TrimAtTag #

Trim the read at the position of a tag.

[[step]]
 action = "TrimAtTag"
 label = "mytag"
 direction = "Start" # or "End"
 keep_tag = false # if true, the tag sequence is kept in the read

This transformation trims the read at the position where a tag was found.

The direction parameter determines whether to trim from the start or end of the tag, and keep_tag determines whether the tag sequence itself is retained.

Mon, 01 Jan 0001 00:00:00 +0000

QuantifyTag #

Count the occurrences of each tag-sequence.

[[step]]
 action = "QuantifyTag"
 label = "mytag"
 infix = "tagcount" # output file is output_prefix_infix.tag.qr.json

This transformation counts how many times each unique tag value appears and outputs the results to a JSON file.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractAnchor #

Extract regions relative to a previously tagged anchor position.

# First create an anchor tag. Iupac, regex, ExtractRegion, your choice.
[[step]]
 action = "ExtractIUPAC"
 search = "CAYA"
 label = "anchor_tag"
 segment = "read1"
 anchor = "Anywhere"
 max_mismatches = 0

# Then extract relative to that anchor
[[step]]
 action = "ExtractAnchor"
 label = "mytag"
 input_label = "anchor_tag" # tag that provides the anchor position
 regions = [[-2, 4], [4, 1]] # [start, length] pairs relative to anchor
 region_separator = "_" # (optional) separator between regions

This transformation uses the leftmost position of a previously established tag as the anchor point and extracts specified regions relative to that position.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLength #

Extract the length of a read as a tag.

[[step]]
 action = "ExtractLength"
 label = "mytag"
 segment = "read1" # Any of your input segments, or 'All'

This transformation creates a tag containing the length of the specified read.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegex #

Extract a regexp result. Stores an empty string if not found.

[[step]]
 action = "ExtractRegex"
 label = "mytag"
 search = "^CT(..)CT"
 replacement = "$1" # standard regex replacement syntax
 segment = "read1" # Any of your input segments

This transformation searches for a regular expression pattern in the specified read and extracts the matching portion as a tag.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegion #

Extract a fixed position region.

[[step]]
 action = "ExtractRegion"
 start = 5
 length = 8
 segment = "read1" # Any of your input segments
 label = "umi"

This transformation extracts a fixed-length region from the specified read at a given position and stores it as a tag.

Use ExtractRegions if your region is actually multiple regions (possibly from different segments).

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegions #

Extract from multiple fixed position regions.

[[step]]
 action = "ExtractRegions"
 regions = [
 {segment = "read1", start = 0, length = 8},
 {segment = "read1", start = 12, length = 4},
 ]
 label = "barcode"

This transformation extracts multiple fixed-length regions from reads and concatenates them into a single tag.

ExtractRegions with only one region are exactly equivalent to ExtractRegion.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegionsOfLowQuality #

Extract regions (min size 1 bp) where bases have quality scores below threshold.

[[step]]
 action = "ExtractRegionsOfLowQuality"
 segment = "read1" # Any of your input segments
 min_quality = 60 # Quality threshold (Phred+33)
 label = "low_quality_regions"

This transformation scans through quality scores of the specified segment and identifies contiguous regions where quality scores are below the specified threshold. Each low-quality region becomes a tagged region with location information (start position and length).

Parameters #

segment: Which read to analyze for low-quality regions
min_quality: Quality score threshold using Phred+33 encoding. See Phred quality score for ASCII character mapping
label: Tag name to store the extracted regions

Example #

With min_quality = 60 (ASCII ‘<’), any bases with quality scores below ‘<’ will be identified as low-quality regions. This is useful for masking or filtering poor-quality sequences.

Mon, 01 Jan 0001 00:00:00 +0000

TagOtherFileByName #

Mark reads based on wether names are present in another file.

[[step]]
 action = "TagOtherFileByName"
 segment = "read1" # which segment's name are we using
 label = "present_in_other"
 filename = "names.fastq" # Can read fastq (also compressed), or sam/bam files
 false_positive_rate = 0.01 # false positive rate (0..1)
 seed = 42 # seed for randomness
 ignore_unaligned = false # in case of BAM/SAM, whether to ignore unaligned reads
 fastq_readname_end_char = " " # (optional) char (byte value) at which to cut input fastq read names before comparing. If not set, no cutting is done.
 reference_readname_end_char = "/" # (optional) char (byte value) at which to cut reference read names before storing them.

This step marks reads by comparing their names against names from another file.

Mon, 01 Jan 0001 00:00:00 +0000

TagOtherFileBySequence #

Marks reads based on wether sequences are present in another file.

[[step]]
 action = "TagOtherFileBySequence"
 label = "present_in_other_file"
 filename = "sequences.fastq" # fastq (also compressed), or sam/bam files
 segment = "read1" # Any of your input segments
 false_positive_rate = 0.01 # false positive rate (0..1)
 seed = 42 # seed for randomness

This step annotates reads by comparing their sequences against sequences from another file.

Please note our remarks about cuckoo filters.

Mon, 01 Jan 0001 00:00:00 +0000

ForgetTag #

Forget about a tag. Useful if you want to store tags in a table, but not this one.

[[step]]
 action = "ForgetTag"
 label = "mytag"

This transformation removes a specified tag from the molecule’s tag collection.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInComment #

Store currently present tags as comments on read names.

[[step]]
 action = "StoreTagInComment"
 label = "mytag" # if set, only store this tag
 segment = "read1" # Any of your input segments, or 'All'
 comment_insert_char = " " # (optional) char at which to insert comments
 comment_separator = "|" # (optional) char to separate comments
 region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple

Comments are key=value pairs, separated by comment_separator which defaults to ‘|’. They get inserted at the first comment_insert_char, which defaults to space.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInSequence #

Store the tag’s replacement in the sequence, replacing the original sequence at that location.

[[step]]
 action = "StoreTagInSequence"
 label = "mytag"
 ignore_missing = true # if false, an error is raised if the tag is missing

This transformation stores the tag’s value back into the sequence, replacing the original sequence at that location.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagLocationInComment #

Store the coordinates of a tag in the comment (start-end, 0-based, half-open).

[[step]]
 action = "StoreTagLocationInComment"
 label = "mytag"
 segment = "read1" # Any of your input segments, or 'All'
 comment_insert_char = " " # (optional) char at which to insert comments
 comment_separator = "|" # (optional) char to separate comments

This transformation stores the location coordinates of a tag as a comment in the read name, useful for tracking where tags were extracted from.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagsInTable #

Store the tags in a TSV table.

[[step]]
 action = "StoreTagsInTable"
 table_filename = "tags.tsv"
 compression = "Raw" # Raw, Gzip, Zstd
 region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple

This transformation writes all current tags to a tab-separated values (TSV) table file for further analysis.

Extract IUPAC

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPAC #

[[step]]
 action = "ExtractIUPAC"
 label = "mytag"
 anchor = 'Left' # Left | Right | Anywhere
 search = "CTN" # what we are searching
 segment = 'read1' # Any of your input segments

Search and extract a sequence from the read, defined by a IUPAC string.

If anchor = ‘Anywhere’, ExtractIUPAC will find the left most occurance.

Extract IUPAC suffix

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACSuffix #

[[step]]
 action = "ExtractIUPACSuffix"
 label = "mytag"
 query = "AGTCA" # the adapter to trim. Straigth bases only, no IUPAC.
 segment = "read1" # Any of your input segments (default: read1)
 min_length = 3 # uint, the minimum length of match between the end of the read and
 # the start of the adapter
 max_mismatches = 1 # How many mismatches to accept

Find a potentially truncated IUPAC string sequence at the end of a read.

Simple comparison with a max mismatch hamming distance, requiring only the first min length bases of the query to match at the end of the read.

Extract IUPAC with Indels

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACWithIndel #

[[step]]
 action = "ExtractIUPACWithIndel"
 label = "adapter"
 search = "AGTC" # IUPAC pattern to align against
 max_mismatches = 1 # allowed substitutions (IUPAC-aware)
 max_indel_bases = 1 # total insertions + deletions allowed
 max_total_edits = 2 # optional overall edit ceiling
 anchor = 'Anywhere' # Left | Right | Anywhere
 segment = 'read1' # defaults to read1

Locate an IUPAC pattern even when the read contains small insertions or deletions relative to the pattern. The extractor performs a semiglobal alignment (pattern vs. read segment) using IUPAC-aware scoring and returns the aligned span as a location tag.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInFastQ #

## Store the content of a tag in a fastq file.
## Needs a 'location 'tag'.
## Can store other tags in the read name.
## quality scores are set to '~'.
# [[step]]
# action = "StoreTagInFastQ"
# label = "mytag" # ${output.prefix}.tag.mytag.fq${.suffix_from_format}
# format = "Raw" # Raw, Gzip, Zstd
## compression_level = 6 # (optional) compression level for gzip (0-9) or zstd (1-22)
						 # defaults: gzip=6, zstd=5
# comment_tags = []# e.g. ["other_tag"] # see StoreTagInComment
# comment_location_tags = ["mytag"] # (optional) tags to add location info for, defaults to [label]
# # set to [] to disable location tracking
# comment_insert_char = ' ' # (optional) char at which to insert comments
# comment_separator = '|' # (optional) char to separate comments
# region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple

Store the sequence of a tag in a fastq file, with other tags optionally stored in the read name as comments.

Mon, 01 Jan 0001 00:00:00 +0000

Options #

There is a small set of runtime knobs exposed under [options]. Most workflows can rely on the defaults.

[options]
 thread_count = -1
 block_size = 10000
 buffer_size = 102400
 accept_duplicate_files = false

Key	Default	Description
`thread_count`	`-1`	Worker threads for transformations. `-1` autotunes per CPU; most runtime is still dominated by decompression threads, so gains are modest.
`block_size`	`10000`	Number of fragments pulled per batch. Increase for very large runs when IO is abundant; decrease to reduce peak memory use.
`buffer_size`	`102400`	Initial bytes reserved per block. The allocator grows buffers on demand, so tuning is rarely necessary.
`accept_duplicate_files`	`false`	Permit the same path to appear multiple times across segments. Useful for fixtures or synthetic tests; keep disabled to catch accidental copy/paste errors.

Changing these knobs can affect memory pressure and concurrency behaviour. Measure before and after if you deviate from defaults.

Mon, 01 Jan 0001 00:00:00 +0000

LowercaseTag #

[[step]]
 action = "LowercaseTag"
 label = "mytag"

[[step]]
	action = "StoreTagInSequence"
	label = "mytag"

Replace the sequence of the tag with it’s lowercase version.

Follow with StoreTagInSequence to apply the lowercase tag back onto the read.

See the tag section for tag generation.

Mon, 01 Jan 0001 00:00:00 +0000

LowercaseSequence #

[[step]]
 action = "LowercaseSequence"
 segment = "read1" # Any of your input segments, or 'All'

Convert the complete sequence to lowercase.

Mon, 01 Jan 0001 00:00:00 +0000

UppercaseSequence #

[[step]]
 action = "UppercaseSequence"
 segment = "read1" # Any of your input segments, or 'All'

Convert the complete sequence to uppercase.

Mon, 01 Jan 0001 00:00:00 +0000

UppercaseTag #

[[step]]
 action = "UppercaseTag"
 label = "mytag"

[[step]]
	action = "StoreTagInSequence"
	label = "mytag"

Replace the sequence of the tag with its uppercase version.

Follow with StoreTagInSequence to apply the uppercase tag back onto the read.

See the tag section for tag generation.

Mon, 01 Jan 0001 00:00:00 +0000

FilterSample #

[[step]]
 action = "FilterSample"
 p = 0.5 # float, the chance for any given read to be kept
 # 0..1
 seed = 42 # (optional) random seed for reproducibility

Randomly sample a percentage of reads. Requires a random seed to ensure reproducibility.

Mon, 01 Jan 0001 00:00:00 +0000

ConvertQuality #

[[step]]
 action = "ConvertQuality"
 from = "Illumina1.8"# Illumin1.8|Illumina1.3|Sanger|Solexa"
 to = "Solexa" # same range as from. Illumina1.8 is an alias for Sanger

Convert quality scores between various encodings / meanings.

See https://en.wikipedia.org/wiki/Phred_quality_score

Will error if from == to.

Corresponding options in other software #

trimmomatic TOPHRED33

Mon, 01 Jan 0001 00:00:00 +0000

CutEnd #

[[step]]
 action = "CutEnd"
 n = 5 # positive integer, cut n nucleotides from the end of the read
 segment = "read1" # Any of your input segments (default: read1)

Cut nucleotides from the end of the read.

May produce empty reads; filter those with FilterEmpty.

Mon, 01 Jan 0001 00:00:00 +0000

CutStart #

[[step]]
 action = "CutStart"
 n = 5 # positive integer, cut n nucleotides from the start of the read
 segment = "read1" # Any of your input segments

Cut nucleotides from the start of the read.

May produce empty reads; filter those with FilterEmpty.

Mon, 01 Jan 0001 00:00:00 +0000

[[step]]
 action = "ExtractRegions"
 regions = [
 {segment= "Read1", start = 0, length = 8},
 {segment= "Read1", start = 12, length = 4},
 ]
 label = "umi"
 region_separator = "_" # (optional) str, what to put between the regions, defaults to '_'

[[step]]
 action = "StoreTagInComment" 
 label = "umi"

Extract a sequence from the read and place it in the read name’s comment section, so a (space separated) ‘key=value’ pair is added to the read name.

Supports multiple region-extraction.

See the tag section for more tag generation options.

Mon, 01 Jan 0001 00:00:00 +0000

Head #

[[step]]
 action = "Head"
 n = 1000 # positive integer, number of reads to keep

Output just the first n molecules.

Mon, 01 Jan 0001 00:00:00 +0000

Rename #

[[step]]
 action = "Rename"
 search = '(.)/([1/2])$'
 replacement = '$1 $2'

Apply a regular expression based renaming to the reads.

It is always applied to all available segments (read1, read2, index1, index2).

The example above fixes old school MGI reads for downstream processing, like fastp’s ‘–fix_mgi’ option

You can use the full power of the rust regex crate here.

Read index placeholder #

After the regex replacement runs, the special literal {{READ_INDEX}} is expanded to the running 1-based index of each logical read. When multiple segments are present (for example read1/read2 pairs), every segment for the same read receives the same index so pairs stay aligned. This makes it easy to re-sequence identifiers:

Mon, 01 Jan 0001 00:00:00 +0000

ReverseComplement #

[[step]]
 action = "ReverseComplement"
 segment = "read1" # Any of your input segments (default: read1)

Reverse-complements the read sequence (and reverses the quality).

This supports IUPAC codes (U is complemented to A, so it’s not strictly reversible). Unknown letters are output verbatim.

Mon, 01 Jan 0001 00:00:00 +0000

Skip #

[[step]]
 action = "Skip"
 n = 1000 # positive integer, number of reads to skip

Skips the first n molecules.

Mon, 01 Jan 0001 00:00:00 +0000

Swap #

[[step]]
 action = "Swap"
 segment_a = "read1"
 segment_b = "read2"

Swaps exactly two segments.

Mon, 01 Jan 0001 00:00:00 +0000

MaxLen #

[[step]]
 action = "Truncate"
 n = 100 # the maximum length of the read. Cut at end if longer
 segment = "read1" # Any of your input segments (default: read1)

Cut the read down to `n’ bases.

Mon, 01 Jan 0001 00:00:00 +0000

Out of scope #

Things mbf-fastq-processor will explicitly not do and that won’t be implemented.

Anything based on averaging phred scores #

Based on the average quality in a sliding window. Arithmetic averaging of phred scores is wrong.

see ExtractMeanQuality

Corresponding options in other software #

Trimmomatic SLIDINGWINDOW
fastp –cut_front
fastp –cut_tail
fastp –cut_right

Fast5 #

https://medium.com/@shiansu/a-look-at-the-nanopore-fast5-format-f711999e2ff6 Oxford Nanopore squiggle data. Apparently no formal spec.

kallisto BUS format #

- a brief barcode/umi format for single cell RNA-seq
- needs an 'equivalance class' - i.e. at least pseudo alignment
- weird length restrictions on barcodes and umis (1(!)-32), 
 but stores the length in an uint32...

Mon, 01 Jan 0001 00:00:00 +0000

Inspect #

Dump a few reads to a file for inspection at this point in the graph.

[[step]]
 action = "Inspect"
 n = 1000 # how many reads
 infix = "inspect_at_point" # output filename infix
 segment = "read1" # Any of your input segments
 suffix = "compressed" # (optional) custom suffix for filename
 format = "gzip" # (optional) compression format: raw, gzip, zstd (defaults to raw)
 compression_level = 1 # (optional) compression level for gzip (0-9) or zstd (1-22)
 # defaults: gzip=6, zstd=5

Output filename pattern:

Mon, 01 Jan 0001 00:00:00 +0000

Progress #

Emit progress to stdout (default) or a .progress log file, if output_infix is set. (filename is {output_prefix}_{infix}.progress)

[[step]]
 action = "Progress"
 n = 100_000
 output_infix = "progress" # optional

Every n reads, report on total progress, total reads per second. At the end, report final runtime and reads/second.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLowComplexity #

[[step]]
 action = "ExtractLowComplexity"
 label = "complexity"
 segment = "read1" # Any of your input segments, or 'All'

Calculate read complexity, based on the percentage of bases that are changed from their predecessor.

A good filter value might be 0.30, which means 30% complexity is required. See FilterByNumericTag.

Corresponding options in other software #

fastp: -low_complexity_filter

Mon, 01 Jan 0001 00:00:00 +0000

TrimQualityEnd #

[[step]]
 action = "ExtractLowQualityEnd"
 label = "low_quality_ends"
 min_qual = 20 # u8, minimum quality to keep (in whatever your score is encoded in)
 # either a char like 'A' or a number 0..128 (typical phred score is 33..75)
 segment = "read1" # Any of your input segments

Define a region of low quality bases at the end of reads.

Corresponding options in other software #

Trimmomatic: TRAILING (if paired with TrimAtTag)

Mon, 01 Jan 0001 00:00:00 +0000

TrimQualityStart #

[[step]]
 action = "ExtractLowQualityStart"
 min_qual = 20 # u8, minimum quality to keep (in whatever your score is encoded in)
 # either a char like 'A' or a number 0..128 (typical phred score is 33..75)
 segment = "read1" # Any of your input segments
 label = "bad_starts"

Define a region with low quality bases (below threshold) at steart of read.

Corresponding options in other software #

Trimmomatic: LEADING (if combined with TrimAtTag)

Mon, 01 Jan 0001 00:00:00 +0000

ExtractMeanQuality #

We don’t support calculating the ‘average quality’.

This is typically a bad idea, see https://www.drive5.com/usearch/manual/avgq.html for a discussion of the issues.

To illustrate, 140 x Q35 + 10 x Q2 reads have an ‘average’ phred of 33, but 6.4 expected wrong bases A read with 150 x Q25 has a much wores ‘average’ phred of 25, but a much lower expected number of errors at 0.5!

Corresponding options in other software #

Trimmomatic: AVGQUAL:
fastp: –average_qual

Mon, 01 Jan 0001 00:00:00 +0000

ExtractPolyTail #

[[step]]
 action = "ExtractPolyTail"
 label = "tag-label"
 segment = "read1" # Any of your input segments (default: read1)
 min_length = 5 # positive integer, the minimum number of repeats of the base
 base = "A" # one of AGTCN., the 'base' to trim (or . for any repeated base)
 max_mismatch_rate = 0.1 # float 0.0..=1.0, how many mismatches are allowed in the repeat
 max_consecutive_mismatches = 3 # how many consecutive mismatches are allowed

[[step]]
 action = "TrimAtTag"
 label ="tag-label"
 direction = "End"
 keep_tag = false

Identify either a specific letter (AGTC or N) repetition, or any base repetition (base = ‘.’) at the end of the read.

Mon, 01 Jan 0001 00:00:00 +0000

#ExtractQualifiedBases

[[step]]
 action = "ExtractQualifiedBases"
 min_quality = 30 # the quality value >= which a base is qualified 
 # In your phred encoding. Typically 33..75
 # a byte or a number 0...255
 segment = "read1" # Any of your input segments, or 'All'
 label = "tag_name"

Calculate by the percentage of bases that are ‘unqualified’, that is below a user defined threshold.

Corresponding options in other software #

fastp : –qualified_quality_phred / –unqualified_percent_limit (if combined with FilterByNumericTag)

Mon, 01 Jan 0001 00:00:00 +0000

FilterDuplicates #

[[step]]
 action = "TagDuplicates"
 false_positive_rate = 0.00001 #
 # the false positive rate of the filter.
 # 0..1
 seed = 59 # required!
 segment = "All" # Any of your input segments, or 'All'
 label = "dups"

[[step]]
 action = "FilterByBoolTag"
 label = "dups"
 keep_or_remove = "Remove" # Keep|Remove

Tag duplicates (2nd onwards) from the stream using a Cuckoo filter.

That’s a probabilistic data structure, accordingly there’s a false positive rate, and a tunable memory requirement.

Needs a seed for the random number generator, and a segment to know which reads to consider for deduplication (filters the complete molecule, like all other filters of course).

Mon, 01 Jan 0001 00:00:00 +0000

HammingCorrect #

Correct a tag to one of a predefined set of ‘barcodes’ using closest hamming distance.

[[step]]
 action = "HammingCorrect"
 label_in = "extracted_tag"
 label_out = "corrected_tag"
 barcodes = "mybarcodelist"
 max_hamming_distance = 1
 on_no_match = 'remove' # 'remove', 'empty', 'keep'

[barcodes.mybarcodelist]
 "AAAA" = "ignored" # only read when demultiplexing

on_no_match controls what happens if the tag cannot be corrected within the max_hamming_distance:

remove: Remove the hit (location and sequence), useful for FilterByTag later.
keep: Keep the original tag (and location)
empty: Keep the original location, but set the tag to empty.

Mon, 01 Jan 0001 00:00:00 +0000

ValidateName #

Verify that all segments expose the same read name (or a shared prefix).

[[step]]
 action = "ValidateName"
 # Optional separator character; the comparison stops at the first match
 readname_end_char = "_" # Optional. Do not set for exact matching. Otherwise, a byte character

When no separator character is provided the entire name must match exactly across all segments.

Otherwise the read names are truncated after the first readname_end_char, and the prefixes must match exactly. For example, use readname_end_char = "_" for typical Illumina _1/_2 suffixes.

Mon, 01 Jan 0001 00:00:00 +0000

ValidateQuality #

Validate that all scores are between 33..=41

[[step]]
 action = "ValidateQuality"
 segment = "read1" # Any of your input segments, or 'All'
 encoding = 'Illumina1.8' # 'Illumina1.8|Illumina1.3|Sanger|Solexa'
 # Illumina1.8 is an alias for Sanger.

The encoding defines the accepted range of values.

If you want to convert quality codes, use ConvertQuality.

See https://pmc.ncbi.nlm.nih.gov/articles/PMC2847217/ , table 1

Mon, 01 Jan 0001 00:00:00 +0000

ValidateSeq #

Validate that only allowed characters are in the sequence.

[[step]]
 action = "ValidateSeq"
 allowed = "AGTC" # String. Example 'ACGTN', the allowed characters
 segment = "read1" # Any of your input segments, or 'All'

(Older) Documentation Versions

Mon, 01 Jan 0001 00:00:00 +0000

Available builds:

main
v0.8.1
v0.8.0-test (this build)
v0.8.0

Extract GC Content

Mon, 01 Jan 0001 00:00:00 +0000

ExtractGCContent #

[[step]]
 action = "ExtractGCContent"
 segment = "read1" # Any of your input segments, or 'All'
 label="gc"

Count what percentage of bases are GC (as opposed to AT). Non-AGTC bases (e.g. N) are ignored in both the numerator and denominator.

Extract N Count

Mon, 01 Jan 0001 00:00:00 +0000

ExtractNCount #

[[step]]
 action = "ExtractNCount"
 segment = "read1" # Any of your input segments, or 'All'
 label="ncount"

Count how many N are present in the read

Corresponding options in other software #

fastp: –n_base_limit (if combined with FilterByNumericTag)