Introduction on fastqrab documentation

Mon, 01 Jan 0001 00:00:00 +0000

Philosophy #

fastqrab transforms (DNA) sequencing reads for downstream analysis.

Its focus are on

correctness
reproducibility
a lack of surprises
friendliness
speed

Correctness #

We strive to do the right thing, always.

To that end, fastqrab is tested with more than 500 end-to-end, input-to-output tests, both during development and via continuous integration.

Reproducibility #

Repeated runs on the same bits (input data & configuration) must deliver the same output bits. Every time.

Mon, 01 Jan 0001 00:00:00 +0000

Getting started with development #

The easiest way to get started with working on fastqrab is to clone the repository:

jj git clone --colocate https://github.com/TyberiusPrime/fastqrab

git clone https://github.com/TyberiusPrime/fastqrab

if you’re not yet convinced that Jujutsu is the better git.

Development environment #

Using nix develop to enter a shell with all the necessary requirements using Nix.

If you don’t use nix, you’re on your own to supply a matching rust compiler, openssl and pkg-config.

Mon, 01 Jan 0001 00:00:00 +0000

Troubleshooting #

I don’t know what to do after an error message #

Well, that’s below our targets, we want our error messages to tell you enough for you to be able to fix it.

Please file a bug report in our issue tracker.

I received a friendly panic message #

These look like this

Well, this is embarrassing.

fastqrab had a problem and crashed. To help us diagnose the problem you can send us a crash report.

We have generated a report file at "/tmp/nix-shell.ty6dWk/report-89ffc0f3-8076-4f55-83f0-c0a5d2fb3b55.toml".

This kind of error message, which wraps a rust ‘panic’ only happens if fastqrab has managed to get into an impossible or unforeseen state.

LLM Configuration Guide

Mon, 01 Jan 0001 00:00:00 +0000

LLM Configuration Generation Guide #

This guide is optimized for Large Language Models to generate valid fastqrab configurations. It provides structured information with explicit types, constraints, and patterns.

Configuration Structure #

Every configuration has 3 required sections and 2 optional sections:

# example-only - structure overview, not valid TOML
[input] # REQUIRED: Define input files
[[step]] # OPTIONAL: Processing steps (0 or more, order matters)
[output] # REQUIRED: Output configuration
[barcodes.*] # OPTIONAL: Barcode definitions for demultiplexing
[options] # OPTIONAL: Global processing options

Quick Start Patterns #

Pattern 1: Basic Quality Report #

Generate reports without modifying sequences.

Mon, 01 Jan 0001 00:00:00 +0000

Implementing your own transformation #

Let’s implement an example step (also called a ’transformation’) that converts your reads to FuNkYcAsE by upper/lower-casing every other letter.

This guide assumes you have basic linux command line knowledge, and that you can edit text files (source code).

We are going to start by devising a test case, making sure it fails, and then step by step adding all the parts we need. This will illustrate all the infrastructure the project has to support you in this.

Mon, 01 Jan 0001 00:00:00 +0000

Command line interface #

fastqrab is configured exclusively through a TOML document. The CLI is therefore intentionally minimal and focuses on selecting the configuration and the working directory.

Usage #

fastqrab process [config.toml|-] [--allow-overwrite]
fastqrab template
fastqrab verify [config.toml] [--output-dir <OUTPUT_DIR>]
fastqrab interactive [config.toml]
fastqrab completions <SHELL>

Process #

Process FASTQ as described in <config.toml>.(see the TOML format reference). Relative paths are resolved against the current shell directory.

Mon, 01 Jan 0001 00:00:00 +0000

Input section #

The [input] table enumerates all read sources that make up a fragment. At least one segment must be declared.

[input]
 read1 = ['fileA_1.fastq', 'fileB_1.fastq.gz', 'fileC_1.fastq.zst'] # required: one or more paths
 read2 = "fileA_2.fastq.gz" # optional
 index1 = ['index1_A.fastq.gz'] # optional
 # interleaved = [...] # optional, see below

Key	Required	Value type	Notes
segment name (e.g. `read1`)	Yes (at least one)	string or array of strings	Each unique key defines a segment; arrays concatenate multiple files in order.
`interleaved`	No	array of strings	Enables interleaved reading; must list segment names in their in-file order.

Additional points:

Mon, 01 Jan 0001 00:00:00 +0000

Output section #

The [output] table controls how transformed reads and reporting artefacts are written.

[output]
 prefix = "output" # required.
 format = "Fastq", # (optional) output format, defaults to 'Fastq'
					 # Valid values are: Fastq, Fasta, BAM and None (for no sequence output)
 compression = "Gzip" # Raw | Uncompressed | Gzip | Zstd | None (default: Raw)
 compression_threads = 5 # (optional) number of threads to use for compressing gzip data
 suffix = ".fq.gz" # optional override; inferred from format when omitted
 compression_level = 6 # gzip: 0-9, zstd: 1-22, bam: 0-9 (BGZF); defaults are gzip=6, zstd=5
 ix_separator = "_" # optional separator between prefix, infixes, and segments. Defaults to '_'

 report_json = false # write prefix.json (default: false)
 report_html = true # write prefix.html (default: false)
 report_timing = false # write prefix.timing.json (default: false)

 output = ["read1", "read2"] # limit which segments become FASTQ files
 interleave = false # emit a single interleaved FASTQ
 stdout = false # stream to stdout instead of files
 chunk_size = 100000 # Write multiple, numbered output files, each a maximum of chunk_size reads/molecules.

 output_hash_uncompressed = false
 output_hash_compressed = false


[output.bam] # only valid when format = bam

 comment_separation_char = ' ' # read parts after this get put into a 'CO' tag (free-text comment, null terminated in BAM)
 tag_to_bam_tag = {'a-tag-label': 'XY'} # write a-tag-label to bam tag 'xz'
 tag_to_reference = {
 tag = 'assigned-barcode-name',
 # needs one of 
 # references_from_bam = "template.bam", # take reference information from here
 # or
 references_from_barcodes = 'barcode-section-name' # take references from this barcode section.
 }

Key	Default	Description
`prefix`	`"output"`	Base name for all files produced by the run.
`format`	`"Fastq"`	Output format. Valid values are: `Fastq`, `Fasta`, `Bam`, and `None` (for no sequence output).
`compression`	`"Uncompressed"`	Compression format for read outputs. Valid values are: `Gzip`, `Zstd`, `Uncompressed` (alias: `"Raw"`). Must not be set for BAM
`compression_threads`	auto	if using gzip compression, how many thread should be used for compression. See threading
`suffix`	derived from format	Override file extension when interop with other tooling demands a specific suffix.
`compression_level`	gzip: 6, zstd: 5	Fine-tune compression effort. Ignored for `Raw`/`None`. `Bam` maps directly to the BGZF level (0–9).
`report_json` / `report_html`	`false`	Toggle structured or interactive reports.
`report_timing`	`false`	Emit a JSON file with detailed timing information for all steps.
`output`	all input segments	Restrict the subset of segments written to disk. Use an empty list to suppress FASTQs while still running steps that depend on fragment data.
`interleave`	`false`	Generate a single interleaved FASTQ (`{prefix}_interleaved.fq*`).
`stdout`	`false`	Write to stdout. Forces `format = "Raw"`. `Sets interleave=true` if more than one fragment is listed in `output`
`output_hash_uncompressed` / `output_hash_compressed`	`false`	Emit SHA-256 checksums.
`ix_separator`	`"_"`	Separator inserted between `prefix`, any infix (demultiplex labels, inspect names, etc.), and segment names.
`chunk_size`	(unlimited)	Split outputs into multiple files, each containing at most `chunk_size` reads/molecules. For non-interleaved output files, it’s `chunk_size` reads, for interleaved files it’s molecules. This means when mixing interleaved and non-interleaved output, you get the same number of files. Files are numbered sequentially, e.g. `output_read1_0.fq.gz`, …, Numbers start at 0 and use the minimum number of (base 10) digits necessary for alphabetical sorting (by renaming already produced files whenever an extension is needed).

Generated filenames join these components with ix_separator (default _), e.g. {prefix}_{segment}{suffix}. Interleaving replaces segment with interleaved; demultiplexing adds per-barcode infixes before the segment. Checksums use .uncompressed.sha256 or .compressed.sha256 suffixes.

Mon, 01 Jan 0001 00:00:00 +0000

FilterByTag #

This transformation filters molecules based on the presence or absence of a specified tag.

Use “Keep” to retain molecules that have the tag, or “Remove” to discard reads that have the tag.

If used on a boolean tag, the boolean value of the tag is used to determine whether to keep or remove the read.

For numeric tags, use FilterByNumericTag.

[[step]]
 action = "FilterByTag"
 in_label = "mytag"
 keep_or_remove = "Keep" # or "Remove"

Mon, 01 Jan 0001 00:00:00 +0000

Report #

Capture data for the final report (see the output section).

You can add multiple reports, at any stage of your transformation chain to get e.g. before/after filtering reports.

Example report

[[step]]
 action = 'Report'
 name = "report" # Key that the report will be listed under. Must be distinct
 count = true # count reads at this position
 base_statistics = false # include base distribution at each read position, q20, q30, total, gc bases
 length_distribution = false # capture read length distribution
 duplicate_count_per_read = false # count duplicates using Cukoo filter on each read1/read2/index1/index2
 duplicate_count_per_fragment = false # count duplicates using Cukoo filter, on concatenated read1/read2/index1/index2
 count_oligos = {name1 = "AGTC", name2 = "ACCCCC"} # map of name -> sequence. Full match only, no iupac
 count_oligos_segment = "all" # segment to count oligos in, can be 'all', 'read1', ...
 tag_histograms = ["mytag"] # Calculate a histogram for this tag

Statistics available (for each ‘segment’. If demultiplexed, per barcode combination):

Barcodes section

Mon, 01 Jan 0001 00:00:00 +0000

`[barcodes.*]` section #

Barcode tables supply the sequence-to-sample-name mappings used by Demultiplex, [HammingCorrect](/fastqrab/main/docs/reference/tag-steps/using/HammingCorrect/ and AssignToReference ).

Each table is an independent named dictionary. The name is chosen by the user and referenced from the step that consumes it.

# ignore_in_test
[barcodes.my_barcodes]
AAAAAA = "sample-1"
CCCCCC = "sample-2"
GGGGGG = "sample-3"

The table may appear anywhere in the TOML file; forward and backward references are both valid.

Key format #

Keys are DNA sequences using uppercase IUPAC nucleotide codes. All standard IUPAC ambiguity codes are accepted (e.g. N, R, Y, W, …).

Mon, 01 Jan 0001 00:00:00 +0000

Demultiplexed output #

Demultiplex is a ‘magic’ transformation that forks the output.

You receive one set of output files per barcode (combination) defined.

Transformations downstream are (virtually) duplicated, so you can for example filter to the head reads in each barcode, and get reports for both: all reads and each separate barcode.

Demultiplexing can be done on barcodes, or on boolean tags, and can happen multiple times.

Based on barcodes #

[[step]]
 action = "Demultiplex"
 in_label = "mytag"
 barcodes = "mybarcodes"
 output_unmatched = true # if set, write reads not matching any barcode
 # to a file like ouput_prefix_no-barcode_1.fq

[barcodes.mybarcodes] # can be before and after.
# separate multiple regions with a _
# a Mapping of barcode -> output name.
AAAAAA_CCCCCC = "sample-1" # output files are named prefix{ix_separator}barcode_prefix{ix_separator}segment.suffix
 # with the separator defaulting to '_', e.g. output_sample-1_1.fq.gz
 # or output_sample-1_report.fq.gz

Based on boolean tags #

[[step]]
 action = "TagOtherFile"
 source = "name:read1"
 out_label = "a_bool_tag"
 filename = "path/to/boolean_tags.tsv"
 false_positive_rate = 0

[[step]]
 action = "Demultiplex"
 in_label = "a_bool_tag"
 # output_unmatched = is not valid for boolean tags

Note that this example does not extract the barcodes from the read (use an extract step, such as ExtractRegion).

Mon, 01 Jan 0001 00:00:00 +0000

FilterByNumericTag #

Remove molecules by thresholding on numeric tag.


[[step]]
 action = "CalcLength"
 out_label = "mytag"
 segment = "read1"

[[step]]
 action = "FilterByNumericTag"
 in_label = "mytag"
 keep_or_remove = "Keep" # or "Remove"
 min_value = 5 # >= this, optional
 max_value = 21 # < this, optional

The example only keeps reads that are between 5 and 20 bases long.

Consider using an EvalExpression for more complicated decisions.

Parser Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Parser Architecture #

Overview #

fastqrab uses a custom-built parser designed for high performance and correctness when processing FASTQ. The parser’s design emphasizes:

Zero-copy parsing where possible to minimize memory allocations
Streaming architecture to handle files of any size
Transparent compression support (raw, gzip, zstd)
Cross-platform compatibility (Unix/Windows line endings)

(FASTA and BAM files are processed differently, see below).

The Zero-Copy Challenge with Compressed Files #

Why Not Pure Zero-Copy? #

A common optimization in bioinformatics tools is “zero-copy” parsing, where the parser operates directly on memory-mapped file contents without allocating separate buffers. This works well for uncompressed files stored on fast storage in suitable file formats.

Parser Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Parser Architecture #

Overview #

fastqrab uses a custom-built parser designed for high performance and correctness when processing FASTQ, FASTA, and BAM files. The parser’s design emphasizes:

Zero-copy parsing where possible to minimize memory allocations
Streaming architecture to handle files of any size
Transparent compression support (uncompressed, gzip, zstd)
Stateful parsing to handle reads spanning block boundaries
Cross-platform compatibility (Unix/Windows line endings)

The Zero-Copy Challenge with Compressed Files #

Why Not Pure Zero-Copy? #

Mon, 01 Jan 0001 00:00:00 +0000

Code Coverage #

Running coverage #

python3 dev/coverage.py

This prints a line-coverage summary with exclusions applied (see below).

Other formats #

python3 dev/coverage.py --lcov # writes coverage.lcov
python3 dev/coverage.py --html # writes coverage-html/html/index.html
python3 dev/coverage.py --json # writes coverage.json
python3 dev/coverage.py --all # all of the above + summary
python3 dev/coverage.py --open # --html and open in browser

The summary, LCOV, and HTML outputs all apply exclusions. HTML is rendered by genhtml (from the lcov package, included in the nix dev shell) from the post-processed lcov file; excluded lines appear as uninstrumented (white, not red). JSON is generated directly by cargo-llvm-cov and does not reflect exclusion comments.

Mon, 01 Jan 0001 00:00:00 +0000

FilterEmpty #

[[step]]
 action = "FilterEmpty"
 segment = "read1" # Any of your input segments, or 'All'

Drop the molecule if the read has length 0. (Use after other processing.)

This get’s expanded to an internal tag and a FilteryByNumeric.

This is necessary if your modification can produce ’empty' reads - downstream aligners like STAR tend to dislike these in their input.

On segment=‘All’, only filters reads that are empty in all parts. Use multiple FilterEmpty steps to filter if any part is empty.

Mon, 01 Jan 0001 00:00:00 +0000

Postfix #

Add DNA to the end of read sequences.

[[step]]
 action = "Postfix"
 seq = "agtc" # DNA sequence to add at end of reads. Checked to be agtcn
 qual = "IIII" # same length as seq. Your responsibility to have valid phred values
 segment = "read1" # Any of your input segments
 if_tag = "mytag"
 encoding = 'Illumina1.8' # optional, default=sanger 'Illumina1.8|Illumina1.3|Sanger|Solexa'
 # Illumina1.8 is an alias for Sanger.

This transformation adds a specified sequence and corresponding quality scores to the end of reads.

Mon, 01 Jan 0001 00:00:00 +0000

Prefix #

Add text to the beginning of read sequences.

[[step]]
 action = "Prefix"
 seq = "agtTCAa" # DNA sequence to add at beginning of reads. Checked to be agtcn
 qual = "IIIBIII" # same length as seq. Your responsibility to have valid phred values
 segment = "read1" # Any of your input segments
 if_tag = "mytag"
 encoding = 'Illumina1.8' # optional, default=sanger 'Illumina1.8|Illumina1.3|Sanger|Solexa'
 # Illumina1.8 is an alias for Sanger.

This transformation adds a specified sequence and corresponding quality scores to the beginning of reads.

Mon, 01 Jan 0001 00:00:00 +0000

ReplaceTagWithLetter #

Replace sequence bases in tagged regions with a specified letter.

[[step]]
 action = "ReplaceTagWithLetter"
 in_label = "mytag" # Tag containing regions to replace
 letter = "N" # Replacement character (defaults to 'N')

This transformation replaces all sequence bases within the regions defined by a tag with a specified replacement character. Quality scores are preserved unchanged. This is commonly used to mask low-quality regions as ‘N’ characters.

Parameters #

in_label: Name of the tag containing regions to be replaced
letter: Single character to replace bases with (defaults to ‘N’ if not specified)

Example Use Cases #

Mask low-quality bases identified by ExtractRegionsOfLowQuality
Replace specific sequence motifs identified by other extraction steps
Convert tagged regions to ambiguous bases for downstream analysis

The tag must have been created by a previous extraction step and must contain location information.

Mon, 01 Jan 0001 00:00:00 +0000

TrimAtTag #

Trim the read at the position of a tag.

[[step]]
 action = "TrimAtTag"
 in_label = "mytag" # must be a location tag
 direction = "Start" # or "End"
 keep_tag = false # if true, the tag sequence is kept in the read

This transformation trims the read at the position where a tag was found.

The direction parameter determines whether to trim from the start or end of the tag, and keep_tag determines whether the tag sequence itself is retained.

Mon, 01 Jan 0001 00:00:00 +0000

QuantifyTag #

Count the occurrences of each tag-sequence.

[[step]]
 action = "QuantifyTag"
 in_label = "mytag"
 infix = "tagcount" # output file is output{ix_separator}tagcount.qr.json (default '_' → output_tagcount.qr.json)
 region_separator = "_" # optional. If the tag consists of multiple regions, join them with this string

This transformation counts how many times each unique tag value appears and outputs the results to a JSON file.

Demultiplex interaction #

Barcodes are counted per demultiplexed stream.

Mon, 01 Jan 0001 00:00:00 +0000

CalcLength #

Extract the length of a read as a tag.

[[step]]
 action = "CalcLength"
 out_label = "mytag"
 segment = "read1" # Any of your input segments, or 'All'

This transformation creates a tag containing the length of the specified read.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegex #

Extract a regexp result. Stores an empty string if not found.

[[step]]
 action = "ExtractRegex"
 out_label = "mytag"
 search = "^CT(..)CT"
 replacement = "$1" # optional. standard regex replacement syntax
 source = "read1" # An input segment (to read from sequence), or name:<segment> to read from a read's name.

This transformation searches for a regular expression pattern in the specified read and extracts the matching portion as a tag.

The value actually ’extracted’ is after replacement has been performed.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegion #

Extract a fixed position region.

[[step]]
 action = "ExtractRegion"
 start = 5
 length = 8
 source = "read1" # Any of your input segments, name:<segment_name> or tag:<any_location_or_string_tag>
 anchor = "Start" # 'Start' or 'End' - start is relative to this
 out_label = "umi"

This transformation extracts a fixed-length region from the specified read at a given position and stores it as a tag.

Start positions are 0-based.

End positions require negative starts (as in python, start=-1, length=1 is the last character).

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegions #

Extract from multiple regions with flexible source and anchoring options.

Basic Usage #

Extract from fixed positions in sequence data:

[[step]]
 action = "ExtractRegions"
 regions = [
 {source = "read1", start = 0, length = 8, anchor= "Start"},
 {source = "read1", start = -12, length = 4, anchor="End"},
 ]
 out_label = "barcode"

Advanced Usage with Sources #

Extract from tag-derived positions:

# First create an anchor tag
[[step]]
 action = "ExtractIUPAC"
 search = "CAYA"
 out_label = "anchor_tag"
 segment = "read1"
 anchor = "Anywhere"
 max_mismatches = 0

# Then extract relative to that anchor
[[step]]
 action = "ExtractRegions"
 regions = [
 { source = "tag:anchor_tag", start = -2, length = 4, anchor = "Start" },
 { source = "tag:anchor_tag", start = 4, length = 1, anchor = "Start" }
 ]
 out_label = "relative_regions"

Extract from read names:

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegionsOfLowQuality #

Extract regions where bases have quality scores below threshold, with a minimum length requirement.

[[step]]
 action = "ExtractRegionsOfLowQuality"
 segment = "read1" # Any of your input segments
 min_quality = 60 # Quality threshold (Phred+33)
 min_length = 5 # Minimum region length (bp)
 out_label = "low_quality_regions"

This transformation scans through quality scores of the specified segment and identifies contiguous regions where quality scores are below the specified threshold. Each low-quality region that meets the minimum length requirement becomes a tagged region with location information (start position and length).

Mon, 01 Jan 0001 00:00:00 +0000

TagOtherFile #

Marks reads based on whether ’they’ are present in another file.

Supports comparing by read sequence, read name, and tags.

[[step]]
 action = "TagOtherFile"
 source = 'read1' # <segment>, name:<segment> or tag<tag_name>
 out_label = "present_in_other_file"
 filename = "names.fastq" # Can read fastq (also compressed), or SAM/BAM, or fasta files
 false_positive_rate = 0.01 # false positive rate (0..1)
 seed = 42 # seed for randomness
 include_mapped = true # in case of BAM/SAM, whether to include aligned reads
 include_unmapped = true # in case of BAM/SAM, whether to include unaligned reads
 # other_read_name_end_character " " # in name: mode, Cut the other files read names at this character

This step annotates reads by comparing them to another file.

Mon, 01 Jan 0001 00:00:00 +0000

ConcatTags #

Concatenate multiple tags into a single tag.

# ignore_in_test
[[step]]
 action = "ConcatTags"
 in_labels = ["mytag", "mytag2"] # list of tags to concatenate (minimum 2)
 out_label = "combined" # output tag name
 on_missing = "merge_present" # required: "merge_present" or "set_missing"
 separator = "_" # (optional) separator for string concatenation

This transformation combines multiple tags into a single output tag. The behavior depends on the types of input tags and how missing tags are handled.

Behavior by Tag Type #

Location Tags Only #

When all input tags are location tags (e.g., from ExtractIUPAC, ExtractAnchor), the transformation:

Mon, 01 Jan 0001 00:00:00 +0000

ForgetAllTags #

Remove every tag currently stored for the read batch.

[[step]]
 action = "ForgetAllTags"

Use this when you want to clear all tag labels before continuing. It is handy after persisting tags to an external table, or before running steps that must not see previously extracted metadata.

Mon, 01 Jan 0001 00:00:00 +0000

ForgetTag #

Forget about a tag. Useful if you want to store tags in a table, but not this one.

[[step]]
 action = "ForgetTag"
 in_label = "mytag"

This transformation removes a specified tag from the molecule’s tag collection.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagBackInSequence #

Store the tag’s replacement in the sequence, replacing the original sequence at that location.

[[step]]
 action = "StoreTagBackInSequence"
 in_label = "mytag"
 ignore_missing = true # if false, an error is raised if the tag is missing

This transformation stores the tag’s value back into the sequence, replacing the original sequence at that location.

Note that if this changes the length of the sequence, existing location tags will loose their location data (retaining their sequence though).

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInComment #

Store currently present tags as comments on read names.

[[step]]
 action = "StoreTagInComment"
 in_labels = ["mytag"] # Store these tags. Not optional. May be a single string as well. Alias 'in_label'
 segment = "read1" # Any of your input segments, or 'All'
 comment_insert_char = " " # (optional) char at which to insert comments
 comment_separator = "|" # (optional) char to separate comments
 region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple

Comments are key=value pairs, separated by comment_separator which defaults to ‘|’. They get inserted before the first comment_insert_char, which defaults to input.options.read_comment_char.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagsInTable #

Store the tags in a TSV table.

[[step]]
 action = "StoreTagsInTable"
 infix = "tags"
 compression = "Raw" # Raw, Gzip, Zstd
 region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple
 in_labels = ["mytag", ] # Store just these tags. Optional, all tags store if not set
 include_read_name = true # (optional) include the ReadName column. Default: true

This transformation writes all current tags to a tab-separated values (TSV) table file for further analysis.

Extract IUPAC

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPAC #

[[step]]
 action = "ExtractIUPAC"
 out_label = "mytag"
 anchor = 'Left' # Left | Right | Anywhere
 search = "CTN" # what we are searching. May also be a list ["CTN", "GAN", ...]
 segment = 'read1' # Any of your input segments
 max_mismatches = 0 # required. How many mismatches are allowed

Search and extract a sequence from the read, defined by a IUPAC string.

Anchor is the regex equivalent of ^ (Left), $ (Right) or no anchor (Anywhere).

Extract IUPAC suffix

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACSuffix #

[[step]]
 action = "ExtractIUPACSuffix"
 out_label = "mytag"
 search = "AGTCA" # the adapter to trim. Straigth bases only, no IUPAC.
 segment = "read1" # Any of your input segments (default: read1)
 min_length = 3 # uint, the minimum length of match between the end of the read and
 # the start of the adapter
 max_mismatches = 0 # required. How many mismatches to accept

Find a potentially truncated IUPAC string sequence at the end of a read.

Simple comparison with a max mismatch hamming distance, requiring only the first min length bases of the query to match at the end of the read.

Extract IUPAC with Indels

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACWithIndel #

[[step]]
 action = "ExtractIUPACWithIndel"
 out_label = "adapter"
 search = "AGTC" # IUPAC pattern to align against
 max_mismatches = 1 # allowed substitutions (IUPAC-aware)
 max_indel_bases = 1 # total insertions + deletions allowed
 max_total_edits = 2 # optional overall edit ceiling
 anchor = 'Anywhere' # Left | Right | Anywhere
 segment = 'read1' # defaults to read1

Locate an IUPAC pattern even when the read contains small insertions or deletions relative to the pattern. The extractor performs a semiglobal alignment (pattern vs. read segment) using IUPAC-aware scoring and returns the aligned span as a location tag.

Store Tag In FASTQ

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInFastQ #

# Store the content of a tag in a fastq file.
# Needs a 'location 'tag'.
# Can store other tags in the read name.
# quality scores are set to '~'.
# With demultiplexing: creates separate files per barcode
 [[step]]
 action = "StoreTagInFastQ"
 in_label = "mytag" # tag to store. File name is derived with this as infix
 format = "Fastq" # FASTQ / FASTA / BAm
 compression = "gz" # or "zstd" | "none" # (optional) compression format, not if format == BAM
 compression_level = 6 # (optional) compression level for gzip (0-9) or zstd (1-22)
 					 # defaults: gzip=6, zstd=5
 comment_tags = []# e.g. ["other_tag"] # see StoreTagInComment
 comment_insert_char = ' ' # (optional) char at which to insert comments
 comment_separator = '|' # (optional) char to separate comments
 region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple

Store the sequence of a tag in a fastq file, with other tags (and virtual tags) optionally stored in the read name as comments.

Mon, 01 Jan 0001 00:00:00 +0000

CalcExpectedError #

Compute aggregated per-base error probabilities (expected errors) for each read assuming PHRED+33 qualities.

[[step]]
 action = "CalcExpectedError"
 out_label = "expected_error"
 aggregate = "sum" # or "max"
 segment = "read1" # Any of your input segments, or 'All'

If your data is not encoded as Phred+33, convert it first (for example, with ConvertQuality) before running this step. Values outside of the Phred+33 range will lead to an stop with an error.

Set aggregate = "sum" to calculate the sum of per-base error probabilities. Use aggregate = "max" to store only the worst base’s error probability for each read or read pair.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInSequence #

Insert a tag’s string value into a read sequence at the position defined by another location tag.

[[step]]
 action = "StoreTagInSequence"
 in_value_label = "mytag" # location or string tag to insert
 in_position_label = "mytag2" # location tag defining where to insert
 anchor = "Start" # "Start"/"left" or "End"/"right"

Parameters #

Parameter	Type	Required	Description
`in_value_label`	location or string tag	yes	Tag whose sequence is inserted into the read
`in_position_label`	location tag	yes	Tag that defines the insertion position
`anchor`	`"Start"` / `"left"` / `"End"` / `"right"`	yes	Whether to insert before the leftmost position (`Start`) or after the rightmost end (`End`) of the position tag

How it works #

in_position_label is a location tag pointing to a region (or multiple regions) in a read. The insertion point is derived from the anchor:

Mon, 01 Jan 0001 00:00:00 +0000

ConvertRegionsToLength #

Turn region tags (such as those produced by ExtractRegion/ExtractRegions) into numeric length tags.

[[step]]
 action = "ExtractRegion"
 out_label = "adapter"
 source = "read1"
 start = 0
 len = 12
 anchor = "Start"

[[step]]
 action = "ConvertRegionsToLength"
 out_label = "adapter_len"
 in_label = "adapter"

The new tag stores the total span (in bases) covered by all regions on each read.
Reads without the source tag receive a length of 0.
label must be different from region_label; the step keeps the original region tag.

Mon, 01 Jan 0001 00:00:00 +0000

EvalExpression #

[[step]]
 action = "EvalExpression"
 out_label = "outtag"
 expression = "log(2, mytag + 1)" # log to base 2
 result_type = "numeric" # or bool.

Calculate a fasteval expression on your tags, which you can then pass to .FilterByTag.

You can use any tags previously defined on the molecule as variables in the expression.

Additional, there’s a series of virtual tags available:

len_<segment-name> - the length of the specified segment (e.g. len_read1).
len_<tag-label> - the length of the specified tag (e.g. len_mytag). For location tags, this is the length of the underlying matched regions (which may change / be lost when reads are truncated - eval before truncation if necessary). For string tags (= ExtractRegex with source=name:...) this is the length of the replaced string.
read_no - the running number of the read (starting with 0)

Language #

Besides the regular arithmetic operators (+, -, *, /, %, ^) this supports log(base, val), e(), pi(), int(), ceil(), floor(), round(), abs(), sign(), min(a,b,…), max(a,b,…) sin(radians), cos(radians), tan(radians), sinh(radians), cosh(radians), tanh(radians), Use any defined tag by name. Location/string tags are converted to booleans by their presence.

Mon, 01 Jan 0001 00:00:00 +0000

Options #

There is a small set of runtime knobs exposed under [options].

Most workflows can rely on the defaults.

[options]
 threads = 10
 max_blocks_in_flight = 100
 block_size = 10000
 buffer_size = 102400
 accept_duplicate_files = false
 spot_check_read_pairing = true

Key	Default	Description
`threads`	(auto)	Worker threads for transformations. See threading.
`max_blocks_in_flight`	`100`	How many blocks may be concurrently being processed. Lowering this limits RAM usage.
`block_size`	`10000`	Number of fragments pulled per batch. Increase for very large runs when IO is abundant; decrease to reduce peak memory use.
`buffer_size`	`102400`	Initial bytes reserved per block. The allocator grows buffers on demand, so tuning is rarely necessary.
`accept_duplicate_files`	`false`	Permit the same path to appear multiple times across segments. Useful for fixtures or synthetic tests; keep disabled to catch accidental copy/paste errors.
`spot_check_read_pairing`	`true`	Sample every 1000th fragment to ensure paired reads still share a name prefix; disable when names are intentionally divergent or rely on `ValidateName` to customise the separator.

Changing these knobs can affect memory pressure and concurrency behavior.

Mon, 01 Jan 0001 00:00:00 +0000

CalcKmers #

Count the number of kmers from a read that match those in a database built from reference sequences.

[[step]]
 action = "CalcKmers"
 out_label = "mytag"
 segment = "read1" # Any of your input segments, or 'All'
 filename = ['reference.fa', 'database.fq'] # Path (string) or list of such
 count_reverse_complement = true # whether to also include each revcomp of a kmer in the database
 k = 21
 min_count = 2 # optional, defaults to 1

This transformation:

Builds a kmer database from the specified sequence files (all input formatws)
Extracts all kmers of length k from the reference sequences
Filters kmers by min_count (minimum occurrences in the reference to be included)
For each read, counts how many of its kmers appear in the database
Creates a numeric tag with the kmer match count

Parameters #

out_label: Tag name to store the kmer count
segment: Which segment to quantify (read1, read2, index1, index2, or ‘All’)
files: List of sequence files to build the kmer database from
count_reverse_complement: (alias: “canonical”) Whether to include reverse complements of kmers in the database (‘canonical kmers’)
k: Kmer length
min_count: Minimum number of times a kmer must appear in the reference files to be included in the database (default: 1). Sum of forward and reverse complement counts if count_reverse_complement is true.

Use Cases #

Contamination detection: Quantify or filter reads matching known contaminant sequences
Quality control: Count kmers from adapter or primer sequences
Species identification: Measure presence of species-specific kmers

Notes #

Only kmers with only valid DNA bases (A, C, G, T) are counted; kmers containing N or other ambiguous bases are skipped
Kmer matching is case-insensitive

Mon, 01 Jan 0001 00:00:00 +0000

Multithreading considerations #

Mbf-fastq-processor is inherently multi-threaded, and strives to make full use of your machine’s cores.

Usually, you should not need to influence the thread counts.

If you want to limit CPU usage, we suggest either systemd based resource control or tools like [cpulimit] (https://github.com/opsengine/cpulimit) instead of coarsely changing thread counts.

Threading architecture #

Mbf-fastq-processor runs the following thread stack for a (non-interleaved configuration):

[decompression / reader threads]
 ↓
[parsers]
 ↓
combining thread
 ↓
[workpool handling steps]
 ↓
[output threads]

Decompression / reader threads #

The number of decompression / reading threads is (in order of precedence)

Mon, 01 Jan 0001 00:00:00 +0000

Benchmark #

For profiling and benchmarking (individual) steps, fastqrab has a special benchmark mode.

This mode focuses on benchmarking the steps, and avoids (most) input and output runtime.

Enable it by adding this TOML section. The output section becomes optional (and ignored) when benchmarking is enabled.

[benchmark]
 enable = true # required to enable benchmark mode
 quiet = false # default. If true, don't output timing information
 molecule_count = 1_000_000

Benchmark mode:

Disables (regular) output
runs in a temp directory,
repeats the first molecule ‘block’ of Options.block_size reads until molecule_count has been exceeded.

The last point means that we will spent very little time in reading & decompression (without rapidgzip / parallel BAM processing the largest runtime parts), and focus on the steps. The drawback here is that your pipeline sees the same reads over and over, which of course will lead to a different ‘hit’ profile for set based tests such as duplication counting, TagOtherFile,
and Demultiplex

Mon, 01 Jan 0001 00:00:00 +0000

Performance Benchmarking #

fastqrab includes a comprehensive benchmarking suite for measuring and analyzing the performance of individual transformation steps.

Overview #

The benchmarking system uses:

Criterion: Industry-standard Rust benchmarking framework
Benchmark mode: Built-in mode that focuses on step performance while minimizing I/O overhead

Quick Start #

Running Benchmarks #

# Run all step benchmarks
cargo bench --bench simple_benchmarks

Benchmark Architecture #

Benchmark Mode #

The processor includes a special benchmark mode that:

Mon, 01 Jan 0001 00:00:00 +0000

Sequencing Adapters #

Paste this count_oligos block into a Report step to identify which adapter is present in your data. See Cookbook 10: Adapter Identification for a complete example.

count_oligos performs exact, full-sequence matching — no mismatches, no IUPAC wildcards.

[[step]] 
 action = 'Report'
 # ...
 count_oligos = {
 # Illumina TruSeq / standard adapters
 # https://support-docs.illumina.com/SHARE/AdapterSequences/adapter-sequences.htm
 "Illumina Nextera/AmpliSeq" = "CTGTCTCTTATACACATCT",
 "Illumina TruSeq R1/miRNA" = "AGATCGGAAGAGCACACGTCTGAACTCCAGTCA",
 "Illumina TruSeq R2" = "AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGT",
 "Illumina Small RNA R2" = "GATCGTCGGACTGTAGAACTCTGAACGTGTAGA",
 "Illumina Single End Adapter 1"= "GATCGGAAGAGCTCGTATGCCGTCTTCTGCTTG",
 "Illumina Paired End Adapter 2"= "GATCGGAAGAGCGGTTCAGCAGGAATGCCGAG",

 # MGI/BGI adapters
 # http://seqanswers.com/forums/showthread.php?t=87647 (2nd post)
 "BGI Forward" = "AAGTCGGAGGCCAAGCGGTCTTAGGAAGACAA",
 "BGI Reverse" = "AAGTCGGATCGTAGCCATGTCGTTCTGTGAGCCAAGGAGTTG",

 # QIASeq
 "QIASeq miRNA" = "AACTGTAGGCACCATCAAT",

 # ABI SOLiD
 "ABI SOLiD3 Adapter A" = "CTGCCCCGGGTTCCTCATTCTCTCAGCAGCATG",
 "ABI SOLiD3 Adapter B" = "CCACTACGCCTCCGCTTTCCTCTCTATGGGCAGTCGGTGAT",

 # Clontech
 "Clontech Universal Primer Mix"= "CTAATACGACTCACTATAGGGCAAGCAGTGGTATCAACGCAGAGT",
 }

More adapters may be found in the fastp source

Mon, 01 Jan 0001 00:00:00 +0000

Segments #

Modern sequencers, particularly Illumina sequencers, can read multiple times from one (amplified) DNA molecule, producing multiple ‘segments’ (often called ‘reads’) that together form a ‘molecule’ or ‘fragment’.

Definition and Configuration #

Segments are defined in the [input] section of your TOML configuration. Each segment corresponds to one FASTQ file (or stream in interleaved formats), and segment names are arbitrary but should be meaningful.

[input]
 read1 = ["sample_R1.fq.gz"]
 read2 = ["sample_R2.fq.gz"]
 index1 = ["sample_I1.fq.gz"]

In this example, three segments are defined: read1, read2, and index1.

Mon, 01 Jan 0001 00:00:00 +0000

Source #

When a step refers to a ‘source’ (instead of a segment), it means the step can read from multiple types of data: segment sequences, segment names, or tag values.

Overview #

The source parameter generalizes the segment parameter, allowing steps to operate on different kinds of string data within a fragment. This flexibility enables advanced workflows like extracting patterns from read names, processing tag-derived sequences, or combining multiple data sources.

Mon, 01 Jan 0001 00:00:00 +0000

Step #

A step is one coherent manipulation of the FASTQ stream and its associated data.

Overview #

Steps are the building blocks of a processing pipeline. Each step is declared as a [[step]] entry in the TOML configuration file, and the complete pipeline executes steps sequentially from top to bottom.

Every step operates on complete fragments (molecules), ensuring that paired segments remain synchronized. If a filtering step removes a fragment based on criteria from read1, the corresponding read2, index1, and any other segments are automatically removed alongside it.

Mon, 01 Jan 0001 00:00:00 +0000

Tag / Label #

A regular tag is a piece of fragment-derived metadata that one step in the pipeline produces, and other steps may consume, transform, or export.

A virtual tag is an on-the-fly create tag that exists just for this step and disappears right afterwards.

Overview - Regular tags #

Tags enable sophisticated workflows by decoupling data extraction from data usage. Instead of hardcoding logic like “trim adapters AND filter by adapter presence” into a single step, you extract adapter locations as a tag, then use that tag in multiple downstream operations.

Mon, 01 Jan 0001 00:00:00 +0000

Changelog #

v0.9.0 #

General #

Renamed project from mbf-fastq-processor to fastqrab
Much improved error messages pinpointing exactly what needs to change.

New & renamed steps #

New step: ConcatTags — concatenate multiple tags into one
New step: Lowercase — unified replacement for LowercaseTag/LowercaseSequence
New step: SpotCheckReadPairing — Hamming-distance based read pairing validation
New step: ExtractIUPACSuffix added
OtherFile unified: OtherFileByName and OtherFileBySequence merged into one step
ExtractAnchor merged into ExtractRegions
NCount -> NContent to be inline with GCContent
NContent/GCContent now support (and require) relative (counts or rate?)

Step changes #

Conditional Swap/ReverseComplement variants merged into the main steps
if tag support on 8 core editing steps for conditional read editing
ExtractIUPAC: multiple queries in one step, max_mismatches now required, large performance improvements
min_length added to ExtractRegionsOfLowQuality
Quality checking added to Prefix/Postfix
Tag replacement within regular expressions

Output changes #

Tag histogram reports; demultiplex data nested under ‘demultiplex’ key in reports

Performance #

Redesigned multi-core engine: workpool based, better controllable, better documented
Default thread count now uses all available CPU cores
Rapidgzip for parallel gzip decompression, now also for FASTA; auto-detected; included in Nix builds
Arena-based parsers for FASTA and BAM
Parallel BAM decoding
Multicore EvalExpression, ReportCountOligos, ReportLengthDistribution
Prefix/Postfix massively improved performance
Merge base statistics ~80% faster
ConcatTags ~15% faster
IUPAC matching: replaced Sassy with optimized pure-Rust implementation
Optimized SwapConditional, TrimAtTag, StoreTagBackInSequence, FilterReservoirSample, Rename
Dynamic cuckoo filter sizing; initial_filter_capacity documented; read count estimation

Other #

verify command: validates a pipeline produces expected output; auto-detects config, captures stdout/stderr
configuration toml can now be read from stdin (incompatible with reads from stdin).
Shell autocompletion for bash, fish, and zsh
benchmark mode and per-step benchmark harness
template command: shows help on error
LLM configuration guide and template.toml rewrite for LLM-assisted config generation
TagLabel type: all tag names are now strongly typed; duplicate tag names produce a clear error
IndexMap replaces HashMap everywhere for deterministic output order
unwrap() replaced with expect() throughout; clippy::unwrap_used now denied
MSRV pinned to match flake.nix Rust version
Security: upgraded bytes crate (GHSA-434x-w66g-qw3r)
Upgraded dependencies

Documentation #

Four new cookbooks for common FastQ processing tasks
Copy-to-clipboard button in docs
Documentation URLs included in validation failure messages
Added mascot

Bug fixes #

Fixed fastp merge algorithm (replaced with direct port of the reference C++ algorithm)
Fixed invalid FASTQ detection when comment line doesn’t start with ‘+’
Fixed Windows newline detection edge case in parser
Fixed Local-Local FastQElement swap
Fixed demultiplex & fragment count in reports
Fixed Head short-circuit (broken by SpotCheckReadPairing)
Fixed ignore_unaligned → now include_mapped / include_unmapped
Fixed barcode overlapping multiple matches

v0.8.1 #

Github release workflow test

v0.8.0 #

Versioned documentation
First revision where very major feature is in place. Changelog starts here.

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 01: Basic Quality Report #

Use Case #

You have FastQ files from a sequencing run and want to generate comprehensive quality reports to assess:

Read quality scores
Base composition
Read length distribution
Duplicate read counts

This is typically the first step in any sequencing data analysis to understand data quality before downstream processing.

What This Pipeline Does #

Reads input FastQ file(s)
Generates a comprehensive quality report including:
- Base quality statistics
- Base distribution across positions
- Read length distribution
- Duplicate read counting
Outputs reports in both HTML (human-readable) and JSON (machine-readable) formats
Passes through all reads unchanged (no filtering)

Input Files #

input/sample_R1.fq - Forward reads (Read 1) from paired-end sequencing

Output Files #

output_R1.fq - Passed-through reads (identical to input)
output.report_initial.html - HTML quality report
output.report_initial.json - JSON quality report with detailed statistics

When to Use This #

First analysis of new sequencing data
Quality control before committing to expensive downstream analysis
Comparing data quality across different sequencing runs
Identifying potential issues (adapter contamination, quality drop-off, etc.)

Download #

Download 01-basic-quality-report.tar.gz for a complete, runnable example including expected output files.

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 02: UMI Extraction #

Use Case #

You have sequencing data with Unique Molecular Identifiers (UMIs) embedded in the reads. UMIs are short random barcodes added during library preparation that allow you to:

Identify and remove PCR duplicates
Distinguish true biological duplicates from amplification artifacts
Improve accuracy in quantitative analyses (RNA-seq, ATAC-seq, etc.)

What This Pipeline Does #

Reads input FastQ file with UMIs at the start of read1
Extracts the UMI sequence (first 8 bases) and creates a tag
Stores the UMI in the read comment (FASTQ header)
Removes the UMI bases from the read sequence (so they don’t interfere with alignment)
Outputs modified reads with UMI preserved in the header

Input Files #

input/sample_R1.fq - Reads with 8bp UMI at the start

Output Files #

output_R1.fq - Reads with UMI in comment, UMI bases removed from sequence

Configuration Highlights #

[[step]]
 # Extract UMI from positions 0-7 (8 bases)
 action = 'ExtractRegions'
 label = 'umi'
 regions = [{source = 'read1', start = 0, length = 8, anchor="Start"}]

[[step]]
 # Store UMI in the FASTQ comment
 action = 'StoreTagInComment'
 label = 'umi'

[[step]]
 # Remove the UMI bases from the read
 action = 'CutStart'
 target = 'Read1'
 n = 8

Workflow Details #

Before processing:

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 03: Lexogen QuantSeq Processing #

Use Case #

Lexogen QuantSeq is a popular 3’ mRNA sequencing protocol optimized for gene expression profiling. The library structure includes:

First 8 bases: UMI (Unique Molecular Identifier) for deduplication
Next 6 bases: Random hexamer primer sequence (needs removal)
Remaining sequence: Actual cDNA from the 3’ end of transcripts

This cookbook demonstrates the standard preprocessing for QuantSeq data before alignment.

What This Pipeline Does #

Extracts the 8bp UMI from the start of reads
Stores the UMI in the read comment (FASTQ header)
Removes the first 14 bases total (8bp UMI + 6bp random hexamer)
Outputs processed reads ready for alignment

Input Files #

input/quantseq_sample.fq - Raw QuantSeq reads with UMI and random hexamer

Output Files #

output_read1.fq - Processed reads with:
- UMI stored in comment
- First 14bp removed
- Ready for alignment to reference genome

Workflow Details #

Raw read structure:

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 04: PhiX Removal #

Use Case #

You have Illumina PhiX spike-in sequences in your dataset and want to remove those contaminating reads before downstream analysis. PhiX is commonly added as a control to increase base diversity during sequencing runs.

What This Pipeline Does #

This cookbook demonstrates how to identify and remove PhiX contamination using k-mer counting:

Count k-mers: Uses CalcKmers to count how many 30-mers from each read match the PhiX genome
Export data: Saves k-mer counts to a TSV table for analysis
Filter reads: Removes reads with high PhiX k-mer counts (≥25 matching k-mers)

Understanding the Approach #

K-mer Counting #

The CalcKmers step counts how many k-mers (short subsequences of length k) from each read are present in the PhiX reference genome:

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 05: Quality Filtering #

Use Case #

You have sequencing data with varying quality and want to remove low-quality reads before downstream analysis. Poor quality reads can introduce errors in variant calling, assembly, and other analyses.

What This Pipeline Does #

This cookbook demonstrates quality-based filtering using expected error calculation:

Calculate Expected Errors: Uses CalcExpectedError to compute the expected number of base call errors per read based on quality scores
Filter Low-Quality Reads: Uses FilterByNumericTag to remove reads exceeding an error threshold
Generate Reports: Creates quality reports before and after filtering to show improvement

Understanding Expected Error #

Expected error (EE) is a more nuanced quality metric than average quality score:

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 06: Adapter Trimming with PolyA Tail Removal #

Use Case #

You have RNA-seq data that contains:

PolyA tails: Stretches of A bases at the 3’ end (or polyT at 5’ for reverse strand)
Sequencing adapters: Illumina or other adapter sequences that need removal before alignment

These artifacts can interfere with alignment and downstream analysis if not removed.

What This Pipeline Does #

This cookbook demonstrates a complete adapter and polyA trimming workflow:

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 07: Demultiplexing by Inline Barcode #

Use Case #

You have pooled sequencing data from multiple samples that were tagged with unique barcode sequences during library preparation and have not been demuliplexed by your sequencing facility.

You need to:

Extract the barcode(s) from each read
Correct sequencing errors in barcodes
Separate reads into individual files per sample

This is common in multiplexed sequencing runs to maximize sequencing efficiency and reduce costs.

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 08: Read Length Filtering and Truncation #

Use Case #

You have sequencing data with variable read lengths and need to:

Remove reads that are too short (may align poorly or represent artifacts)
Remove reads that are too long (may indicate technical issues)
Truncate all reads to a uniform length (required by some downstream tools)

Read length filtering is important for:

Quality control after adapter trimming
Preparing data for tools that require uniform read lengths
Removing degraded or artifactual sequences

What This Pipeline Does #

This cookbook demonstrates comprehensive read length management:

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 09: Fastp-Equivalent Workflow #

Use Case #

You want to replicate the default behavior of fastp — a popular all-in-one FASTQ preprocessor — using a configurable pipeline. This is useful when you need reproducible, step-by-step control over each filtering stage, or want to extend the workflow beyond what fastp offers.

What This Pipeline Does #

This cookbook replicates fastp’s default single-end processing pipeline:

PolyG Trimming: Uses ExtractPolyTail + TrimAtTag to remove polyG tails (Illumina NextSeq/NovaSeq artifact)
Adapter Trimming: Uses ExtractIUPAC + TrimAtTag to remove the Illumina TruSeq R1 adapter
N-base Filtering: Uses CalcNCount + FilterByNumericTag to remove reads with too many ambiguous bases (--n_base_limit 5)
Quality Filtering: Uses CalcQualifiedBases + FilterByNumericTag to remove reads with too many low-quality bases (--qualified_quality_phred 15, --unqualified_percent_limit 40)
Length Filtering: Uses CalcLength + FilterByNumericTag to remove reads shorter than 15bp (--length_required 15)

Fastp Defaults Replicated #

Fastp parameter	Value	Pipeline step
`--poly_g_min_len`	10	`ExtractPolyTail min_length = 10`
`--adapter_sequence`	`AGATCGGAAGAGCACACGTCTGAACTCCAGTCA`	`ExtractIUPAC query = ...`
`--n_base_limit`	5	`FilterByNumericTag max_value = 6`
`--qualified_quality_phred`	15 (Phred) → 48 (ASCII)	`CalcQualifiedBases threshold = 48`
`--unqualified_percent_limit`	40% → keep if ≥ 60% qualified	`FilterByNumericTag min_value = 0.60`
`--length_required`	15	`FilterByNumericTag min_value = 15`

Note on quality scores: Quality values in FASTQ files are ASCII-encoded. Phred Q15 corresponds to ASCII character 48 (15 + 33 = 48).

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 10: Adapter Identification #

Use Case #

You have a FASTQ file and want to identify which sequencing adapter is present before trimming — or to confirm no adapter contamination remains after trimming. This is useful when the adapter type is unknown, when working with data from multiple library prep kits, or when validating a trimming step.

What This Pipeline Does #

Runs a single Report step that counts exact occurrences of each common adapter sequence in every read (count_oligos)
Writes an HTML and JSON report — no reads are filtered or written to disk

How count_oligos Works #

count_oligos performs exact, full-sequence matching across every read. A read is counted if the probe sequence appears verbatim anywhere within it. There are no mismatches and no IUPAC wildcards. A non-zero count means reads carry at least one complete copy of that adapter.

Mon, 01 Jan 0001 00:00:00 +0000

FilterReservoirSample #

[[step]]
 action = "FilterReservoirSample"
 n = 10_000
 seed = 59014

Filter for a fixed number of reads based on reservoir sampling, that is all input reads have an equal probability of being selected.

This means we need to keep n reads in memory, and they get processed as one large block at the end by all downstream steps.

That means it’s not the right tool if you want to sample to millions of reads, but it is the right tool if you want to have lowish fixed number of reads.

Mon, 01 Jan 0001 00:00:00 +0000

FilterSample #

[[step]]
 action = "FilterSample"
 p = 0.5 # float, the chance for any given read to be kept
 # 0..1
 seed = 42 # (optional) random seed for reproducibility

Randomly sample a percentage of reads. Requires a random seed to ensure reproducibility.

Mon, 01 Jan 0001 00:00:00 +0000

ConvertQuality #

[[step]]
 action = "ConvertQuality"
 from = "Illumina1.8"# Illumin1.8|Illumina1.3|Sanger|Solexa"
 to = "Solexa" # same range as from. Illumina1.8 is an alias for Sanger

Convert quality scores between various encodings / meanings.

See https://en.wikipedia.org/wiki/Phred_quality_score

Will error if from == to.

This step introduces a ValidateQuality step automatically before it.

Corresponding options in other software #

trimmomatic TOPHRED33

Mon, 01 Jan 0001 00:00:00 +0000

CutEnd #

[[step]]
 action = "CutEnd"
 n = 5 # positive integer, cut n nucleotides from the end of the read
 segment = "read1" # Any of your input segments (default: read1)
 if_tag = "mytag"

Cut nucleotides from the end of the read.

May produce empty reads; filter those with FilterEmpty.

Optionally only applies if a tag is truthy.

Mon, 01 Jan 0001 00:00:00 +0000

CutStart #

[[step]]
 action = "CutStart"
 n = 5 # positive integer, cut n nucleotides from the start of the read
 segment = "read1" # Any of your input segments
 if_tag = "mytag"

Cut nucleotides from the start of the read.

May produce empty reads; filter those with FilterEmpty.

Optionally only applies if a tag is truthy.

Mon, 01 Jan 0001 00:00:00 +0000

[[step]]
 action = "ExtractRegions"
 regions = [
 {segment= "Read1", start = 0, length = 8},
 {segment= "Read1", start = 12, length = 4},
 ]
 out_label = "umi"
 region_separator = "_" # (optional) str, what to put between the regions, defaults to '_'

[[step]]
 action = "StoreTagInComment" 
 in_label = "umi"

Extract a sequence from the read and place it in the read name’s comment section, so a (space separated) ‘key=value’ pair is added to the read name.

Supports multiple region-extraction.

See the tag section for more tag generation options.

Mon, 01 Jan 0001 00:00:00 +0000

Head #

[[step]]
 action = "Head"
 n = 1000 # positive integer, number of reads to keep

Output just the first n molecules.

Demultiplex interaction #

If present after a demultiplex step, includes n molecules in each stream.

Mon, 01 Jan 0001 00:00:00 +0000

weight: 150 #

Lowercase #

Convert sequences, tags, or read names to lowercase.

[[step]]
 action = "Lowercase"
 target = "read1" # Any input segment, 'All', 'tag:mytag', or 'name:read1'
 #if_tag = "mytag" # Optional: only apply if tag is truthy

Target Options #

Segment: "read1", "read2", "index1", "index2", or "All" - lowercase’s sequence
Tag: "tag:mytag" - lowercase’s tag’s sequence content (Location-type tags only)
Name: "name:read1" - lowercase’s read name (not including comments)

Optionally only applies if a tag is truthy via if_tag.

Mon, 01 Jan 0001 00:00:00 +0000

MergeReads #

[[step]]
 action = "MergeReads"

 segment1 = "read1" # First segment
 segment2 = "read2" # Second segment
 reverse_complement_segment2 = true # Whether to reverse complement segment2 (suggested: true)

 algorithm = "FastpSeemsWeird" # Algorithm: "fastp_seems_weird". Further algorithms are in planning
 min_overlap = 30 # Minimum overlap length required
 max_mismatch_rate = 0.2 # Maximum allowed mismatch rate (0.0-1.0) (suggested: 0.2)
 max_mismatch_count = 5 # Maximum allowed absolute mismatches (suggested: 5)
 no_overlap_strategy = "as_is" # What to do when no overlap found: "as_is" or "concatenate"
 concatenate_spacer = "NNNN" # (optional) Required if no_overlap_strategy = "concatenate". Spacer sequence to insert between reads
 spacer_quality_char = 33 # (optional) Quality score for spacer bases (suggested: 33 = Phred quality 0)

 # out_label = "merged" # (optional) output Tag label for boolean merge status

Merges paired-end reads from two segments by detecting their overlap and resolving mismatches.

Mon, 01 Jan 0001 00:00:00 +0000

Rename #

[[step]]
 action = "Rename"
 search = '(.)/([1/2])$'
 replacement = '$1 $2'

Apply a regular expression based renaming to the reads.

It is always applied to all available segments (read1, read2, index1, index2).

The example above fixes old school MGI reads for downstream processing, like fastp’s ‘–fix_mgi’ option

You can use the full power of the rust regex crate here.

Read index placeholder #

After the regex replacement runs, the special literal {{READ_INDEX}} is expanded to the running 1-based index of each logical read. When multiple segments are present (for example read1/read2 pairs), every segment for the same read receives the same index so pairs stay aligned. This makes it easy to re-sequence identifiers:

Mon, 01 Jan 0001 00:00:00 +0000

ReverseComplement #

[[step]]
 action = "ReverseComplement"
 segment = "read1" # Any of your input segments (default: read1)
 if_tag = "mytag"

Reverse-complements the read sequence (and reverses the quality).

This supports IUPAC codes (U is complemented to A, so it’s not strictly reversible). Unknown letters are output verbatim.

Useful to combine with Swap.

Optionally only swaps if a tag is truthy.

Mon, 01 Jan 0001 00:00:00 +0000

Skip #

[[step]]
 action = "Skip"
 n = 1000 # positive integer, number of reads to skip

Skips the first n molecules.

Demultiplex interaction #

If present after a demultiplex step, skips the first n molecules in that stream.

Mon, 01 Jan 0001 00:00:00 +0000

Swap #

[[step]]
 action = "Swap"
 segment_a = "read1"
 segment_b = "read2"
 if_tag = "mytag"

Swaps exactly two segments.

Arguments segment_a/segment_b are only necessary if there are more than two segments defined in the input.

Optionally only applies if a tag is truthy.

Mon, 01 Jan 0001 00:00:00 +0000

MaxLen #

[[step]]
 action = "Truncate"
 n = 100 # the maximum length of the read. Cut at end if longer
 segment = "read1" # Any of your input segments (default: read1)
 if_tag = "mytag"

Cut the read down to `n’ bases.

Optionally only applies if a tag is truthy.

Mon, 01 Jan 0001 00:00:00 +0000

weight: 150 #

Uppercase #

Convert sequences, tags, or read names to uppercase.

[[step]]
 action = "Uppercase"
 target = "read1" # Any input segment, 'All', 'tag:mytag', or 'name:read1'
 #if_tag = "mytag" # Optional: only apply if tag is truthy

Target Options #

Segment: "read1", "read2", "index1", "index2", or "All" - lowercase’s sequence
Tag: "tag:mytag" - lowercase’s tag’s sequence content (Location-type tags only)
Name: "name:read1" - lowercase’s read name (not including comments)

Optionally only applies if a tag is truthy via if_tag.

Mon, 01 Jan 0001 00:00:00 +0000

Out of scope #

Things fastqrab will explicitly not do and that won’t be implemented.

Anything based on averaging phred scores #

Based on the average quality in a sliding window. Arithmetic averaging of phred scores is wrong.

see ExtractMeanQuality

Corresponding options in other software #

Trimmomatic SLIDINGWINDOW
fastp –cut_front
fastp –cut_tail
fastp –cut_right

Fast5 #

https://medium.com/@shiansu/a-look-at-the-nanopore-fast5-format-f711999e2ff6 Oxford Nanopore squiggle data. Apparently no formal spec.

kallisto BUS format #

- a brief barcode/umi format for single cell RNA-seq
- needs an 'equivalance class' - i.e. at least pseudo alignment
- weird length restrictions on barcodes and umis (1(!)-32), 
 but stores the length in an uint32...

Alignment #

While it’s tempting to leverage the fastq parsing for an aligner, aligning molecules to references is out of scope for the 1.0 target.

Mon, 01 Jan 0001 00:00:00 +0000

Inspect #

Dump a few reads to a FASTQ file for inspection at this point in the graph.

[[step]]
 action = "Inspect"
 n = 1000 # how many molecules 
 infix = "inspect_at_point" # output filename infix
 segment = "read1" # Any of your input segments (use "all" for interleaved output)
 format = "FASTQ" # output format: FASTQ, FASTA (no BAM)
 suffix = "compressed" # (optional) custom suffix for filename
 compression = "gzip" # (optional) compression format: raw, gzip, zstd. Defaults to uncompressed
 compression_level = 1 # (optional) compression level for gzip/zstd/bam (gzip, zstd: 1-22)
 # defaults: gzip=6, zstd=5

Output filename pattern:

Mon, 01 Jan 0001 00:00:00 +0000

Progress #

Emit progress to stdout (default) or a .progress log file, if output_infix is set. (filename is {output_prefix}{ix_separator}{infix}.progress, default separator _).

[[step]]
 action = "Progress"
 n = 100_000
 output_infix = "progress" # optional

Every n reads, report on total progress, total reads per second. At the end, report final runtime and reads/second.

Mon, 01 Jan 0001 00:00:00 +0000

CalcComplexity #

[[step]]
 action = "CalcComplexity"
 out_label = "complexity"
 segment = "read1" # Any of your input segments, or 'All'

Calculate read complexity, based on the percentage of bases that are changed from their predecessor. Ranges from 0.0 to 1.0.

A good filter value might be 0.30, which means 30% complexity is required. See FilterByNumericTag.

Corresponding options in other software #

fastp: -low_complexity_filter

Mon, 01 Jan 0001 00:00:00 +0000

CalcMeanQuality #

We don’t support calculating the ‘average quality’.

This is typically a bad idea, see https://www.drive5.com/usearch/manual/avgq.html for a discussion of the issues.

To illustrate, 140 x Q35 + 10 x Q2 reads have an ‘average’ phred of 33, but 6.4 expected wrong bases. A read with 150 x Q25 has a much worse ‘average’ phred of 25, but a much lower expected number of errors at 0.5.

Corresponding options in other software #

Trimmomatic: AVGQUAL:
fastp: –average_qual

Mon, 01 Jan 0001 00:00:00 +0000

CalcQualifiedBases #

[[step]]
 action = "CalcQualifiedBases"
 threshold = 'C' # the quality value >= which a base is qualified 
 # In your phred encoding. Typically 33..75
 # a byte or a number 0...255
 op = 'worse' # see below.
 segment = "read1" # Any of your input segments, or 'All'
 out_label = "tag_name"
 relative = true # a rate (true) or a count (false)

Calculate the number of bases that are ‘qualified’, that is abov/below a user defined threshold.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLongestPolyX #

Find the longest homopolymer stretch anywhere in the read (unlike ExtractPolyTail, which only considers suffixes).

[[step]]
 action = "ExtractLongestPolyX"
 out_label = "my_tag"
 segment = "read1"
 min_length = 5
 base = '.' # search for any homopolymer (A/C/G/T/N)
 max_mismatch_rate = 0.15
 max_consecutive_mismatches = 2

base accepts a concrete nucleotide (A, C, G, T, N) or . to search all of ACGT and report the longest hit.
max_mismatch_rate and max_consecutive_mismatches mirror ExtractPolyTail; they control how permissive the run detection is.
When no run satisfies min_length, the tag is reported as missing.
only one run is reported, even if multiple runs of the same length exist; in this case, the first run found is reported.

Mon, 01 Jan 0001 00:00:00 +0000

TrimQualityEnd #

[[step]]
 action = "ExtractLowQualityEnd"
 out_label = "low_quality_ends"
 min_qual = 20 # u8, minimum quality to keep (in whatever your score is encoded in)
 # either a char like 'A' or a number 0..128 (typical phred score is 33..75)
 segment = "read1" # Any of your input segments

Define a region of low quality bases at the end of reads.

Corresponding options in other software #

Trimmomatic: TRAILING (if paired with TrimAtTag)

Mon, 01 Jan 0001 00:00:00 +0000

TrimQualityStart #

[[step]]
 action = "ExtractLowQualityStart"
 min_qual = 20 # u8, minimum quality to keep (in whatever your score is encoded in)
 # either a char like 'A' or a number 0..128 (typical phred score is 33..75)
 segment = "read1" # Any of your input segments
 out_label = "bad_starts"

Define a region with low quality bases (below threshold) at steart of read.

Corresponding options in other software #

Trimmomatic: LEADING (if combined with TrimAtTag)

Mon, 01 Jan 0001 00:00:00 +0000

ExtractPolyTail #

[[step]]
 action = "ExtractPolyTail"
 out_label = "tag_label"
 segment = "read1" # Any of your input segments (default: read1)
 min_length = 5 # positive integer, the minimum number of repeats of the base
 base = "A" # one of AGTCN., the 'base' to trim (or . for any repeated base)
 max_mismatch_rate = 0.1 # float 0.0..=1.0, how many mismatches are allowed in the repeat
 max_consecutive_mismatches = 3 # how many consecutive mismatches are allowed

Identify either a specific letter (AGTC or N) repetition, or any base repetition (base = ‘.’) at the end of the read.

Mon, 01 Jan 0001 00:00:00 +0000

FilterDuplicates #

[[step]]
 action = "TagDuplicates"
 false_positive_rate = 0.00001 #
 # the false positive rate of the filter.
 # 0..1
 seed = 59 # required!
 source = "All" # Any input segment, 'All', 'tag:<tag-name>' or 'name:<segment>'
 # split_character = "/" # required (and accepted only iff using name:<segment>
 out_label = "dups"
 # initial_filter_capacity = 10_000_000 # optional. Auto detected by default

[[step]]
 action = "FilterByTag"
 in_label = "dups"
 keep_or_remove = "Remove" # Keep|Remove

Tag duplicates (2nd onwards) from the stream using a Cuckoo filter.

Mon, 01 Jan 0001 00:00:00 +0000

AssignToReference #

Assign each query sequence to the closest entry in barcodes section, using Hamming distance.

( As opposed to HammingCorrect which will correct to the closest barcode sequence).

At start-up the step builds an efficient Hamming-distance index over the database. For every read, the tag supplied in in_label is looked up in the index and the name of the closest matching reference entry is written to out_label as a string tag.

Mon, 01 Jan 0001 00:00:00 +0000

HammingCorrect #

Correct a tag to one of a predefined set of ‘barcodes’ using closest hamming distance.

[[step]]
 action = "HammingCorrect"
 in_label = "mytag"
 out_label = "my_corrected_tag"
 barcodes = "mybarcodelist"
 max_hamming_distance = 1
 on_no_match = 'remove' # 'remove', 'empty', 'keep'
 name_split_character = '|' # optional

[barcodes.mybarcodelist]
 "AAAA" = "label_ignored" # only read when demultiplexing

on_no_match controls what happens if the tag cannot be corrected within the max_hamming_distance:

remove: Remove the hit (location and sequence), useful for FilterByTag later.
keep: Keep the original tag (and location)
empty: Keep the original location, but set the tag to empty.

Note that hamming_correction removes the location information on tags if they spanned more than one region. (This is an implementation limitation, not a conceptual one).

Mon, 01 Jan 0001 00:00:00 +0000

ValidateAllReadsSameLength #

[[step]]
 action = "ValidateAllReadsSameLength"
 source = "read1" # Any segment, All, tag:<name> or 'name:segment>'

Validates that all reads have the same sequence/tag/name length.

Useful when you want to verify read length consistency in your pipeline.

(For names, the names without comments - that is up to the first input.options.read_comment_character are used).

Mon, 01 Jan 0001 00:00:00 +0000

ValidateName #

Verify that all segments have the same read name (or a shared prefix).

[[step]]
 action = "ValidateName"
 # Optional separator character; the comparison stops at the first match
 readname_end_char = "_" # Optional. Do not set for exact matching. Otherwise, a byte character
 sample_stride = 1000 # Check every nth fragment, default 1000. Must be > 0. Starts with first read

When no separator character (readname_end_char) is provided the entire name must match exactly across all segments.

Mon, 01 Jan 0001 00:00:00 +0000

ValidateQuality #

Validate that all scores are between 33..=41

[[step]]
 action = "ValidateQuality"
 segment = "read1" # Any of your input segments, or 'All'
 encoding = 'Illumina1.8' # 'Illumina1.8|Illumina1.3|Sanger|Solexa'
 # Illumina1.8 is an alias for Sanger.

The encoding defines the accepted range of values.

If you want to convert quality codes, use ConvertQuality.

See https://pmc.ncbi.nlm.nih.gov/articles/PMC2847217/ , table 1

Mon, 01 Jan 0001 00:00:00 +0000

ValidateReadNamesPrintable #

Validate that every read name conforms to the SAM/BAM specification.

The SAM specification requires that query names (QNAME) match [!-?A-~]{1,254}: printable ASCII characters excluding @ and space, with a maximum length of 254.

[[step]]
 action = "ValidateReadNamesPrintable"

No additional parameters are needed — the allowed character set is fixed by the SAM spec.

When to use #

Add this step when your pipeline writes non-BAM output that will become BAM eventually and you suspect read names may contain invalid characters.

Mon, 01 Jan 0001 00:00:00 +0000

ValidateReadPairing #

Confirms for every sample_strideth read ‘pair’ that the names are identical but for one letter.

[[step]]
 action = "ValidateReadPairing"
 sample_stride = 1000 # Check every nth fragment, default 1000. Must be > 0. Starts with first read

Ensures

read names between segments have the same length
read names between segments have a hamming distance of at most one.

Note that this validation requires at least two input segments.

(See also: ValidateName, which validates after truncating on a character occurance).

Mon, 01 Jan 0001 00:00:00 +0000

ValidateSeq #

Validate that only allowed characters are in the sequence.

[[step]]
 action = "ValidateSeq"
 allowed = "AGTC" # String. Example 'ACGTN', the allowed characters
 segment = "read1" # Any of your input segments, or 'All'

(Older) Documentation Versions

Mon, 01 Jan 0001 00:00:00 +0000

Available builds:

main (this build)
v0.8.1
v0.8.0-test
v0.8.0

adapters

Mon, 01 Jan 0001 00:00:00 +0000

AssignToReference

Mon, 01 Jan 0001 00:00:00 +0000

barcodes

Mon, 01 Jan 0001 00:00:00 +0000

benchmark-section

Mon, 01 Jan 0001 00:00:00 +0000

Calc Base Content

Mon, 01 Jan 0001 00:00:00 +0000

CalcBaseContent #

[[step]]
 action = "CalcBaseContent"
 segment = "read1" # Any of your input segments, or 'All'
 out_label = "at_content"
 bases_to_count = "AT"
 bases_to_ignore = "N"
 relative = true # default

Counts the rate (0..=1) of bases that match bases_to_count, while removing any bases listed in bases_to_ignore from the denominator (if relative = True).

Both lists are case-insensitive, and accept only ascii letters. When no bases remain after filtering, the step returns 0.

Set relative = false to emit absolute base counts instead of a rate. Absolute mode requires bases_to_ignore to remain unset, otherwise the configuration check fails.

Calc GC Content

Mon, 01 Jan 0001 00:00:00 +0000

CalcGCContent #

[[step]]
 action = "CalcGCContent"
 segment = "read1" # Any of your input segments, or 'All'
 out_label = "gc"
 relative = true # a rate (true) or a count (false)

Count what percentage of bases are GC (as opposed to AT). Non-AGTC bases (e.g. N) are ignored.

Output is 0..100.

Wrapper around CalcBaseContent with bases = "GC", ignore="N", relative=true).

CalcBaseContent

Mon, 01 Jan 0001 00:00:00 +0000

CalcComplexity

Mon, 01 Jan 0001 00:00:00 +0000

CalcExpectedError

Mon, 01 Jan 0001 00:00:00 +0000

CalcGCContent

Mon, 01 Jan 0001 00:00:00 +0000

CalcKmers

Mon, 01 Jan 0001 00:00:00 +0000

CalcLength

Mon, 01 Jan 0001 00:00:00 +0000

CalcMeanQuality

Mon, 01 Jan 0001 00:00:00 +0000

CalcNContent

Mon, 01 Jan 0001 00:00:00 +0000

CalcNContent

Mon, 01 Jan 0001 00:00:00 +0000

CalcNCount #

[[step]]
 action = "CalcNContent"
 segment = "read1" # Any of your input segments, or 'All'
 out_label = "ncount"
 relative = true # a rate (true) or a count (false)

Count how many N are present in the read.

This step is a convenient wrapper for CalcBaseContent with bases_to_count = "N".

Corresponding options in other software #

fastp: –n_base_limit (if combined with FilterByNumericTag)

CalcQualifiedBases

Mon, 01 Jan 0001 00:00:00 +0000

CLI

Mon, 01 Jan 0001 00:00:00 +0000

ConcatTags

Mon, 01 Jan 0001 00:00:00 +0000

Convert To Rate

Mon, 01 Jan 0001 00:00:00 +0000

ConvertToRate #

[[step]]
 action = "CalcBaseContent"
 segment = "read1"
 bases_to_count = "A"
 relative = false
 out_label = "a_count" # absolute A-base count

[[step]]
 action = "ConvertToRate"
 in_label = "a_count" # The numeric tag to divide by read length
 out_label = "a_rate" # Output tag label
 segment = "read1" # Segment to measure length from, or 'All' for total length (default: only segment if single-read input)

Divide an existing numeric tag by the read length to produce a normalized rate.

Typical use case: Divide EvalExpression by the read length to get a fraction (rate).

ConvertQuality

Mon, 01 Jan 0001 00:00:00 +0000

ConvertRegionsToLength

Mon, 01 Jan 0001 00:00:00 +0000

ConvertToRate

Mon, 01 Jan 0001 00:00:00 +0000

CutEnd

Mon, 01 Jan 0001 00:00:00 +0000

CutStart

Mon, 01 Jan 0001 00:00:00 +0000

Demultiplex

Mon, 01 Jan 0001 00:00:00 +0000

EvalExpression

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPAC

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACSuffix

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACWithIndel

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLongestPolyX

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLowQualityEnd

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLowQualityStart

Mon, 01 Jan 0001 00:00:00 +0000

ExtractPolyTail

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegex

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegion

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegions

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegionsOfLowQuality

Mon, 01 Jan 0001 00:00:00 +0000

ExtractToName

Mon, 01 Jan 0001 00:00:00 +0000

FilterByNumericTag

Mon, 01 Jan 0001 00:00:00 +0000

FilterByTag

Mon, 01 Jan 0001 00:00:00 +0000

FilterEmpty

Mon, 01 Jan 0001 00:00:00 +0000

FilterReservoirSample

Mon, 01 Jan 0001 00:00:00 +0000

FilterSample

Mon, 01 Jan 0001 00:00:00 +0000

ForgetAllTags

Mon, 01 Jan 0001 00:00:00 +0000

ForgetTag

Mon, 01 Jan 0001 00:00:00 +0000

HammingCorrect

Mon, 01 Jan 0001 00:00:00 +0000

Head

Mon, 01 Jan 0001 00:00:00 +0000

input-section

Mon, 01 Jan 0001 00:00:00 +0000

Inspect

Mon, 01 Jan 0001 00:00:00 +0000

llm-guide

Mon, 01 Jan 0001 00:00:00 +0000

Lowercase

Mon, 01 Jan 0001 00:00:00 +0000

MergeReads

Mon, 01 Jan 0001 00:00:00 +0000

Options

Mon, 01 Jan 0001 00:00:00 +0000

Out_Of_Scope

Mon, 01 Jan 0001 00:00:00 +0000

output-section

Mon, 01 Jan 0001 00:00:00 +0000

Postfix

Mon, 01 Jan 0001 00:00:00 +0000

Prefix

Mon, 01 Jan 0001 00:00:00 +0000

Progress

Mon, 01 Jan 0001 00:00:00 +0000

QuantifyTag

Mon, 01 Jan 0001 00:00:00 +0000

Rename

Mon, 01 Jan 0001 00:00:00 +0000

ReplaceTagWithLetter

Mon, 01 Jan 0001 00:00:00 +0000

Report

Mon, 01 Jan 0001 00:00:00 +0000

ReverseComplement

Mon, 01 Jan 0001 00:00:00 +0000

Skip

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagBackInSequence

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInComment

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInFastQ

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInSequence

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagsInTable

Mon, 01 Jan 0001 00:00:00 +0000

Swap

Mon, 01 Jan 0001 00:00:00 +0000

TagDuplicates

Mon, 01 Jan 0001 00:00:00 +0000

TagOtherFile

Mon, 01 Jan 0001 00:00:00 +0000

threading

Mon, 01 Jan 0001 00:00:00 +0000

TrimAtTag

Mon, 01 Jan 0001 00:00:00 +0000

Truncate

Mon, 01 Jan 0001 00:00:00 +0000

Uppercase

Mon, 01 Jan 0001 00:00:00 +0000

ValidateAllReadsSameLength

Mon, 01 Jan 0001 00:00:00 +0000

ValidateName

Mon, 01 Jan 0001 00:00:00 +0000

ValidateQuality

Mon, 01 Jan 0001 00:00:00 +0000

ValidateReadNamesPrintable

Mon, 01 Jan 0001 00:00:00 +0000

ValidateReadPairing

Mon, 01 Jan 0001 00:00:00 +0000

ValidateSeq

Mon, 01 Jan 0001 00:00:00 +0000

Introduction on fastqrab documentation

Philosophy #

Correctness #

Reproducibility #

Getting started with development #

Development environment #

Troubleshooting #

I don’t know what to do after an error message #

I received a friendly panic message #

LLM Configuration Guide

LLM Configuration Generation Guide #

Configuration Structure #

Quick Start Patterns #

Pattern 1: Basic Quality Report #

Implementing your own transformation #

Command line interface #

Usage #

Process #

Input section #

Output section #

FilterByTag #

Report #

Barcodes section

[barcodes.*] section #

Key format #

Demultiplexed output #

Based on barcodes #

Based on boolean tags #

FilterByNumericTag #

Parser Architecture

Parser Architecture #

Overview #

The Zero-Copy Challenge with Compressed Files #

Why Not Pure Zero-Copy? #

Parser Architecture

Parser Architecture #

Overview #

The Zero-Copy Challenge with Compressed Files #

Why Not Pure Zero-Copy? #

Code Coverage #

Running coverage #

Other formats #

FilterEmpty #

Postfix #

Prefix #

ReplaceTagWithLetter #

Parameters #

Example Use Cases #

TrimAtTag #

QuantifyTag #

Demultiplex interaction #

CalcLength #

ExtractRegex #

ExtractRegion #

ExtractRegions #

Basic Usage #

Advanced Usage with Sources #

ExtractRegionsOfLowQuality #

TagOtherFile #

ConcatTags #

Behavior by Tag Type #

Location Tags Only #

ForgetAllTags #

ForgetTag #

StoreTagBackInSequence #

StoreTagInComment #

StoreTagsInTable #

Extract IUPAC

ExtractIUPAC #

Extract IUPAC suffix

ExtractIUPACSuffix #

Extract IUPAC with Indels

ExtractIUPACWithIndel #

Store Tag In FASTQ

StoreTagInFastQ #

CalcExpectedError #

StoreTagInSequence #

Parameters #

How it works #

ConvertRegionsToLength #

`[barcodes.*]` section #