Introduction on mbf-fastq-processor documentation

Mon, 01 Jan 0001 00:00:00 +0000

Command line interface #

mbf-fastq-processor is configured exclusively through a TOML document. The CLI is therefore intentionally minimal and focuses on selecting the configuration and the working directory.

Usage #

mbf-fastq-processor process [config.toml] [--allow-overwrite]
mbf-fastq-processor template
mbf-fastq-processor interactive [config.toml]

Process #

Process FASTQ as described in <config.toml>.(see the TOML format reference). Relative paths are resolved against the current shell directory.

The config.toml argument can be left off iff there’s one .toml in the current directory, and it contains an [input] and an [output] section

Mon, 01 Jan 0001 00:00:00 +0000

Input section #

The [input] table enumerates all read sources that make up a fragment. At least one segment must be declared.

[input]
 read1 = ['fileA_1.fastq', 'fileB_1.fastq.gz', 'fileC_1.fastq.zst'] # required: one or more paths
 read2 = "fileA_2.fastq.gz" # optional
 index1 = ['index1_A.fastq.gz'] # optional
 # interleaved = [...] # optional, see below

Key	Required	Value type	Notes
segment name (e.g. `read1`)	Yes (at least one)	string or array of strings	Each unique key defines a segment; arrays concatenate multiple files in order.
`interleaved`	No	array of strings	Enables interleaved reading; must list segment names in their in-file order.

Additional points:

Mon, 01 Jan 0001 00:00:00 +0000

Output section #

The [output] table controls how transformed reads and reporting artefacts are written.

[output]
 prefix = "output" # required.
 format = "Fastq", # (optional) output format, defaults to 'Fastq'
					 # Valid values are: Fastq, Fasta, BAM and None (for no sequence output)
 compression = "Gzip" # Raw | Uncompressed | Gzip | Zstd | None (default: Raw)
 suffix = ".fq.gz" # optional override; inferred from format when omitted
 compression_level = 6 # gzip: 0-9, zstd: 1-22, bam: 0-9 (BGZF); defaults are gzip=6, zstd=5
 ix_separator = "_" # optional separator between prefix, infixes, and segments. Defaults to '_'

 report_json = false # write prefix.json
 report_html = true # write prefix.html

 output = ["read1", "read2"] # limit which segments become FASTQ files
 interleave = false # emit a single interleaved FASTQ
 stdout = false # stream to stdout instead of files
 chunk_size = 100000 # Write multiple, numbered output files, each a maximum of chunk_size reads/molecules.

 output_hash_uncompressed = false
 output_hash_compressed = false

Key	Default	Description
`prefix`	`"output"`	Base name for all files produced by the run.
`format`	`"Fastq"`	Output format. Valid values are: `Fastq`, `Fasta`, `Bam`, and `None` (for no sequence output).
`compression`	`"Uncompressed"`	Compression format for read outputs. Valid values are: `Gzip`, `Zstd`, `Uncompressed` (alias: `"Raw"`). Must not be set for BAM
`suffix`	derived from format	Override file extension when interop with other tooling demands a specific suffix.
`compression_level`	gzip: 6, zstd: 5	Fine-tune compression effort. Ignored for `Raw`/`None`. `Bam` maps directly to the BGZF level (0–9).
`report_json` / `report_html`	`false`	Toggle structured or interactive reports.
`output`	all input segments	Restrict the subset of segments written to disk. Use an empty list to suppress FASTQs while still running steps that depend on fragment data.
`interleave`	`false`	Generate a single interleaved FASTQ (`{prefix}_interleaved.fq*`).
`stdout`	`false`	Write to stdout. Forces `format = "Raw"`. `Sets interleave=true` if more than one fragment is listed in `output`
`output_hash_uncompressed` / `output_hash_compressed`	`false`	Emit SHA-256 checksums.
`ix_separator`	`"_"`	Separator inserted between `prefix`, any infix (demultiplex labels, inspect names, etc.), and segment names.
`chunk_size`	(unlimited)	Split outputs into multiple files, each containing at most `chunk_size` reads/molecules. For non-interleaved output files, it’s `chunk_size` reads, for interleaved files it’s molecules. This means when mixing interleaved and non-interleaved output, you get the same number of files. Files are numbered sequentially, e.g. `output_read1_0.fq.gz`, …, Numbers start at 0 and use the minimum number of (base 10) digits necessary for alphabetical sorting (by renaming already produced files whenever an extension is needed).

Generated filenames join these components with ix_separator (default _), e.g. {prefix}_{segment}{suffix}. Interleaving replaces segment with interleaved; demultiplexing adds per-barcode infixes before the segment. Checksums use .uncompressed.sha256 or .compressed.sha256 suffixes.

Mon, 01 Jan 0001 00:00:00 +0000

FilterByTag #

This transformation filters molecules based on the presence or absence of a specified tag.

Use “Keep” to retain molecules that have the tag, or “Remove” to discard reads that have the tag.

If used on a boolean tag, the boolean value of the tag is used to determine whether to keep or remove the read.

For numeric tags, use FilterByNumericTag.

[[step]]
 action = "FilterByTag"
 in_label = "mytag"
 keep_or_remove = "Keep" # or "Remove"

Mon, 01 Jan 0001 00:00:00 +0000

Report #

Capture data for the final report (see the output section).

You can add multiple reports, at any stage of your transformation chain to get e.g. before/after filtering reports.

Example report

[[step]]
 action = 'Report'
 name = "report" # Key that the report will be listed under. Must be distinct
 count = true # count reads at this position
 base_statistics = false # include base distribution at each read position, q20, q30, total, gc bases
 length_distribution = false # capture read length distribution
 duplicate_count_per_read = false # count duplicates using Cukoo filter on each read1/read2/index1/index2
 duplicate_count_per_fragment = false # count duplicates using Cukoo filter, on concatenated read1/read2/index1/index2
 count_oligos = [] # if set, count these oligos in the segment. Full match only, no iupac
 count_oligos_segment = "all" # segment to count oligos in, can be 'all', 'read1', ...

Statistics available (for each ‘segment’. If demultiplexd, per barcode combination):

Mon, 01 Jan 0001 00:00:00 +0000

Demultiplexed output #

Demultiplex is a magic transformation that forks the output.

You receive one set of output files per barcode (combination) defined.

Transformations downstream are (virtually) duplicated, so you can for example filter to the head reads in each barcode, and get reports for both: all reads and each separate barcode.

Demultiplexing can be done on barcodes, or on boolean tags.

Based on barcodes #

[[step]]
 action = "Demultiplex"
 in_label = "mytag"
 barcodes = "mybarcodes"
 output_unmatched = true # if set, write reads not matching any barcode
 # to a file like ouput_prefix_no-barcode_1.fq

[barcodes.mybarcodes] # can be before and after.
# separate multiple regions with a _
# a Mapping of barcode -> output name.
AAAAAA_CCCCCC = "sample-1" # output files are named prefix{ix_separator}barcode_prefix{ix_separator}segment.suffix
 # with the separator defaulting to '_', e.g. output_sample-1_1.fq.gz
 # or output_sample-1_report.fq.gz

Based on boolean tags #

[[step]]
 segment = "read1"
 action = "TagOtherFileByName"
 out_label = "a_bool_tag"
 filename = "path/to/boolean_tags.tsv"
 false_positive_rate = 0

[[step]]
 action = "Demultiplex"
 in_label = "a_bool_tag"

Note that this does not extract the barcodes from the read (use an extract step, such as ExtractRegion).

Mon, 01 Jan 0001 00:00:00 +0000

FilterByNumericTag #

Remove molecules by thresholding on numeric tag.


[[step]]
 action = "CalcLength"
 out_label = "mytag"
 segment = "read1"

[[step]]
 action = "FilterByNumericTag"
 in_label = "mytag"
 keep_or_remove = "Keep" # or "Remove"
 min_value = 5 # >= this, optional
 max_value = 21 # < this, optional

The example only keeps reads that are between 5 and 20 bases long.

Consider using an EvalExpression for more complicated decisions.

Mon, 01 Jan 0001 00:00:00 +0000

FilterEmpty #

[[step]]
 action = "FilterEmpty"
 segment = "read1" # Any of your input segments, or 'All'

Drop the molecule if the read has length 0. (Use after other processing.)

This get’s expanded to an internal tag and a FilteryByNumeric.

This is necessary if your modification can produce ’empty' reads - downstream aligners like STAR tend to dislike these in their input.

On segment=‘All’, only filters reads that are empty in all parts. Use multiple FilterEmpty steps to filter if any part is empty.

Mon, 01 Jan 0001 00:00:00 +0000

Postfix #

Add DNA to the end of read sequences.

[[step]]
 action = "Postfix"
 seq = "agtc" # DNA sequence to add at end of reads. Checked to be agtcn
 qual = "IIII" # same length as seq. Your responsibility to have valid phred values
 segment = "read1" # Any of your input segments

This transformation adds a specified sequence and corresponding quality scores to the end of reads.

Mon, 01 Jan 0001 00:00:00 +0000

Prefix #

Add text to the beginning of read sequences.

[[step]]
 action = "Prefix"
 seq = "agtTCAa" # DNA sequence to add at beginning of reads. Checked to be agtcn
 qual = "IIIBIII" # same length as seq. Your responsibility to have valid phred values
 segment = "read1" # Any of your input segments

This transformation adds a specified sequence and corresponding quality scores to the beginning of reads.

Mon, 01 Jan 0001 00:00:00 +0000

ReplaceTagWithLetter #

Replace sequence bases in tagged regions with a specified letter.

[[step]]
 action = "ReplaceTagWithLetter"
 in_label = "mytag" # Tag containing regions to replace
 letter = "N" # Replacement character (defaults to 'N')

This transformation replaces all sequence bases within the regions defined by a tag with a specified replacement character. Quality scores are preserved unchanged. This is commonly used to mask low-quality regions as ‘N’ characters.

Parameters #

in_label: Name of the tag containing regions to be replaced
letter: Single character to replace bases with (defaults to ‘N’ if not specified)

Example Use Cases #

Mask low-quality bases identified by ExtractRegionsOfLowQuality
Replace specific sequence motifs identified by other extraction steps
Convert tagged regions to ambiguous bases for downstream analysis

The tag must have been created by a previous extraction step and must contain location information.

Mon, 01 Jan 0001 00:00:00 +0000

TrimAtTag #

Trim the read at the position of a tag.

[[step]]
 action = "TrimAtTag"
 in_label = "mytag"
 direction = "Start" # or "End"
 keep_tag = false # if true, the tag sequence is kept in the read

This transformation trims the read at the position where a tag was found.

The direction parameter determines whether to trim from the start or end of the tag, and keep_tag determines whether the tag sequence itself is retained.

Mon, 01 Jan 0001 00:00:00 +0000

QuantifyTag #

Count the occurrences of each tag-sequence.

[[step]]
 action = "QuantifyTag"
 in_label = "mytag"
 infix = "tagcount" # output file is output{ix_separator}tagcount.qr.json (default '_' → output_tagcount.qr.json)

This transformation counts how many times each unique tag value appears and outputs the results to a JSON file.

Demultiplex interaction #

Barcodes are counted per demultiplexed stream.

Mon, 01 Jan 0001 00:00:00 +0000

CalcLength #

Extract the length of a read as a tag.

[[step]]
 action = "CalcLength"
 out_label = "mytag"
 segment = "read1" # Any of your input segments, or 'All'

This transformation creates a tag containing the length of the specified read.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractAnchor #

Extract regions relative to a previously tagged anchor position.

# First create an anchor tag. Iupac, regex, ExtractRegion, your choice.
[[step]]
 action = "ExtractIUPAC"
 search = "CAYA"
 out_label = "anchor_tag"
 segment = "read1"
 anchor = "Anywhere"
 max_mismatches = 0

# Then extract relative to that anchor
[[step]]
 action = "ExtractAnchor"
 out_label = "mytag"
 in_label = "anchor_tag" # tag that provides the anchor position
 regions = [[-2, 4], [4, 1]] # [start, length] pairs relative to anchor
 region_separator = "_" # (optional) separator between regions

This transformation uses the leftmost position of a previously established tag as the anchor point and extracts specified regions relative to that position.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegex #

Extract a regexp result. Stores an empty string if not found.

[[step]]
 action = "ExtractRegex"
 out_label = "mytag"
 search = "^CT(..)CT"
 replacement = "$1" # standard regex replacement syntax
 source = "read1" # An input segment (to read from sequence), or name:<segment> to read from a tag

This transformation searches for a regular expression pattern in the specified read and extracts the matching portion as a tag.

The value actually ’extracted’ is after replacement has been performed.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegion #

Extract a fixed position region.

[[step]]
 action = "ExtractRegion"
 start = 5
 length = 8
 segment = "read1" # Any of your input segments
 out_label = "umi"

This transformation extracts a fixed-length region from the specified read at a given position and stores it as a tag.

Use ExtractRegions if your region is actually multiple regions (possibly from different segments).

If the read is shorter than requested, the region will be shorter (and might be of varying length).

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegions #

Extract from multiple fixed position regions.

[[step]]
 action = "ExtractRegions"
 regions = [
 {segment = "read1", start = 0, length = 8},
 {segment = "read1", start = 12, length = 4},
 ]
 out_label = "barcode"

This transformation extracts multiple fixed-length regions from reads and concatenates them into a single tag.

ExtractRegions with only one region are exactly equivalent to ExtractRegion.

If the read is shorter than requested, the region will be shorter (and might be of varying length).

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegionsOfLowQuality #

Extract regions (min size 1 bp) where bases have quality scores below threshold.

[[step]]
 action = "ExtractRegionsOfLowQuality"
 segment = "read1" # Any of your input segments
 min_quality = 60 # Quality threshold (Phred+33)
 out_label = "low_quality_regions"

This transformation scans through quality scores of the specified segment and identifies contiguous regions where quality scores are below the specified threshold. Each low-quality region becomes a tagged region with location information (start position and length).

Parameters #

segment: Which read to analyze for low-quality regions
min_quality: Quality score threshold using Phred+33 encoding. See Phred quality score for ASCII character mapping
out_label: Tag name to store the extracted regions

Example #

With min_quality = 60 (ASCII ‘<’), any bases with quality scores below ‘<’ will be identified as low-quality regions. This is useful for masking or filtering poor-quality sequences.

Mon, 01 Jan 0001 00:00:00 +0000

TagOtherFileByName #

Mark reads based on wether names are present in another file.

[[step]]
 action = "TagOtherFileByName"
 segment = "read1" # which segment's name are we using
 out_label = "present_in_other"
 filename = "names.fastq" # Can read fastq (also compressed), or SAM/BAM, or fasta files
 false_positive_rate = 0.01 # false positive rate (0..1)
 seed = 42 # seed for randomness
 ignore_unaligned = false # in case of BAM/SAM, whether to ignore unaligned reads. Mapped reads are always considered
 fastq_readname_end_char = " " # (optional) char (byte value) at which to cut input fastq read names before comparing. If not set, no cutting is done.
 reference_readname_end_char = "/" # (optional) char (byte value) at which to cut reference read names before storing them.

This step marks reads by comparing their names against names from another file.

Mon, 01 Jan 0001 00:00:00 +0000

TagOtherFileBySequence #

Marks reads based on wether sequences are present in another file.

[[step]]
 action = "TagOtherFileBySequence"
 out_label = "present_in_other_file"
 filename = "names.fastq" # Can read fastq (also compressed), or SAM/BAM, or fasta files
 segment = "read1" # Any of your input segments
 false_positive_rate = 0.01 # false positive rate (0..1)
 seed = 42 # seed for randomness
 ignore_unaligned = false # in case of BAM/SAM, whether to ignore unaligned reads. Mapped reads are always considered

This step annotates reads by comparing their sequences against sequences from another file.

Mon, 01 Jan 0001 00:00:00 +0000

ForgetAllTags #

Remove every tag currently stored for the read batch.

[[step]]
 action = "ForgetAllTags"

Use this when you want to clear all tag labels before continuing. It is handy after persisting tags to an external table, or before running steps that must not see previously extracted metadata.

Mon, 01 Jan 0001 00:00:00 +0000

ForgetTag #

Forget about a tag. Useful if you want to store tags in a table, but not this one.

[[step]]
 action = "ForgetTag"
 in_label = "mytag"

This transformation removes a specified tag from the molecule’s tag collection.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInComment #

Store currently present tags as comments on read names.

[[step]]
 action = "StoreTagInComment"
 in_label = "mytag" # if set, only store this tag
 segment = "read1" # Any of your input segments, or 'All'
 comment_insert_char = " " # (optional) char at which to insert comments
 comment_separator = "|" # (optional) char to separate comments
 region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple

Comments are key=value pairs, separated by comment_separator which defaults to ‘|’. They get inserted before the first comment_insert_char, which defaults to space.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInSequence #

Store the tag’s replacement in the sequence, replacing the original sequence at that location.

[[step]]
 action = "StoreTagInSequence"
 in_label = "mytag"
 ignore_missing = true # if false, an error is raised if the tag is missing

This transformation stores the tag’s value back into the sequence, replacing the original sequence at that location.

Note that if this changes the length of the sequence, existing location tags will loose their location data (retaining their sequence though).

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagLocationInComment #

Store the coordinates of a tag in the comment (start-end, 0-based, half-open).

[[step]]
 action = "StoreTagLocationInComment"
 in_label = "mytag"
 segment = "read1" # Any of your input segments, or 'All'
 comment_insert_char = " " # (optional) char at which to insert comments
 comment_separator = "|" # (optional) char to separate comments

This transformation stores the location coordinates of a tag as a comment in the read name, useful for tracking where tags were extracted from.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagsInTable #

Store the tags in a TSV table.

[[step]]
 action = "StoreTagsInTable"
 infix = "tags"
 compression = "Raw" # Raw, Gzip, Zstd
 region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple

This transformation writes all current tags to a tab-separated values (TSV) table file for further analysis.

The output filename is constructed as {prefix}_{infix}.tsv (or with custom separator if configured).

Interaction with demultiplexing #

When demultiplexing is used, separate TSV files are created for each barcode: {prefix}_{infix}_{barcode}.tsv.

Extract IUPAC

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPAC #

[[step]]
 action = "ExtractIUPAC"
 out_label = "mytag"
 anchor = 'Left' # Left | Right | Anywhere
 search = "CTN" # what we are searching
 segment = 'read1' # Any of your input segments

Search and extract a sequence from the read, defined by a IUPAC string.

If anchor = ‘Anywhere’, ExtractIUPAC will find the left most occurance.

Extract IUPAC suffix

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACSuffix #

[[step]]
 action = "ExtractIUPACSuffix"
 out_label = "mytag"
 query = "AGTCA" # the adapter to trim. Straigth bases only, no IUPAC.
 segment = "read1" # Any of your input segments (default: read1)
 min_length = 3 # uint, the minimum length of match between the end of the read and
 # the start of the adapter
 max_mismatches = 1 # How many mismatches to accept

Find a potentially truncated IUPAC string sequence at the end of a read.

Simple comparison with a max mismatch hamming distance, requiring only the first min length bases of the query to match at the end of the read.

Extract IUPAC with Indels

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACWithIndel #

[[step]]
 action = "ExtractIUPACWithIndel"
 out_label = "adapter"
 search = "AGTC" # IUPAC pattern to align against
 max_mismatches = 1 # allowed substitutions (IUPAC-aware)
 max_indel_bases = 1 # total insertions + deletions allowed
 max_total_edits = 2 # optional overall edit ceiling
 anchor = 'Anywhere' # Left | Right | Anywhere
 segment = 'read1' # defaults to read1

Locate an IUPAC pattern even when the read contains small insertions or deletions relative to the pattern. The extractor performs a semiglobal alignment (pattern vs. read segment) using IUPAC-aware scoring and returns the aligned span as a location tag.

Mon, 01 Jan 0001 00:00:00 +0000

StoreTagInFastQ #

# Store the content of a tag in a fastq file.
# Needs a 'location 'tag'.
# Can store other tags in the read name.
# quality scores are set to '~'.
# With demultiplexing: creates separate files per barcode
 [[step]]
 action = "StoreTagInFastQ"
 in_label = "mytag" # tag to store. If no infix is set, infix = tag={in_label}
 format = "Fastq" # FASTQ / FASTA / BAm
 compression = "gz" # or "zstd" | "none" # (optional) compression format, not if format == BAM
 compression_level = 6 # (optional) compression level for gzip (0-9) or zstd (1-22)
 					 # defaults: gzip=6, zstd=5
 comment_tags = []# e.g. ["other_tag"] # see StoreTagInComment
 comment_location_tags = ["mytag"] # (optional) tags to add location info for, defaults to [in_label]
 # set to [] to disable location tracking
 comment_insert_char = ' ' # (optional) char at which to insert comments
 comment_separator = '|' # (optional) char to separate comments
 region_separator = "_" # (optional) char to separate regions in a tag, if it has multiple

Store the sequence of a tag in a fastq file, with other tags optionally stored in the read name as comments.

Mon, 01 Jan 0001 00:00:00 +0000

CalcExpectedError #

Compute aggregated per-base error probabilities (expected errors) for each read assuming PHRED+33 qualities.

[[step]]
 action = "CalcExpectedError"
 out_label = "expected_error"
 aggregate = "sum" # or "max"
 segment = "read1" # Any of your input segments, or 'All'

If your data is not encoded as Phred+33, convert it first (for example, with ConvertQuality) before running this step. Values outside of the Phred+33 range will lead to an stop with an error.

Set aggregate = "sum" to calculate the sum of per-base error probabilities. Use aggregate = "max" to store only the worst base’s error probability for each read or read pair.

Mon, 01 Jan 0001 00:00:00 +0000

ConvertRegionsToLength #

Turn region tags (such as those produced by ExtractRegion/ExtractRegions) into numeric length tags.

[[step]]
 action = "ExtractRegion"
 out_label = "adapter"
 segment = "read1"
 start = 0
 len = 12

[[step]]
 action = "ConvertRegionsToLength"
 out_label = "adapter_len"
 in_label = "adapter"

The new tag stores the total span (in bases) covered by all regions on each read.
Reads without the source tag receive a length of 0.
label must be different from region_label; the step keeps the original region tag.

Mon, 01 Jan 0001 00:00:00 +0000

EvalExpression #

[[step]]
 action = "EvalExpression"
 out_label = "outtag"
 expression = "log(2, mytag + 1)" # log to base 2
 result_type = "numeric" # or bool.

Calculate a fasteval expression on your tags, which you can then pass to .FilterByTag.

You can use any tags previously defined on the molecule as variables in the expression.

Additional, there’s a series of virtual tags available:

len_<segment-name> - the length of the specified segment (e.g. len_read1).
len_<tag-label> - the length of the specified tag (e.g. len_mytag). For location tags, this is the length of the underlying matched regions (which may change / be lost when reads are truncated - eval before truncation if necessary). For string tags (= ExtractRegex with source=name:...) this is the length of the replaced string.

Mon, 01 Jan 0001 00:00:00 +0000

Options #

There is a small set of runtime knobs exposed under [options]. Most workflows can rely on the defaults.

[options]
 thread_count = -1
 block_size = 10000
 buffer_size = 102400
 accept_duplicate_files = false
 spot_check_read_pairing = true

Key	Default	Description
`thread_count`	`-1`	Worker threads for transformations. `-1` autotunes per CPU; most runtime is still dominated by decompression threads, so gains are modest.
`block_size`	`10000`	Number of fragments pulled per batch. Increase for very large runs when IO is abundant; decrease to reduce peak memory use.
`buffer_size`	`102400`	Initial bytes reserved per block. The allocator grows buffers on demand, so tuning is rarely necessary.
`accept_duplicate_files`	`false`	Permit the same path to appear multiple times across segments. Useful for fixtures or synthetic tests; keep disabled to catch accidental copy/paste errors.
`spot_check_read_pairing`	`true`	Sample every 1000th fragment to ensure paired reads still share a name prefix; disable when names are intentionally divergent or rely on `ValidateName` to customise the separator.

Changing these knobs can affect memory pressure and concurrency behaviour. Measure before and after if you deviate from defaults.

Mon, 01 Jan 0001 00:00:00 +0000

CalcKmers #

Count the number of kmers from a read that match those in a database built from reference sequences.

[[step]]
 action = "CalcKmers"
 out_label = "mytag"
 segment = "read1" # Any of your input segments, or 'All'
 files = ['reference.fa', 'database.fq']
 count_reverse_complement = true # whether to also include each revcomp of a kmer in the database
 k = 21
 min_count = 2 # optional, defaults to 1

This transformation:

Builds a kmer database from the specified sequence files (all input formatws)
Extracts all kmers of length k from the reference sequences
Filters kmers by min_count (minimum occurrences in the reference to be included)
For each read, counts how many of its kmers appear in the database
Creates a numeric tag with the kmer match count

Parameters #

out_label: Tag name to store the kmer count
segment: Which segment to quantify (read1, read2, index1, index2, or ‘All’)
files: List of sequence files to build the kmer database from
count_reverse_complement: (alias: “canonical”) Whether to include reverse complements of kmers in the database (‘canonical kmers’)
k: Kmer length
min_count: Minimum number of times a kmer must appear in the reference files to be included in the database (default: 1). Sum of forward and reverse complement counts if count_reverse_complement is true.

Use Cases #

Contamination detection: Quantify or filter reads matching known contaminant sequences
Quality control: Count kmers from adapter or primer sequences
Species identification: Measure presence of species-specific kmers

Notes #

Only kmers with only valid DNA bases (A, C, G, T) are counted; kmers containing N or other ambiguous bases are skipped
Kmer matching is case-insensitive

Mon, 01 Jan 0001 00:00:00 +0000

LowercaseTag #

[[step]]
 action = "LowercaseTag"
 in_label = "mytag"

Replace the sequence of the tag with it’s lowercase version.

Follow with StoreTagInSequence to apply the lowercase tag back onto the read.

See the tag section for tag generation.

Mon, 01 Jan 0001 00:00:00 +0000

LowercaseSequence #

[[step]]
 action = "LowercaseSequence"
 segment = "read1" # Any of your input segments, or 'All'

Convert the complete sequence to lowercase.

Mon, 01 Jan 0001 00:00:00 +0000

UppercaseSequence #

[[step]]
 action = "UppercaseSequence"
 segment = "read1" # Any of your input segments, or 'All'

Convert the complete sequence to uppercase.

Mon, 01 Jan 0001 00:00:00 +0000

UppercaseTag #

[[step]]
 action = "UppercaseTag"
 in_label = "mytag"

Replace the sequence of the tag with its uppercase version.

Follow with StoreTagInSequence to apply the uppercase tag back onto the read.

See the tag section for tag generation.

Mon, 01 Jan 0001 00:00:00 +0000

Segments #

Modern sequencers, particularity Illumina sequencers can read multiple times from one (amplified) DNA molecule, producing multiple ‘segments’ that together form a ‘molecule’.

The user’s define the available segments in the input-section, common names are ‘read1’, ‘read2’ (for paired-end reads) and ‘index1’, ‘index2’ (for dual-indexed libraries).

These are then commonly available to steps taking a segment or a source option. Often there is the option to work on all all defined segments at once by using the ‘All’ ‘segment/source’.

Mon, 01 Jan 0001 00:00:00 +0000

Source #

When a step refers to a ‘source’ (instead of a segment), it means that it can read from segment, a segment’s name (e.g. “name:read1”) or a (location/string) tag value (e.g. “tag:mytag”)

Mon, 01 Jan 0001 00:00:00 +0000

Step #

A step is one coherent manipulation of the FASTQ stream and it’s associated data.

Mon, 01 Jan 0001 00:00:00 +0000

Tag / Label #

A tag is a piece of molecule derived data that one step in the pipeline produces, and others may use (or output).

Tags have types: - Location+Sequence (which may loose their location data) - Sequence-only - numeric - boolean

For example ExtractIUPAC produces a ’location’ tag.

That location tag then allows filtering for it’s presence using FilterByTag, cutting the segment at it’s location using TrimAtTag or replacing it’s sequence with LowercaseTag.

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 01: Basic Quality Report #

Use Case #

You have FastQ files from a sequencing run and want to generate comprehensive quality reports to assess:

Read quality scores
Base composition
Read length distribution
Duplicate read counts

This is typically the first step in any sequencing data analysis to understand data quality before downstream processing.

What This Pipeline Does #

Reads input FastQ file(s)
Generates a comprehensive quality report including:
- Base quality statistics
- Base distribution across positions
- Read length distribution
- Duplicate read counting
Outputs reports in both HTML (human-readable) and JSON (machine-readable) formats
Passes through all reads unchanged (no filtering)

Input Files #

input/sample_R1.fq - Forward reads (Read 1) from paired-end sequencing

Output Files #

output_R1.fq - Passed-through reads (identical to input)
output.report_initial.html - HTML quality report
output.report_initial.json - JSON quality report with detailed statistics

When to Use This #

First analysis of new sequencing data
Quality control before committing to expensive downstream analysis
Comparing data quality across different sequencing runs
Identifying potential issues (adapter contamination, quality drop-off, etc.)

Download #

Download 01-basic-quality-report.tar.gz for a complete, runnable example including expected output files.

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 02: UMI Extraction #

Use Case #

You have sequencing data with Unique Molecular Identifiers (UMIs) embedded in the reads. UMIs are short random barcodes added during library preparation that allow you to:

Identify and remove PCR duplicates
Distinguish true biological duplicates from amplification artifacts
Improve accuracy in quantitative analyses (RNA-seq, ATAC-seq, etc.)

What This Pipeline Does #

Reads input FastQ file with UMIs at the start of read1
Extracts the UMI sequence (first 8 bases) and creates a tag
Stores the UMI in the read comment (FASTQ header)
Removes the UMI bases from the read sequence (so they don’t interfere with alignment)
Outputs modified reads with UMI preserved in the header

Input Files #

input/sample_R1.fq - Reads with 8bp UMI at the start

Output Files #

output_R1.fq - Reads with UMI in comment, UMI bases removed from sequence

Configuration Highlights #

[[step]]
 # Extract UMI from positions 0-7 (8 bases)
 action = 'ExtractRegions'
 label = 'umi'
 regions = [{segment = 'read1', start = 0, length = 8}]

[[step]]
 # Store UMI in the FASTQ comment
 action = 'StoreTagInComment'
 label = 'umi'

[[step]]
 # Remove the UMI bases from the read
 action = 'CutStart'
 target = 'Read1'
 n = 8

Workflow Details #

Before processing:

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 03: Lexogen QuantSeq Processing #

Use Case #

Lexogen QuantSeq is a popular 3’ mRNA sequencing protocol optimized for gene expression profiling. The library structure includes:

First 8 bases: UMI (Unique Molecular Identifier) for deduplication
Next 6 bases: Random hexamer primer sequence (needs removal)
Remaining sequence: Actual cDNA from the 3’ end of transcripts

This cookbook demonstrates the standard preprocessing for QuantSeq data before alignment.

What This Pipeline Does #

Extracts the 8bp UMI from the start of reads
Stores the UMI in the read comment (FASTQ header)
Removes the first 14 bases total (8bp UMI + 6bp random hexamer)
Outputs processed reads ready for alignment

Input Files #

input/quantseq_sample.fq - Raw QuantSeq reads with UMI and random hexamer

Output Files #

output_read1.fq - Processed reads with:
- UMI stored in comment
- First 14bp removed
- Ready for alignment to reference genome

Workflow Details #

Raw read structure:

Mon, 01 Jan 0001 00:00:00 +0000

Cookbook 04: PhiX Removal #

Use Case #

You have Illumina PhiX spike-in sequences in your dataset and want to remove those contaminating reads before downstream analysis. PhiX is commonly added as a control to increase base diversity during sequencing runs.

What This Pipeline Does #

This cookbook demonstrates how to identify and remove PhiX contamination using k-mer counting:

Count k-mers: Uses CalcKmers to count how many 30-mers from each read match the PhiX genome
Export data: Saves k-mer counts to a TSV table for analysis
Filter reads: Removes reads with high PhiX k-mer counts (≥25 matching k-mers)

Understanding the Approach #

K-mer Counting #

The CalcKmers step counts how many k-mers (short subsequences of length k) from each read are present in the PhiX reference genome:

Mon, 01 Jan 0001 00:00:00 +0000

FilterReservoirSample #

[[step]]
 action = "FilterReservoirSample"
 n = 10_000
 seed = 59014

Filter for a fixed number of reads based on reservoir sampling, that is all input reads have an equal probability of being selected.

This means we need to keep n reads in memory, and they get processed as one large block at the end by all downstream steps.

That means it’s not the right tool if you want to sample to millions of reads, but it is the right tool if you want to have lowish fixed number of reads.

Mon, 01 Jan 0001 00:00:00 +0000

FilterSample #

[[step]]
 action = "FilterSample"
 p = 0.5 # float, the chance for any given read to be kept
 # 0..1
 seed = 42 # (optional) random seed for reproducibility

Randomly sample a percentage of reads. Requires a random seed to ensure reproducibility.

Mon, 01 Jan 0001 00:00:00 +0000

ConvertQuality #

[[step]]
 action = "ConvertQuality"
 from = "Illumina1.8"# Illumin1.8|Illumina1.3|Sanger|Solexa"
 to = "Solexa" # same range as from. Illumina1.8 is an alias for Sanger

Convert quality scores between various encodings / meanings.

See https://en.wikipedia.org/wiki/Phred_quality_score

Will error if from == to.

This step introduces a ValidateQuality step automatically before it.

Corresponding options in other software #

trimmomatic TOPHRED33

Mon, 01 Jan 0001 00:00:00 +0000

CutEnd #

[[step]]
 action = "CutEnd"
 n = 5 # positive integer, cut n nucleotides from the end of the read
 segment = "read1" # Any of your input segments (default: read1)

Cut nucleotides from the end of the read.

May produce empty reads; filter those with FilterEmpty.

Mon, 01 Jan 0001 00:00:00 +0000

CutStart #

[[step]]
 action = "CutStart"
 n = 5 # positive integer, cut n nucleotides from the start of the read
 segment = "read1" # Any of your input segments

Cut nucleotides from the start of the read.

May produce empty reads; filter those with FilterEmpty.

Mon, 01 Jan 0001 00:00:00 +0000

[[step]]
 action = "ExtractRegions"
 regions = [
 {segment= "Read1", start = 0, length = 8},
 {segment= "Read1", start = 12, length = 4},
 ]
 out_label = "umi"
 region_separator = "_" # (optional) str, what to put between the regions, defaults to '_'

[[step]]
 action = "StoreTagInComment" 
 in_label = "umi"

Extract a sequence from the read and place it in the read name’s comment section, so a (space separated) ‘key=value’ pair is added to the read name.

Supports multiple region-extraction.

See the tag section for more tag generation options.

Mon, 01 Jan 0001 00:00:00 +0000

Head #

[[step]]
 action = "Head"
 n = 1000 # positive integer, number of reads to keep

Output just the first n molecules.

Demultiplex interaction #

If present after a demultiplex step, includes n molecules in each stream.

Mon, 01 Jan 0001 00:00:00 +0000

MergeReads #

[[step]]
 action = "MergeReads"

 segment1 = "read1" # First segment
 segment2 = "read2" # Second segment
 reverse_complement_segment2 = true # Whether to reverse complement segment2 (suggested: true)

 algorithm = "FastpSeemsWeird" # Algorithm: "fastp_seems_weird". Further algorithms are in planning
 min_overlap = 30 # Minimum overlap length required
 max_mismatch_rate = 0.2 # Maximum allowed mismatch rate (0.0-1.0) (suggested: 0.2)
 max_mismatch_count = 5 # Maximum allowed absolute mismatches (suggested: 5)
 no_overlap_strategy = "as_is" # What to do when no overlap found: "as_is" or "concatenate"
 concatenate_spacer = "NNNN" # (optional) Required if no_overlap_strategy = "concatenate". Spacer sequence to insert between reads
 spacer_quality_char = 33 # (optional) Quality score for spacer bases (suggested: 33 = Phred quality 0)

 # out_label = "merged" # (optional) output Tag label for boolean merge status

Merges paired-end reads from two segments by detecting their overlap and resolving mismatches.

Mon, 01 Jan 0001 00:00:00 +0000

Rename #

[[step]]
 action = "Rename"
 search = '(.)/([1/2])$'
 replacement = '$1 $2'

Apply a regular expression based renaming to the reads.

It is always applied to all available segments (read1, read2, index1, index2).

The example above fixes old school MGI reads for downstream processing, like fastp’s ‘–fix_mgi’ option

You can use the full power of the rust regex crate here.

Read index placeholder #

After the regex replacement runs, the special literal {{READ_INDEX}} is expanded to the running 1-based index of each logical read. When multiple segments are present (for example read1/read2 pairs), every segment for the same read receives the same index so pairs stay aligned. This makes it easy to re-sequence identifiers:

Mon, 01 Jan 0001 00:00:00 +0000

ReverseComplement #

[[step]]
 action = "ReverseComplement"
 segment = "read1" # Any of your input segments (default: read1)

Reverse-complements the read sequence (and reverses the quality).

This supports IUPAC codes (U is complemented to A, so it’s not strictly reversible). Unknown letters are output verbatim.

Useful to combine with SwapConditional.

Mon, 01 Jan 0001 00:00:00 +0000

ReverseComplementConditional #

[[step]]
 action = "ReverseComplementConditional"
 in_label = "mytag" # Boolean tag that determines whether to reverse complement
 segment = "read1" # Any of your input segments (default: read1)

Conditionally reverse-complements the read sequence (and reverses the quality) based on a boolean tag value.

For each read, if the boolean tag is true, the read is reverse-complemented. If the tag is false, the read is left unchanged.

This is useful for conditionally orienting reads based on strand information, adapter detection, or other criteria determined by prior steps.

Mon, 01 Jan 0001 00:00:00 +0000

Skip #

[[step]]
 action = "Skip"
 n = 1000 # positive integer, number of reads to skip

Skips the first n molecules.

Demultiplex interaction #

If present after a demultiplex step, skips the first n molecules in that stream.

Mon, 01 Jan 0001 00:00:00 +0000

Swap #

[[step]]
 action = "Swap"
 segment_a = "read1"
 segment_b = "read2"

Swaps exactly two segments.

Arguments segment_a/segment_b are only necessary if there are more than two segments defined in the input.

Mon, 01 Jan 0001 00:00:00 +0000

SwapConditional #

[[step]]
 action = "SwapConditional"
 in_label = "mytag" # Boolean tag that determines whether to swap
 segment_a = "read1" # Optional
 segment_b = "read2" # Optional

Conditionally swaps exactly two segments based on a boolean tag value.

For each read, if the boolean tag is true, the segments are swapped for that specific read. If the tag is false, the read is left unchanged.

This is useful for conditionally reorganizing paired-end data based on read properties, such as swapping reads based on length, quality, or other criteria determined by prior steps.

Mon, 01 Jan 0001 00:00:00 +0000

MaxLen #

[[step]]
 action = "Truncate"
 n = 100 # the maximum length of the read. Cut at end if longer
 segment = "read1" # Any of your input segments (default: read1)

Cut the read down to `n’ bases.

Mon, 01 Jan 0001 00:00:00 +0000

Out of scope #

Things mbf-fastq-processor will explicitly not do and that won’t be implemented.

Anything based on averaging phred scores #

Based on the average quality in a sliding window. Arithmetic averaging of phred scores is wrong.

see ExtractMeanQuality

Corresponding options in other software #

Trimmomatic SLIDINGWINDOW
fastp –cut_front
fastp –cut_tail
fastp –cut_right

Fast5 #

https://medium.com/@shiansu/a-look-at-the-nanopore-fast5-format-f711999e2ff6 Oxford Nanopore squiggle data. Apparently no formal spec.

kallisto BUS format #

- a brief barcode/umi format for single cell RNA-seq
- needs an 'equivalance class' - i.e. at least pseudo alignment
- weird length restrictions on barcodes and umis (1(!)-32), 
 but stores the length in an uint32...

Mon, 01 Jan 0001 00:00:00 +0000

Inspect #

Dump a few reads to a FASTQ file for inspection at this point in the graph.

[[step]]
 action = "Inspect"
 n = 1000 # how many molecules 
 infix = "inspect_at_point" # output filename infix
 segment = "read1" # Any of your input segments (use "all" for interleaved output)
 format = "FASTQ" # output format: FASTQ, FASTA or BAM
 suffix = "compressed" # (optional) custom suffix for filename
 compression = "gzip" # (optional) compression format: raw, gzip, zstd. Defaults to uncompressed
 compression_level = 1 # (optional) compression level for gzip/zstd/bam (gzip, zstd: 1-22)
 # defaults: gzip=6, zstd=5

Output filename pattern:

Mon, 01 Jan 0001 00:00:00 +0000

Progress #

Emit progress to stdout (default) or a .progress log file, if output_infix is set. (filename is {output_prefix}{ix_separator}{infix}.progress, default separator _).

[[step]]
 action = "Progress"
 n = 100_000
 output_infix = "progress" # optional

Every n reads, report on total progress, total reads per second. At the end, report final runtime and reads/second.

Mon, 01 Jan 0001 00:00:00 +0000

CalcComplexity #

[[step]]
 action = "CalcComplexity"
 out_label = "complexity"
 segment = "read1" # Any of your input segments, or 'All'

Calculate read complexity, based on the percentage of bases that are changed from their predecessor.

A good filter value might be 0.30, which means 30% complexity is required. See FilterByNumericTag.

Corresponding options in other software #

fastp: -low_complexity_filter

Mon, 01 Jan 0001 00:00:00 +0000

CalcMeanQuality #

We don’t support calculating the ‘average quality’.

This is typically a bad idea, see https://www.drive5.com/usearch/manual/avgq.html for a discussion of the issues.

To illustrate, 140 x Q35 + 10 x Q2 reads have an ‘average’ phred of 33, but 6.4 expected wrong bases. A read with 150 x Q25 has a much worse ‘average’ phred of 25, but a much lower expected number of errors at 0.5.

Corresponding options in other software #

Trimmomatic: AVGQUAL:
fastp: –average_qual

Mon, 01 Jan 0001 00:00:00 +0000

#CalcQualifiedBases

[[step]]
 action = "CalcQualifiedBases"
 threshold = 'C' # the quality value >= which a base is qualified 
 # In your phred encoding. Typically 33..75
 # a byte or a number 0...255
 op = 'worse' # see below.
 segment = "read1" # Any of your input segments, or 'All'
 out_label = "tag_name"

Calculate the number of bases that are ‘qualified’, that is abov/below a user defined threshold.

Note that smaller Phred values are better.

To remove confusion, op may be ‘Better’/‘Worse’ instead of ‘Below’/‘Above’.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLongestPolyX #

Find the longest homopolymer stretch anywhere in the read (unlike ExtractPolyTail, which only considers suffixes).

[[step]]
 action = "ExtractLongestPolyX"
 out_label = "my_tag"
 segment = "read1"
 min_length = 5
 base = '.' # search for any homopolymer (A/C/G/T/N)
 max_mismatch_rate = 0.15
 max_consecutive_mismatches = 2

base accepts a concrete nucleotide (A, C, G, T, N) or . to search all of ACGT and report the longest hit.
max_mismatch_rate and max_consecutive_mismatches mirror ExtractPolyTail; they control how permissive the run detection is.
When no run satisfies min_length, the tag is reported as missing.
only one run is reported, even if multiple runs of the same length exist; in this case, the first run found is reported.

Mon, 01 Jan 0001 00:00:00 +0000

TrimQualityEnd #

[[step]]
 action = "ExtractLowQualityEnd"
 out_label = "low_quality_ends"
 min_qual = 20 # u8, minimum quality to keep (in whatever your score is encoded in)
 # either a char like 'A' or a number 0..128 (typical phred score is 33..75)
 segment = "read1" # Any of your input segments

Define a region of low quality bases at the end of reads.

Corresponding options in other software #

Trimmomatic: TRAILING (if paired with TrimAtTag)

Mon, 01 Jan 0001 00:00:00 +0000

TrimQualityStart #

[[step]]
 action = "ExtractLowQualityStart"
 min_qual = 20 # u8, minimum quality to keep (in whatever your score is encoded in)
 # either a char like 'A' or a number 0..128 (typical phred score is 33..75)
 segment = "read1" # Any of your input segments
 out_label = "bad_starts"

Define a region with low quality bases (below threshold) at steart of read.

Corresponding options in other software #

Trimmomatic: LEADING (if combined with TrimAtTag)

Mon, 01 Jan 0001 00:00:00 +0000

ExtractPolyTail #

[[step]]
 action = "ExtractPolyTail"
 out_label = "tag_label"
 segment = "read1" # Any of your input segments (default: read1)
 min_length = 5 # positive integer, the minimum number of repeats of the base
 base = "A" # one of AGTCN., the 'base' to trim (or . for any repeated base)
 max_mismatch_rate = 0.1 # float 0.0..=1.0, how many mismatches are allowed in the repeat
 max_consecutive_mismatches = 3 # how many consecutive mismatches are allowed

Identify either a specific letter (AGTC or N) repetition, or any base repetition (base = ‘.’) at the end of the read.

Mon, 01 Jan 0001 00:00:00 +0000

FilterDuplicates #

[[step]]
 action = "TagDuplicates"
 false_positive_rate = 0.00001 #
 # the false positive rate of the filter.
 # 0..1
 seed = 59 # required!
 source = "All" # Any input segment, 'All', 'tag:<tag-name>' or 'name:<segment>'
 # split_character = "/" # required (and accepted only iff using name:<segment>
 out_label = "dups"

[[step]]
 action = "FilterByTag"
 in_label = "dups"
 keep_or_remove = "Remove" # Keep|Remove

Tag duplicates (2nd onwards) from the stream using a Cuckoo filter.

That’s a probabilistic data structure, accordingly there’s a false positive rate, and a tunable memory requirement.

Mon, 01 Jan 0001 00:00:00 +0000

HammingCorrect #

Correct a tag to one of a predefined set of ‘barcodes’ using closest hamming distance.

[[step]]
 action = "HammingCorrect"
 in_label = "mytag"
 out_label = "my_corrected_tag"
 barcodes = "mybarcodelist"
 max_hamming_distance = 1
 on_no_match = 'remove' # 'remove', 'empty', 'keep'

[barcodes.mybarcodelist]
 "AAAA" = "ignored" # only read when demultiplexing

on_no_match controls what happens if the tag cannot be corrected within the max_hamming_distance:

remove: Remove the hit (location and sequence), useful for FilterByTag later.
keep: Keep the original tag (and location)
empty: Keep the original location, but set the tag to empty.

Mon, 01 Jan 0001 00:00:00 +0000

SpotCheckReadPairing #

Confirms for every sample_strideth read ‘pair’ that the names share the same prefix (part before ‘readname_end_char’).

[[step]]
 action = "SpotCheckReadPairing"
 sample_stride = 1000 # Every nth fragment, default 1000. Must be > 0. 
 readname_end_char = '/' # u8/byte-char, Defaults to no character, meaning the entire name is compared.

When no separator character is provided the entire name must match exactly across all segments.

Otherwise the read names are truncated after the first readname_end_char, and the prefixes must match exactly. For example, use readname_end_char = “_” for typical Illumina _1/_2 suffixes.

Mon, 01 Jan 0001 00:00:00 +0000

ValidateAllReadsSameLength #

[[step]]
 action = "ValidateAllReadsSameLength"
 source = "read1" # Any segment, All, tag:<name> or 'name:segment>'

Validates that all reads have the same sequence/tag/read length.

Useful when you want to verify read length consistency in your pipeline.

Mon, 01 Jan 0001 00:00:00 +0000

ValidateName #

Verify that all segments have the same read name (or a shared prefix).

[[step]]
 action = "ValidateName"
 # Optional separator character; the comparison stops at the first match
 readname_end_char = "_" # Optional. Do not set for exact matching. Otherwise, a byte character

When no separator character is provided the entire name must match exactly across all segments.

Otherwise the read names are truncated after the first readname_end_char, and the prefixes must match exactly. For example, use readname_end_char = "_" for typical Illumina _1/_2 suffixes.

Mon, 01 Jan 0001 00:00:00 +0000

ValidateQuality #

Validate that all scores are between 33..=41

[[step]]
 action = "ValidateQuality"
 segment = "read1" # Any of your input segments, or 'All'
 encoding = 'Illumina1.8' # 'Illumina1.8|Illumina1.3|Sanger|Solexa'
 # Illumina1.8 is an alias for Sanger.

The encoding defines the accepted range of values.

If you want to convert quality codes, use ConvertQuality.

See https://pmc.ncbi.nlm.nih.gov/articles/PMC2847217/ , table 1

Mon, 01 Jan 0001 00:00:00 +0000

ValidateSeq #

Validate that only allowed characters are in the sequence.

[[step]]
 action = "ValidateSeq"
 allowed = "AGTC" # String. Example 'ACGTN', the allowed characters
 segment = "read1" # Any of your input segments, or 'All'

(Older) Documentation Versions

Mon, 01 Jan 0001 00:00:00 +0000

Available builds:

main
v0.8.1 (this build)
v0.8.0-test
v0.8.0

Calc Base Content

Mon, 01 Jan 0001 00:00:00 +0000

CalcBaseContent #

[[step]]
 action = "CalcBaseContent"
 segment = "read1" # Any of your input segments, or 'All'
 out_label = "at_content"
 bases_to_count = "AT"
 bases_to_ignore = "N"
 relative = true # default

Counts the percentage of bases that match bases_to_count, while removing any bases listed in bases_to_ignore from the denominator (if relative = True).

Both lists are case-insensitive, and accept only ascii letters. When no bases remain after filtering, the step returns 0.

Set relative = false to emit absolute base counts instead of percentages. Absolute mode requires bases_to_ignore to remain unset, otherwise the configuration check fails.

Calc GC Content

Mon, 01 Jan 0001 00:00:00 +0000

CalcGCContent #

[[step]]
 action = "CalcGCContent"
 segment = "read1" # Any of your input segments, or 'All'
 out_label = "gc"

Count what percentage of bases are GC (as opposed to AT). Non-AGTC bases (e.g. N) are ignored.

Output is 0..100.

Wrapper around CalcBaseContent with bases = "GC", ignore="N", relative=true).

Calc N Count

Mon, 01 Jan 0001 00:00:00 +0000

CalcNCount #

[[step]]
 action = "CalcNCount"
 segment = "read1" # Any of your input segments, or 'All'
 out_label="ncount"

Count how many N are present in the read.

This step is a convenient wrapper for CalcBaseContent with bases_to_count = "N" and relative = false.

Corresponding options in other software #

fastp: –n_base_limit (if combined with FilterByNumericTag)