mbf-fastq-processor documentation

Mon, 01 Jan 0001 00:00:00 +0000

Postfix #

Add DNA to the end of read sequences.

[[step]]
 action = "Postfix"
 seq = "agtc" # DNA sequence to add at end of reads. Checked to be agtcn
 qual = "IIII" # same length as seq. Your responsibility to have valid phred values
 segment = "read1" # Any of your input segments

This transformation adds a specified sequence and corresponding quality scores to the end of reads.

Mon, 01 Jan 0001 00:00:00 +0000

Prefix #

Add text to the beginning of read sequences.

[[step]]
 action = "Prefix"
 seq = "agtTCAa" # DNA sequence to add at beginning of reads. Checked to be agtcn
 qual = "IIIBIII" # same length as seq. Your responsibility to have valid phred values
 segment = "read1" # Any of your input segments

This transformation adds a specified sequence and corresponding quality scores to the beginning of reads.

Mon, 01 Jan 0001 00:00:00 +0000

ReplaceTagWithLetter #

Replace sequence bases in tagged regions with a specified letter.

[[step]]
 action = "ReplaceTagWithLetter"
 in_label = "mytag" # Tag containing regions to replace
 letter = "N" # Replacement character (defaults to 'N')

This transformation replaces all sequence bases within the regions defined by a tag with a specified replacement character. Quality scores are preserved unchanged. This is commonly used to mask low-quality regions as ‘N’ characters.

Parameters #

in_label: Name of the tag containing regions to be replaced
letter: Single character to replace bases with (defaults to ‘N’ if not specified)

Example Use Cases #

Mask low-quality bases identified by ExtractRegionsOfLowQuality
Replace specific sequence motifs identified by other extraction steps
Convert tagged regions to ambiguous bases for downstream analysis

The tag must have been created by a previous extraction step and must contain location information.

Mon, 01 Jan 0001 00:00:00 +0000

TrimAtTag #

Trim the read at the position of a tag.

[[step]]
 action = "TrimAtTag"
 in_label = "mytag"
 direction = "Start" # or "End"
 keep_tag = false # if true, the tag sequence is kept in the read

This transformation trims the read at the position where a tag was found.

The direction parameter determines whether to trim from the start or end of the tag, and keep_tag determines whether the tag sequence itself is retained.

Mon, 01 Jan 0001 00:00:00 +0000

LowercaseTag #

[[step]]
 action = "LowercaseTag"
 in_label = "mytag"

Replace the sequence of the tag with it’s lowercase version.

Follow with StoreTagInSequence to apply the lowercase tag back onto the read.

See the tag section for tag generation.

Mon, 01 Jan 0001 00:00:00 +0000

LowercaseSequence #

[[step]]
 action = "LowercaseSequence"
 segment = "read1" # Any of your input segments, or 'All'

Convert the complete sequence to lowercase.

Mon, 01 Jan 0001 00:00:00 +0000

UppercaseSequence #

[[step]]
 action = "UppercaseSequence"
 segment = "read1" # Any of your input segments, or 'All'

Convert the complete sequence to uppercase.

Mon, 01 Jan 0001 00:00:00 +0000

UppercaseTag #

[[step]]
 action = "UppercaseTag"
 in_label = "mytag"

Replace the sequence of the tag with its uppercase version.

Follow with StoreTagInSequence to apply the uppercase tag back onto the read.

See the tag section for tag generation.

Mon, 01 Jan 0001 00:00:00 +0000

ConvertQuality #

[[step]]
 action = "ConvertQuality"
 from = "Illumina1.8"# Illumin1.8|Illumina1.3|Sanger|Solexa"
 to = "Solexa" # same range as from. Illumina1.8 is an alias for Sanger

Convert quality scores between various encodings / meanings.

See https://en.wikipedia.org/wiki/Phred_quality_score

Will error if from == to.

This step introduces a ValidateQuality step automatically before it.

Corresponding options in other software #

trimmomatic TOPHRED33

Mon, 01 Jan 0001 00:00:00 +0000

CutEnd #

[[step]]
 action = "CutEnd"
 n = 5 # positive integer, cut n nucleotides from the end of the read
 segment = "read1" # Any of your input segments (default: read1)

Cut nucleotides from the end of the read.

May produce empty reads; filter those with FilterEmpty.

Mon, 01 Jan 0001 00:00:00 +0000

CutStart #

[[step]]
 action = "CutStart"
 n = 5 # positive integer, cut n nucleotides from the start of the read
 segment = "read1" # Any of your input segments

Cut nucleotides from the start of the read.

May produce empty reads; filter those with FilterEmpty.

Mon, 01 Jan 0001 00:00:00 +0000

[[step]]
 action = "ExtractRegions"
 regions = [
 {segment= "Read1", start = 0, length = 8},
 {segment= "Read1", start = 12, length = 4},
 ]
 out_label = "umi"
 region_separator = "_" # (optional) str, what to put between the regions, defaults to '_'

[[step]]
 action = "StoreTagInComment" 
 in_label = "umi"

Extract a sequence from the read and place it in the read name’s comment section, so a (space separated) ‘key=value’ pair is added to the read name.

Supports multiple region-extraction.

See the tag section for more tag generation options.

Mon, 01 Jan 0001 00:00:00 +0000

Head #

[[step]]
 action = "Head"
 n = 1000 # positive integer, number of reads to keep

Output just the first n molecules.

Demultiplex interaction #

If present after a demultiplex step, includes n molecules in each stream.

Mon, 01 Jan 0001 00:00:00 +0000

MergeReads #

[[step]]
 action = "MergeReads"

 segment1 = "read1" # First segment
 segment2 = "read2" # Second segment
 reverse_complement_segment2 = true # Whether to reverse complement segment2 (suggested: true)

 algorithm = "FastpSeemsWeird" # Algorithm: "fastp_seems_weird". Further algorithms are in planning
 min_overlap = 30 # Minimum overlap length required
 max_mismatch_rate = 0.2 # Maximum allowed mismatch rate (0.0-1.0) (suggested: 0.2)
 max_mismatch_count = 5 # Maximum allowed absolute mismatches (suggested: 5)
 no_overlap_strategy = "as_is" # What to do when no overlap found: "as_is" or "concatenate"
 concatenate_spacer = "NNNN" # (optional) Required if no_overlap_strategy = "concatenate". Spacer sequence to insert between reads
 spacer_quality_char = 33 # (optional) Quality score for spacer bases (suggested: 33 = Phred quality 0)

 # out_label = "merged" # (optional) output Tag label for boolean merge status

Merges paired-end reads from two segments by detecting their overlap and resolving mismatches.

Mon, 01 Jan 0001 00:00:00 +0000

Rename #

[[step]]
 action = "Rename"
 search = '(.)/([1/2])$'
 replacement = '$1 $2'

Apply a regular expression based renaming to the reads.

It is always applied to all available segments (read1, read2, index1, index2).

The example above fixes old school MGI reads for downstream processing, like fastp’s ‘–fix_mgi’ option

You can use the full power of the rust regex crate here.

Read index placeholder #

After the regex replacement runs, the special literal {{READ_INDEX}} is expanded to the running 1-based index of each logical read. When multiple segments are present (for example read1/read2 pairs), every segment for the same read receives the same index so pairs stay aligned. This makes it easy to re-sequence identifiers:

Mon, 01 Jan 0001 00:00:00 +0000

ReverseComplement #

[[step]]
 action = "ReverseComplement"
 segment = "read1" # Any of your input segments (default: read1)

Reverse-complements the read sequence (and reverses the quality).

This supports IUPAC codes (U is complemented to A, so it’s not strictly reversible). Unknown letters are output verbatim.

Useful to combine with SwapConditional.

Mon, 01 Jan 0001 00:00:00 +0000

ReverseComplementConditional #

[[step]]
 action = "ReverseComplementConditional"
 in_label = "mytag" # Boolean tag that determines whether to reverse complement
 segment = "read1" # Any of your input segments (default: read1)

Conditionally reverse-complements the read sequence (and reverses the quality) based on a boolean tag value.

For each read, if the boolean tag is true, the read is reverse-complemented. If the tag is false, the read is left unchanged.

This is useful for conditionally orienting reads based on strand information, adapter detection, or other criteria determined by prior steps.

Mon, 01 Jan 0001 00:00:00 +0000

Skip #

[[step]]
 action = "Skip"
 n = 1000 # positive integer, number of reads to skip

Skips the first n molecules.

Demultiplex interaction #

If present after a demultiplex step, skips the first n molecules in that stream.

Mon, 01 Jan 0001 00:00:00 +0000

Swap #

[[step]]
 action = "Swap"
 segment_a = "read1"
 segment_b = "read2"

Swaps exactly two segments.

Arguments segment_a/segment_b are only necessary if there are more than two segments defined in the input.

Mon, 01 Jan 0001 00:00:00 +0000

SwapConditional #

[[step]]
 action = "SwapConditional"
 in_label = "mytag" # Boolean tag that determines whether to swap
 segment_a = "read1" # Optional
 segment_b = "read2" # Optional

Conditionally swaps exactly two segments based on a boolean tag value.

For each read, if the boolean tag is true, the segments are swapped for that specific read. If the tag is false, the read is left unchanged.

This is useful for conditionally reorganizing paired-end data based on read properties, such as swapping reads based on length, quality, or other criteria determined by prior steps.

Mon, 01 Jan 0001 00:00:00 +0000

MaxLen #

[[step]]
 action = "Truncate"
 n = 100 # the maximum length of the read. Cut at end if longer
 segment = "read1" # Any of your input segments (default: read1)

Cut the read down to `n’ bases.