Tag Generation on mbf-fastq-processor documentation

Mon, 01 Jan 0001 00:00:00 +0000

ExtractAnchor #

Extract regions relative to a previously tagged anchor position.

# First create an anchor tag. Iupac, regex, ExtractRegion, your choice.
[[step]]
 action = "ExtractIUPAC"
 search = "CAYA"
 label = "anchor_tag"
 segment = "read1"
 anchor = "Anywhere"
 max_mismatches = 0

# Then extract relative to that anchor
[[step]]
 action = "ExtractAnchor"
 label = "mytag"
 input_label = "anchor_tag" # tag that provides the anchor position
 regions = [[-2, 4], [4, 1]] # [start, length] pairs relative to anchor
 region_separator = "_" # (optional) separator between regions

This transformation uses the leftmost position of a previously established tag as the anchor point and extracts specified regions relative to that position.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLength #

Extract the length of a read as a tag.

[[step]]
 action = "ExtractLength"
 label = "mytag"
 segment = "read1" # Any of your input segments, or 'All'

This transformation creates a tag containing the length of the specified read.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegex #

Extract a regexp result. Stores an empty string if not found.

[[step]]
 action = "ExtractRegex"
 label = "mytag"
 search = "^CT(..)CT"
 replacement = "$1" # standard regex replacement syntax
 segment = "read1" # Any of your input segments

This transformation searches for a regular expression pattern in the specified read and extracts the matching portion as a tag.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegion #

Extract a fixed position region.

[[step]]
 action = "ExtractRegion"
 start = 5
 length = 8
 segment = "read1" # Any of your input segments
 label = "umi"

This transformation extracts a fixed-length region from the specified read at a given position and stores it as a tag.

Use ExtractRegions if your region is actually multiple regions (possibly from different segments).

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegions #

Extract from multiple fixed position regions.

[[step]]
 action = "ExtractRegions"
 regions = [
 {segment = "read1", start = 0, length = 8},
 {segment = "read1", start = 12, length = 4},
 ]
 label = "barcode"

This transformation extracts multiple fixed-length regions from reads and concatenates them into a single tag.

ExtractRegions with only one region are exactly equivalent to ExtractRegion.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractRegionsOfLowQuality #

Extract regions (min size 1 bp) where bases have quality scores below threshold.

[[step]]
 action = "ExtractRegionsOfLowQuality"
 segment = "read1" # Any of your input segments
 min_quality = 60 # Quality threshold (Phred+33)
 label = "low_quality_regions"

This transformation scans through quality scores of the specified segment and identifies contiguous regions where quality scores are below the specified threshold. Each low-quality region becomes a tagged region with location information (start position and length).

Parameters #

segment: Which read to analyze for low-quality regions
min_quality: Quality score threshold using Phred+33 encoding. See Phred quality score for ASCII character mapping
label: Tag name to store the extracted regions

Example #

With min_quality = 60 (ASCII ‘<’), any bases with quality scores below ‘<’ will be identified as low-quality regions. This is useful for masking or filtering poor-quality sequences.

Mon, 01 Jan 0001 00:00:00 +0000

TagOtherFileByName #

Mark reads based on wether names are present in another file.

[[step]]
 action = "TagOtherFileByName"
 segment = "read1" # which segment's name are we using
 label = "present_in_other"
 filename = "names.fastq" # Can read fastq (also compressed), or sam/bam files
 false_positive_rate = 0.01 # false positive rate (0..1)
 seed = 42 # seed for randomness
 ignore_unaligned = false # in case of BAM/SAM, whether to ignore unaligned reads
 fastq_readname_end_char = " " # (optional) char (byte value) at which to cut input fastq read names before comparing. If not set, no cutting is done.
 reference_readname_end_char = "/" # (optional) char (byte value) at which to cut reference read names before storing them.

This step marks reads by comparing their names against names from another file.

Mon, 01 Jan 0001 00:00:00 +0000

TagOtherFileBySequence #

Marks reads based on wether sequences are present in another file.

[[step]]
 action = "TagOtherFileBySequence"
 label = "present_in_other_file"
 filename = "sequences.fastq" # fastq (also compressed), or sam/bam files
 segment = "read1" # Any of your input segments
 false_positive_rate = 0.01 # false positive rate (0..1)
 seed = 42 # seed for randomness

This step annotates reads by comparing their sequences against sequences from another file.

Please note our remarks about cuckoo filters.

Extract IUPAC

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPAC #

[[step]]
 action = "ExtractIUPAC"
 label = "mytag"
 anchor = 'Left' # Left | Right | Anywhere
 search = "CTN" # what we are searching
 segment = 'read1' # Any of your input segments

Search and extract a sequence from the read, defined by a IUPAC string.

If anchor = ‘Anywhere’, ExtractIUPAC will find the left most occurance.

Extract IUPAC suffix

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACSuffix #

[[step]]
 action = "ExtractIUPACSuffix"
 label = "mytag"
 query = "AGTCA" # the adapter to trim. Straigth bases only, no IUPAC.
 segment = "read1" # Any of your input segments (default: read1)
 min_length = 3 # uint, the minimum length of match between the end of the read and
 # the start of the adapter
 max_mismatches = 1 # How many mismatches to accept

Find a potentially truncated IUPAC string sequence at the end of a read.

Simple comparison with a max mismatch hamming distance, requiring only the first min length bases of the query to match at the end of the read.

Extract IUPAC with Indels

Mon, 01 Jan 0001 00:00:00 +0000

ExtractIUPACWithIndel #

[[step]]
 action = "ExtractIUPACWithIndel"
 label = "adapter"
 search = "AGTC" # IUPAC pattern to align against
 max_mismatches = 1 # allowed substitutions (IUPAC-aware)
 max_indel_bases = 1 # total insertions + deletions allowed
 max_total_edits = 2 # optional overall edit ceiling
 anchor = 'Anywhere' # Left | Right | Anywhere
 segment = 'read1' # defaults to read1

Locate an IUPAC pattern even when the read contains small insertions or deletions relative to the pattern. The extractor performs a semiglobal alignment (pattern vs. read segment) using IUPAC-aware scoring and returns the aligned span as a location tag.

Mon, 01 Jan 0001 00:00:00 +0000

ExtractLowComplexity #

[[step]]
 action = "ExtractLowComplexity"
 label = "complexity"
 segment = "read1" # Any of your input segments, or 'All'

Calculate read complexity, based on the percentage of bases that are changed from their predecessor.

A good filter value might be 0.30, which means 30% complexity is required. See FilterByNumericTag.

Corresponding options in other software #

fastp: -low_complexity_filter

Mon, 01 Jan 0001 00:00:00 +0000

TrimQualityEnd #

[[step]]
 action = "ExtractLowQualityEnd"
 label = "low_quality_ends"
 min_qual = 20 # u8, minimum quality to keep (in whatever your score is encoded in)
 # either a char like 'A' or a number 0..128 (typical phred score is 33..75)
 segment = "read1" # Any of your input segments

Define a region of low quality bases at the end of reads.

Corresponding options in other software #

Trimmomatic: TRAILING (if paired with TrimAtTag)

Mon, 01 Jan 0001 00:00:00 +0000

TrimQualityStart #

[[step]]
 action = "ExtractLowQualityStart"
 min_qual = 20 # u8, minimum quality to keep (in whatever your score is encoded in)
 # either a char like 'A' or a number 0..128 (typical phred score is 33..75)
 segment = "read1" # Any of your input segments
 label = "bad_starts"

Define a region with low quality bases (below threshold) at steart of read.

Corresponding options in other software #

Trimmomatic: LEADING (if combined with TrimAtTag)

Mon, 01 Jan 0001 00:00:00 +0000

ExtractMeanQuality #

We don’t support calculating the ‘average quality’.

This is typically a bad idea, see https://www.drive5.com/usearch/manual/avgq.html for a discussion of the issues.

To illustrate, 140 x Q35 + 10 x Q2 reads have an ‘average’ phred of 33, but 6.4 expected wrong bases A read with 150 x Q25 has a much wores ‘average’ phred of 25, but a much lower expected number of errors at 0.5!

Corresponding options in other software #

Trimmomatic: AVGQUAL:
fastp: –average_qual

Mon, 01 Jan 0001 00:00:00 +0000

ExtractPolyTail #

[[step]]
 action = "ExtractPolyTail"
 label = "tag-label"
 segment = "read1" # Any of your input segments (default: read1)
 min_length = 5 # positive integer, the minimum number of repeats of the base
 base = "A" # one of AGTCN., the 'base' to trim (or . for any repeated base)
 max_mismatch_rate = 0.1 # float 0.0..=1.0, how many mismatches are allowed in the repeat
 max_consecutive_mismatches = 3 # how many consecutive mismatches are allowed

[[step]]
 action = "TrimAtTag"
 label ="tag-label"
 direction = "End"
 keep_tag = false

Identify either a specific letter (AGTC or N) repetition, or any base repetition (base = ‘.’) at the end of the read.

Mon, 01 Jan 0001 00:00:00 +0000

#ExtractQualifiedBases

[[step]]
 action = "ExtractQualifiedBases"
 min_quality = 30 # the quality value >= which a base is qualified 
 # In your phred encoding. Typically 33..75
 # a byte or a number 0...255
 segment = "read1" # Any of your input segments, or 'All'
 label = "tag_name"

Calculate by the percentage of bases that are ‘unqualified’, that is below a user defined threshold.

Corresponding options in other software #

fastp : –qualified_quality_phred / –unqualified_percent_limit (if combined with FilterByNumericTag)

Mon, 01 Jan 0001 00:00:00 +0000

FilterDuplicates #

[[step]]
 action = "TagDuplicates"
 false_positive_rate = 0.00001 #
 # the false positive rate of the filter.
 # 0..1
 seed = 59 # required!
 segment = "All" # Any of your input segments, or 'All'
 label = "dups"

[[step]]
 action = "FilterByBoolTag"
 label = "dups"
 keep_or_remove = "Remove" # Keep|Remove

Tag duplicates (2nd onwards) from the stream using a Cuckoo filter.

That’s a probabilistic data structure, accordingly there’s a false positive rate, and a tunable memory requirement.

Needs a seed for the random number generator, and a segment to know which reads to consider for deduplication (filters the complete molecule, like all other filters of course).

Extract GC Content

Mon, 01 Jan 0001 00:00:00 +0000

ExtractGCContent #

[[step]]
 action = "ExtractGCContent"
 segment = "read1" # Any of your input segments, or 'All'
 label="gc"

Count what percentage of bases are GC (as opposed to AT). Non-AGTC bases (e.g. N) are ignored in both the numerator and denominator.

Extract N Count

Mon, 01 Jan 0001 00:00:00 +0000

ExtractNCount #

[[step]]
 action = "ExtractNCount"
 segment = "read1" # Any of your input segments, or 'All'
 label="ncount"

Count how many N are present in the read

Corresponding options in other software #

fastp: –n_base_limit (if combined with FilterByNumericTag)