A high-performance Python tool for parallel extraction and deduplication of COI-5P barcodes from BOLD (Barcode of Life Data Systems) FASTA files or compressed archives.
Designed to handle large datasets efficiently with multi-process support, while providing flexible output options for sequences and metadata.
BoldSeqFilter.py streamlines two critical steps in barcode data processing:
COI-5P Extraction: Automatically pulls COI-5P sequences from BOLD-formatted FASTA files (or compressed archives like .gz, .zip, .tar.gz).
Deduplication: Removes redundant sequences to ensure data uniqueness.
Scalability: Uses multi-process parallelism to speed up processing of large datasets.
Flexible Output: Generates both deduplicated FASTA sequences and structured TSV metadata for downstream analysis.
Python: 3.7+
Dependencies (install via pip):
pip install biopython pandas
For compressed file support (.gz, .zip, .tar.gz), ensure system tools like gzip, unzip, or tar are available (pre-installed on most Linux/macOS systems; Windows users may need WSL or third-party tools).
Run the tool via the command line. Basic syntax:
python BoldSeqFilter.py -i [INPUT_FILES] -o [OUTPUT_FASTA] -t [OUTPUT_TSV] [OPTIONS]
-h, --help Show this help message and exit.
-i INPUT [INPUT ...] Path(s) to input files. Supports plain FASTA files and compressed archives (.gz, .zip, .tar.gz, .tgz).
Example: -i data1.fasta data2.tar.gz
-o OUTPUT_FASTA Path to save the deduplicated COI-5P sequences (FASTA format).
Example: -o deduplicated_coi.fasta
-t OUTPUT_TSV Path to save metadata (TSV format, tab-separated). Includes sequence IDs, source info, and deduplication metrics.
Example: -t coi_metadata.tsv
--use-sqlite Use an SQLite database for deduplication/aggregation. Use case: large datasets (reduces memory usage but slows down processing). Disabled
--sqlite-db SQLITE_DB Path to the SQLite database file (required if --use-sqlite is enabled).
Example: --sqlite-db coi_dedup.db N/A (must specify)
--test Test mode: process only the first --test-n COI-5P records per worker. Use case: validate tool setup without full dataset processing. Disabled
--test-n TEST_N Number of COI-5P records to process per worker in test mode (only works with --test). 1000
--workers WORKERS Number of parallel worker processes. Recommendation: set to (CPU core count - 1) to avoid system overload. CPU core count - 1
--verbose, -v Enable detailed logging (shows file read progress, deduplication stats, and errors). Disabled\
- Basic Extraction & Deduplication
Process two input files (a FASTA and a .tar.gz archive), output deduplicated sequences and metadata:
python BoldSeqFilter.py \
-i raw_data/coi_batch1.fasta raw_data/coi_batch2.tar.gz \
-o results/deduplicated_coi.fasta \
-t results/coi_metadata.tsv- Test Mode (Quick Validation)
Verify tool setup by processing only 500 records per worker:
python BoldSeqFilter.py \
-i test_data/small_coi.zip \
-o test_results/test_dedup.fasta \
-t test_results/test_meta.tsv \
--test --test-n 500 \
-v # Show detailed logs for debugging- Large Dataset (Memory Optimization with SQLite)
Handle a massive .gz archive using SQLite to reduce memory usage, with 4 worker processes:
python BoldSeqFilter.py \
-i large_datasets/global_coi.gz \
-o results/large_dedup_coi.fasta \
-t results/large_coi_meta.tsv \
--use-sqlite --sqlite-db temp/coi_db.db \
--workers 4 \
-vInput File Compatibility: Ensure input FASTA files follow BOLD formatting (sequence headers include standard BOLD metadata, e.g., >BOLD:AAA1234|species|location).
SQLite Tradeoffs: --use-sqlite is ideal for datasets larger than ~1M sequences (reduces RAM usage) but typically adds ~20–30% processing time.
Worker Count: Avoid setting --workers to the full number of CPU cores (reserve 1 core for system tasks to prevent freezing).
Output Overwriting: The tool will overwrite existing files at the --output-fasta and --output-tsv paths. Back up data if needed.
Troubleshooting: Use -v (verbose mode) to debug issues such as missing sequences or compressed file errors.\
Filter alignment columns by gap/ambiguous fraction (memory-friendly).
A small, memory-efficient script to remove MAFFT (or other Biopython-supported) alignment columns that exceed configurable thresholds of gap / ambiguous characters. Supports streaming/block processing, multiple threshold outputs, customizable gap characters, and simple output formatting options.
FilterMafftAlignment.py is designed to:
Filter alignment columns based on the fraction of gap or ambiguous characters (e.g., N, -, ., ?).
Be memory-friendly by processing alignments in a streaming or block-wise manner instead of loading everything into RAM.
Support multiple thresholds: pass several percentage thresholds and obtain filtered outputs for each threshold for easy comparison.
Be configurable: specify which characters count as gaps/ambiguous, choose comparison mode (gt vs ge), set output wrapping, and show progress.
Python (recommended 3.7+)
Biopython (for SeqIO reading/writing)
Install Biopython:
pip install biopython
Basic command-line syntax:
FilterMafftAlignment.py [-h] [--thresholds THRESHOLDS [THRESHOLDS ...]]
[--gap-chars GAP_CHARS] [--mode {gt,ge}]
[--format FORMAT] [--line-width LINE_WIDTH]
[--show-progress]
infile outprefix
Short description: read an alignment infile, filter columns by gap/ambiguous fraction for one or more thresholds, and write filtered files using outprefix.
Argument Description
infile Input alignment file (MAFFT output or any format supported by Biopython SeqIO, e.g., FASTA).
outprefix Output prefix used to name one or more filtered output files.
-h, --help Show help message and exit. —
--thresholds THRESHOLDS [THRESHOLDS ...] One or more threshold percentages (e.g., --thresholds 10 30 50). The script will produce outputs for each threshold. (None — specify explicitly)
--gap-chars GAP_CHARS Characters counted as gap/ambiguous (string). Default: "Nn-.?". Add IUPAC or other characters if needed. "Nn-.?"
--mode {gt,ge} Comparison mode: gt delete when fraction > threshold (default), ge delete when fraction >= threshold. gt
--format FORMAT Biopython SeqIO format name for input/output (default fasta). fasta
--line-width LINE_WIDTH Wrap sequence lines in output FASTA to this width (default 80). Use 0 to disable wrapping. 80
--show-progress Print progress/summary lines during processing (useful for long runs). Disabled
- Basic filtering (single threshold)
Filter columns using a 30% threshold:
FilterMafftAlignment.py \
--thresholds 30 \
input_alignment.fasta \
results/filter_30This produces a filtered alignment file named using results/filter_30 as prefix (exact file naming depends on the script’s implementation).
- Multiple thresholds (produce multiple outputs)
Generate filtered outputs for 10%, 30%, and 50% thresholds:
FilterMafftAlignment.py \
--thresholds 10 30 50 \
input_alignment.fasta \
results/filter_multi- Custom gap characters and ge mode
Count X and - as gaps and remove columns when fraction >= 20%:
FilterMafftAlignment.py \
--thresholds 20 \
--gap-chars "NnXx-." \
--mode ge \
input_alignment.fasta \
results/filter_ge20- Disable line wrapping in FASTA output
Output sequences as single uninterrupted lines (useful for some downstream tools):
FilterMafftAlignment.py \
--thresholds 25 \
--line-width 0 \
input_alignment.fasta \
results/filter_no_wrap- Show progress during processing
Useful for large alignments to monitor progress:
FilterMafftAlignment.py \
--thresholds 30 \
--show-progress \
input_alignment.fasta \
results/filter_verboseInput format: The script relies on Biopython SeqIO. Ensure --format matches your file if it is not FASTA.
Threshold semantics: gt (>) vs ge (>=) changes whether a column whose gap fraction is exactly equal to the threshold is removed.
Gap characters: Default "Nn-.?" may not include every symbol your dataset uses to indicate missing/ambiguous data — add characters via --gap-chars as needed.
Output naming: Files are created using outprefix — the exact filename convention (e.g., suffixes containing threshold values) depends on implementation; test with small input first to confirm naming.
Memory vs time: The script emphasizes memory efficiency and so may trade off some speed. Processing time scales with alignment size and number of thresholds.
Downstream compatibility: After filtering, check sequence lengths and retained sites to ensure they meet the needs of downstream analyses (e.g., phylogeny, diversity metrics).
Rename FASTA sequence titles based on BLAST results and lineage, with optional translation checks and an Excel verification report.
A command-line utility to relabel FASTA headers using a 3-column BLAST/lineage summary, apply identity/coverage thresholds, restrict by lineage levels, optionally check protein translation, and output renamed FASTA plus logs (and an Excel check sheet when requested).
RenameByBlast.py processes a FASTA file whose headers contain sequence IDs and a BLAST summary file (plain text with three columns: origID, blastResult, Lineage). Based on identity thresholds and optional lineage filters, the script produces renamed FASTA headers that reflect taxonomic assignment at different confidence levels. It can also:
Check translation of nucleotide sequences into protein (using a specified NCBI codon table) to validate coding-frame plausibility.
Add extra information (identity, coverage, original ID, lineage) to the new FASTA header if requested.
Generate an Excel “check” sheet containing the original data plus the new title and protein/translation information (or the translated protein if translation fails).
Python 3.x
Recommended packages: pandas, openpyxl (for Excel output), biopython (if translation uses SeqIO/Seq objects). Install typical dependencies:
pip install pandas openpyxl biopython
Usage
Basic syntax (help shown by -h):
python3 RenameByBlast.py -f FASTA -b BLASTFILE [-o OUT_PREFIX] [--thr_high THR_HIGH] [--thr_mid THR_MID]
[--thr_low THR_LOW] [--cov_min COV_MIN] [--cov_max COV_MAX]
[--lineage1 LINEAGE1] [--lineage2 LINEAGE2] [--lineage3 LINEAGE3]
[--no-translate] [--codon_table CODON_TABLE]
[--tentative_tag TENTATIVE_TAG] [--include_extra_info]
[--force_overwrite] [--check]
-f FASTA, --fasta FASTA
Path to the input FASTA file (sequence headers must contain the sequence ID that matches origID in the BLAST file).
-b BLASTFILE, --blastfile BLASTFILE
Plain text 3-column file without header: origID blastResult Lineage.
origID: sequence identifier matching FASTA header ID.
blastResult: a textual BLAST hit / identity string (the script expects to parse identity and coverage from this or a fixed format).
Lineage: taxonomic lineage string (e.g., Animalia;Arthropoda;Insecta;...).
-o OUT_PREFIX Output prefix. Script writes {prefix}.renamed.fasta and {prefix}.log by default. If omitted, a default prefix or behavior depends on script.
--thr_high THR_HIGH High identity threshold (>= this → assign species). 98.0
--thr_mid THR_MID Mid identity threshold (>= this → assign genus sp.). 95.0
--thr_low THR_LOW Low identity threshold (>= this → assign genus sp. tentative). 85.0
--cov_min COV_MIN Minimum coverage cutoff (e.g., bp). Sequences below this are treated as low coverage. 500
--cov_max COV_MAX Maximum coverage cutoff. Sequences above this may be flagged. 700
--lineage1 LINEAGE1 Restrict matches to specific lineage level 1 (e.g., Insecta). Multiple values separated by ;. no restriction
--lineage2 LINEAGE2 Restrict lineage level 2 (e.g., Coleoptera). ; separated. no restriction
--lineage3 LINEAGE3 Restrict lineage level 3 (e.g., Staphylinidae). ; separated. no restriction
--no-translate Disable translation/protein check. Translation check enabled by default
--codon_table CODON_TABLE NCBI codon table number to use for translation (integer). 5 (Invertebrate Mitochondrial)
--tentative_tag TENTATIVE_TAG Tag text appended for low-similarity tentative genus assignments. "[tentative]"
--include_extra_info Include origID, identity, coverage and lineage in the new FASTA header. Disabled
--force_overwrite Overwrite existing output files. Without this, the script will error if outputs exist. Disabled
--check Generate an Excel check file containing original 3 columns + new_title + protein (or protein translation on failure). Disabled\
- Basic rename (use defaults)
python3 RenameByBlast.py -f input.fasta -b blast_summary.txt -o results/myrunGenerates results/myrun.renamed.fasta and results/myrun.log.
- Set identity thresholds and coverage window
python3 RenameByBlast.py \
-f input.fasta -b blast_summary.txt -o results/thresholded \
--thr_high 98.0 --thr_mid 95.0 --thr_low 85.0 \
--cov_min 500 --cov_max 700- Restrict to a taxonomic group and add extra info
python3 RenameByBlast.py \
-f input.fasta -b blast_summary.txt -o results/insecta_only \
--lineage1 Insecta \
--include_extra_info- Disable translation check (faster, but less validation)
python3 RenameByBlast.py \
-f input.fasta -b blast_summary.txt -o results/no_trans \
--no-translate- Generate Excel verification/check sheet and force overwrite
python3 RenameByBlast.py \
-f input.fasta -b blast_summary.txt -o results/check_run \
--check --force_overwriteThis produces an Excel file with the original three columns plus new_title and protein (or the translated protein string / failure note).
Typical outputs (names based on --out_prefix):
{prefix}.renamed.fasta — FASTA with renamed headers.
{prefix}.log — text log describing renaming decisions and summaries.
{prefix}.xlsx (if --check) — Excel file containing original columns plus new_title and protein/translation results.
BLAST file format: Ensure your BLASTFILE is exactly the expected 3-column tab-delimited format with no header. Confirm how identity and coverage are encoded in blastResult or ensure the script’s parser matches your BLAST output format.
Threshold semantics: thr_high/thr_mid/thr_low are inclusive per help (>= used for thresholds in descriptions). Confirm script behavior for edge cases (equality) if exact behavior is critical.
Coverage units: cov_min and cov_max are numeric and typically represent nucleotide positions / alignment length; check your BLAST summary units to match them.
Translation check: By default the script translates sequences (NCBI codon table 5). This helps catch frameshifts/stop codons and filter suspect sequences—disable only if unnecessary.
Lineage filtering: You can supply multiple allowed values separated by ; (e.g., --lineage1 Insecta;Arachnida). If no lineage options are given, no taxonomic filtering is applied.
Backup: The script overwrites outputs only if --force_overwrite is set. Back up original FASTA and BLAST summary before bulk runs.
Testing: Use --check on a small subset to inspect renaming logic, header format, and translation behavior before running across all data.
python remove_chinese_from_fasta.py <输入FASTA文件> <输出FASTA文件>
SQLite-based species clustering pipeline (sorted output).
A lightweight pipeline that uses vsearch for sequence clustering and an SQLite backend to manage intermediate results, enabling scalable, resumable, and sorted clustering of FASTA input.
SeqVsearch.py automates species-level (or user-defined identity-level) clustering using vsearch and records results in an SQLite database. It supports parallel workers, per-job threading, resuming from previous runs (using existing DB/TSV/parts), and configurable thresholds for triggering clustering. The pipeline is designed for robust large-dataset handling while keeping outputs reproducible and sorted.
Python 3.x
vsearch (installed and available on PATH or provided via --vsearch)
Standard Python packages (e.g., sqlite3, argparse, subprocess, possibly pandas if the script uses it for TSV export). Install vsearch (examples):
On Linux (conda):
conda install -c bioconda vsearch
Or download prebuilt binaries from the vsearch releases page.
Ensure sufficient disk space for temporary files and SQLite DB.
Basic command-line syntax:
SeqVsearch.py -i INPUT -o OUT_PREFIX [--vsearch VSEARCH] [--id ID]
[--workers WORKERS] [--threads-per-job THREADS_PER_JOB]
[--tmp-dir TMP_DIR] [--resume]
[--min-to-cluster MIN_TO_CLUSTER] [--version]
Short description: cluster sequences in INPUT FASTA with vsearch at identity --id, using multiple workers and storing state in an SQLite DB so runs can be resumed or inspected.
-i INPUT, --input INPUT Input FASTA file containing sequences to cluster.
-o OUT_PREFIX, --out-prefix OUT_PREFIX Output prefix for generated files (DB, TSV, cluster outputs).\
-h, --help Show help message and exit. —
--vsearch VSEARCH Path to vsearch executable (if not on PATH). vsearch on PATH
--id ID Clustering similarity threshold for vsearch (e.g., 0.97 for 97% identity). Script default (check script)
--workers WORKERS Number of parallel worker processes to run (controls pipeline parallelism). CPU-dependent recommendation (e.g., cores or user-specified)
--threads-per-job THREADS_PER_JOB Number of threads passed to each vsearch invocation (useful when threads-per-job * workers ≤ total cores). 1
--tmp-dir TMP_DIR Temporary directory for intermediate files and per-worker parts. System temp (or script default)
--resume Resume from existing DB/TSV/parts instead of starting from scratch. Useful after interruption. Disabled
--min-to-cluster MIN_TO_CLUSTER Minimum number of sequences required to trigger clustering on a partition/group. Prevents running vsearch for tiny partitions. Script default (e.g., 2 or user-specified)
--version Show program version and exit. —
-
Basic clustering at 97% identity
SeqVsearch.py -i input_sequences.fasta -o results/seqv_97 --id 0.97 -
Use a local vsearch binary and 4 workers
SeqVsearch.py \
-i input_sequences.fasta \
-o results/seqv_97_4w \
--vsearch /path/to/vsearch \
--id 0.97 \
--workers 4 \
--threads-per-job 2(With 4 workers × 2 threads per job = 8 threads total — plan according to available CPU cores.)
- Resume an interrupted run
SeqVsearch.py -i input_sequences.fasta -o results/seqv_97_4w --id 0.97 --resume
This will pick up from the existing database / partial outputs in results/ and avoid reprocessing completed parts.
- Only cluster partitions with at least 10 sequences
SeqVsearch.py -i input_sequences.fasta -o results/seqv_min10 --id 0.99 --min-to-cluster 10
Small partitions below the threshold will be kept as singleton groups or handled without running vsearch, depending on implementation.
Typical outputs (names depend on OUT_PREFIX and script internals):
{outprefix}.db (SQLite DB) — stores partitioning, cluster assignments, and intermediate metadata.
{outprefix}.clusters.tsv — final sorted TSV of clusters (cluster id, representative sequence, member IDs, sizes, etc.).
Per-worker or temporary files in --tmp-dir (or in a folder named by outprefix), e.g., partial TSVs or FASTA slices.
Logs printed to STDOUT/STDERR or written to {outprefix}.log (if implemented).
vsearch installation: Confirm vsearch is callable (run vsearch --version). If multiple versions exist, pass the absolute path with --vsearch.
Choosing --id: Typical species-level thresholds for COI/mitochondrial markers are 0.97–0.99 depending on taxa and aims. Adjust according to marker and dataset.
Parallelism strategy: Balance --workers and --threads-per-job so total threads ≤ available cores. I/O and disk speed also matter for performance.
Resume safety: Use --resume if previous run produced a DB/parts in the outprefix directory. Avoid manual editing of intermediate files unless you know the schema.
--min-to-cluster behaviour: This prevents invoking vsearch on extremely small groups. Check how the script treats groups below the threshold (singleton output vs skipped).
Disk & temp: For very large datasets, set --tmp-dir to a location with ample space and fast I/O (SSD preferred).
Testing: Run on a small subset first to verify output naming and behavior, especially if you plan to resume or re-run multiple times.