bioStream

bioStream is a modular, containerized multi-omics processing backend powered by Nextflow. It orchestrates automated pipelines for bulk and single-cell genomics data — including RNA-seq, ATAC-seq, ChIP-seq, scRNA-seq, scATAC-seq, and scMultiome (scRNA+scATAC) — transforming raw public sequencing entries (SRA) into standardized, analysis-ready expression and accessibility matrices.

Features

Unified Reference Architecture: All pipelines share the same refdata-cellranger-arc root reference, ensuring perfect coordinate synchronization across bulk and single-cell modalities (STAR, RSEM, Bowtie2 indices, ENCODE blacklist, rRNA BED).
End-to-End Automation: SRA download → FastQ → QC → alignment → peak calling/quantification → MultiQC/CellRanger reports.
Modality-Specific Reports:
- Bulk (RNA/ATAC/ChIP): MultiQC reports with custom QC metrics (rRNA Ratio, Mito Ratio, NRF/PBC, FRiP)
- Single-cell: Native CellRanger HTML reports
Containerized Execution: Docker-based, all tools pre-installed in zhaoyybio/biostream:latest
Six Workflows: scRNA, scMultiome, scATAC, ATAC, ChIP, RNA

Installation

1. Clone the Repository

git clone https://github.com/your-repo/bioStream.git
cd bioStream

2. Pull Pre-built Docker Image

docker pull zhaoyybio/biostream:latest

The image includes:

Conda environment bioStream with all bioinformatics tools
CellRanger 10.0.0, CellRanger ATAC 2.2.0, CellRanger ARC 2.2.0
RSEM, STAR, Bowtie2, MACS2, deepTools, RSeQC, MultiQC
UCSC tools (gtfToGenePred, genePredToBed, bedGraphToBigWig)
SRA Toolkit, FastQC, Trim Galore, Picard

3. Install Nextflow

# Requires Java 17+
curl -s https://get.nextflow.io | bash
# Or via conda
conda install -c bioconda nextflow

Reference Data Setup

You have two options: download pre-built references from Zenodo (bulk indices), or build from scratch using 10x Genomics sources.

Option A: Download Pre-built References (Recommended)

bash download_refs.sh

This script performs two steps:

Step 1: Downloads and extracts pre-built bulk indices from Zenodo (~30GB total):

STAR index, RSEM index, Bowtie2 index
ENCODE v2 blacklists, BED12 gene models, RSeQC rRNA BED

Step 2: Prompts you to download 10x Genomics references (GEX and ARC) — required for single-cell workflows. These are NOT bundled in the Zenodo archives to avoid redundancy.

The final references/ structure will be:

references/
├── hg38/                          # Bulk indices (Zenodo)
│   ├── star/GRCh38/
│   ├── rsem/GRCh38.gtf, GRCh38.*.idx
│   ├── bowtie/GRCh38/
│   ├── bed12/GRCh38_bed12.bed
│   ├── blacklists/hg38-blacklist.v2.bed
│   ├── RSeQC/hg38_rRNA.bed
│   └── gene_anno/gene_anno_GRCh38.csv
├── hg38_10x/                      # 10x references (from 10x Genomics website)
│   ├── refdata-cellranger-arc-GRCh38-2024-A/
│   └── refdata-gex-GRCh38-2024-A/
├── mm10/                          # Bulk indices (Zenodo)
│   └── ... (same structure as hg38/)
└── mm10_10x/                      # 10x references (from 10x Genomics website)
    ├── refdata-cellranger-arc-mm10-2020-A-2.0.0/
    └── refdata-gex-mm10-2020-A/

Note: The Zenodo archives contain ONLY the bulk analytical indices. The official 10x Genomics GEX and ARC reference folders are excluded to minimize hosting redundancy and must be downloaded directly from the 10x Genomics support site. The download_refs.sh script automates this step with an interactive prompt.

Option B: Build References from Scratch

If you need a different species, or want to build all indices from 10x sources directly:

# Edit build_references.sh to select the species you want
# Then run:
bash build_references.sh

This script downloads:

10x Cell Ranger ARC reference (fasta + gtf) → generates STAR/RSEM/Bowtie2 indices
10x Cell Ranger GEX reference (for single-cell RNA workflows)
ENCODE blacklist BED
RSeQC rRNA BED

And automatically generates:

STAR index
RSEM index
Bowtie2 index
BED12 gene annotation
Gene annotation CSV

Quick Start

1. Set Up Reference Path

The pipeline automatically looks for references at ../../reference relative to the workflow directory. You can override this by setting --genomes_base:

# In nextflow.config or via CLI
--genomes_base /path/to/your/references

2. Choose a Workflow

Workflow	Use Case	Single/Multi-Cell
`scRNA`	10x single-cell RNA-seq	Single-cell
`scATAC`	10x single-cell ATAC-seq	Single-cell
`scMultiome`	10x RNA+ATAC multiome	Single-cell
`ATAC`	Bulk ATAC-seq	Bulk
`ChIP`	Bulk ChIP-seq (H3K27ac etc.)	Bulk
`RNA`	Bulk RNA-seq	Bulk

3. Prepare Your Input CSV

Each workflow requires a specific CSV format. See Examples below.

4. Run the Pipeline

cd examples/test_RNA

# Edit run.sh if needed (paths, species)
bash run.sh

Or directly with Nextflow:

nextflow run ../../workflows/RNA/ \
    --input_csv ../data/RNA_meta.csv \
    --outdir ./results/ \
    -profile docker \
    -resume

Workflows

RNA — Bulk RNA-seq

Pipeline: Download SRA → SRA to FastQ → Trim Galore → STAR alignment → RSEM quantification → RSeQC (GeneBodyCoverage, read distribution, rRNA split, Tin) → BAM to BigWig → MultiQC

Input CSV columns: Sample_ID, GSE_ID, GSM_ID, Title, Type, SRR, SRR_AWS_URL, SRR_AWS_MD5

Reference used: params.star_index, params.rsem_index, params.gtf, params.bed12, params.rRNA, params.gene_anno

Output: BAM, BigWig, RSEM genes.count matrix, isoform counts, RSeQC metrics, MultiQC report

RNA_meta.csv example:
Sample_ID,GSE_ID,GSM_ID,Title,Type,SRR,SRR_AWS_URL,SRR_AWS_MD5
ESCs_rep1,GSE93029,GSM2442725,ESCs(RNAseq)-rep1,RNA,SRR5136352,https://sra-pub-run-odp.s3.amazonaws.com/...,9da98fb806102ad960591ce115c8a44b

ChIP — Bulk ChIP-seq

Pipeline: Download SRA → SRA to FastQ → Trim Galore → Bowtie2 → Mito Ratio QC → Filter/Blacklist → BAM coverage → MACS2 (with control) → FRiP score → deepTools QC → MultiQC

Input CSV columns: Sample_ID, GSE_ID, GSM_ID, Title, Type, Antibody, Control_ID, SRR, SRR_AWS_URL, SRR_AWS_MD5

Reference used: params.bowtie2_index, params.blacklist, params.tss_bed

Output: Filtered BAM, BigWig, peaks (NarrowPeak), FRiP score, MultiQC report

Note: Control_ID links IP samples to their input controls for MACS2 callpeak.

ChIP_meta.csv example:
Sample_ID,GSE_ID,GSM_ID,Title,Type,Antibody,Control_ID,SRR,SRR_AWS_URL,SRR_AWS_MD5
ESCs_H3K27ac,GSE93029,GSM2593532,ESCs-H3K27ac-Chipseq,ChIP,H3K27ac,ESCs_input,SRR5489303,https://...,b6027c21a1f858d93213a7ba6b069448
ESCs_input,GSE93029,GSM2593533,ESCs-input-Chipseq,ChIP,input,,SRR5489304,https://...,3a2cbaee9c9510701ab8d8a07b569b37

ATAC — Bulk ATAC-seq

Pipeline: Download SRA → SRA to FastQ → Trim Galore → Bowtie2 → Mito Ratio QC → Filter/Blacklist → TN5 shift → MACS2 peak calling → BAM coverage → deepTools QC → MultiQC

Input CSV columns: Sample_ID, GSE_ID, GSM_ID, Title, Type, SRR, SRR_AWS_URL, SRR_AWS_MD5

Reference used: params.bowtie2_index, params.blacklist, params.tss_bed

Output: Filtered BAM, BigWig, peaks (NarrowPeak), FRiP score, MultiQC report with custom metrics (rRNA Ratio, Mito Ratio, NRF/PBC)

ATAC_meta.csv example:
Sample_ID,GSE_ID,GSM_ID,Title,Type,SRR,SRR_AWS_URL,SRR_AWS_MD5
ESCs_rep1,GSE93029,GSM2442665,ESCs(ATAC-seq)-rep1,ATAC,SRR5136376,https://sra-pub-run-odp.s3.amazonaws.com/...,fc4f942f9d56c5e80022f7d6751d9f1b

scRNA — 10x Single-Cell RNA-seq

Pipeline: Download SRA → SRA to FastQ → Rename/Prepare → CellRanger count

Input CSV columns: Sample_ID, GSE_ID, GSM_ID, Title, Type, Library_type, SRR, SRR_AWS_URL, SRR_AWS_MD5

Reference used: params.ref_cellranger (10x GEX reference, e.g., refdata-gex-GRCh38-2024-A)

Output: CellRanger count matrix, web_summary.html, cloupe.cloupe

Note: Each sample may have multiple SRR files (representing multiple flow cells) that get merged automatically.

scRNA_meta.csv example:
Sample_ID,GSE_ID,GSM_ID,Title,Type,Library_type,SRR,SRR_AWS_URL,SRR_AWS_MD5
Ovary_3m_1,GSE232309,GSM7325156,Ovary-3months-1,scRNA,GEX,SRR24516566,https://sra-pub-run-odp.s3.amazonaws.com/...,374e3bb6188ce736f5c0e1956f452c15

scATAC — 10x Single-Cell ATAC-seq

Pipeline: Download SRA → SRA to FastQ → Rename/Prepare → CellRanger ATAC count

Input CSV columns: Sample_ID, GSE_ID, GSM_ID, Title, Type, Library_type, SRR, SRR_AWS_URL, SRR_AWS_MD5

Reference used: params.ref_cellranger_arc (10x ARC reference for ATAC)

Output: CellRanger ATAC count matrix, web_summary.html, cloupe.cloupe

Note: Each sample may have multiple SRR files (representing multiple flow cells) that get merged automatically.

scATAC_meta.csv example:
Sample_ID,GSE_ID,GSM_ID,Title,Type,Library_type,SRR,SRR_AWS_URL,SRR_AWS_MD5
kidney_rep1,GSE172008,GSM5239693,Human kidney snATAC-seq_rep1,scATAC,ATAC,SRR14230891,https://sra-pub-run-odp.s3.amazonaws.com/...,1d865b001ec756960a50f989e28366be
kidney_rep1,GSE172008,GSM5239693,Human kidney snATAC-seq_rep1,scATAC,ATAC,SRR14230888,https://sra-pub-run-odp.s3.amazonaws.com/...,63cfd94fafe1a60c5585de2c91ecbd56
...

scMultiome — 10x Multiome (RNA + ATAC)

Pipeline: Download SRA → SRA to FastQ → Rename/Prepare → CellRanger ARC count (joint RNA+ATAC)

Input CSV columns: Sample_ID, GSE_ID, GSM_ID, Title, Type, Library_type, SRR, SRR_AWS_URL, SRR_AWS_MD5

Reference used: params.ref_cellranger_arc (10x ARC reference)

Output: CellRanger ARC count matrix with both RNA and ATAC modalities

Note: Each sample requires both GEX and ATAC libraries. The pipeline groups them by Sample_ID and generates a libraries.csv for CellRanger ARC.

scMultiome_meta.csv example:
Sample_ID,GSE_ID,GSM_ID,Title,Type,Library_type,SRR,SRR_AWS_URL,SRR_AWS_MD5
GSE240061_01,GSE240061,GSM7680744,Circadian control (Pre),ATAC,scMultiome,ATAC,SRR25516307,https://...,29308ed3df1c43e1478c34a7205d0d9f
GSE240061_01,GSE240061,GSM7680744,Circadian control (Pre),ATAC,scMultiome,ATAC,SRR25516308,https://...,0d291288ad210c075715f35e5c9c5284
GSE240061_01,GSE240061,GSM7680724,Circadian control (Pre),GEX,scMultiome,GEX,SRR25516267,https://...,953fbfa16a99bbc6fff9595f285a38fa
GSE240061_01,GSE240061,GSM7680724,Circadian control (Pre),GEX,scMultiome,GEX,SRR25516268,https://...,d34d802db83403c42570c9baf3c4045b
...

Directory Structure

bioStream/
├── bioStream.yaml              # Conda environment specification
├── Dockerfile                  # Docker image build file
├── download_refs.sh            # Download pre-built references from Zenodo
├── build_references.sh         # Build reference indices from 10x sources
├── fig/
│   ├── bioStream.png           # Architecture figure
│   └── logo.png                # logo figure
├── common/
│   ├── bin/
│   │   └── rename_fastq.py    # FastQ renaming script for single-cell
│   ├── conf/
│   │   ├── base.config        # Base Nextflow config (container, resources)
│   │   ├── genomes.config     # Reference genome paths (hg38, mm10)
│   │   ├── resources.config   # Software paths (cellranger, etc.)
│   │   └── multiqc_config.yaml# Multiqc config
│   └── modules/
│       ├── download.nf        # SRA download module
│       ├── to_fastq.nf        # SRA to FastQ conversion
│       ├── rename_fastq.nf    # FastQ preparation (single-cell + bulk)
│       ├── cellranger.nf      # CellRanger count processes
│       ├── bowtie2.nf         # Bowtie2 alignment
│       ├── trim.nf            # Trim Galore
│       ├── macs2.nf           # MACS2 peak calling
│       ├── bam_coverage.nf    # BAM to BigWig
│       ├── atac/
│       │   └── tn5_shift.nf   # TN5 shift for ATAC
│       ├── qc/
│       │   ├── fastq_qc.nf    # FastQC
│       │   ├── bam_qc.nf      # Mito Ratio, BAM filter, NRF/PBC, deepTools QC
│       │   ├── peak_qc.nf     # FRiP score
│       │   ├── rseqc.nf       # RSeQC metrics
│       │   └── summary.nf     # MultiQC summary
│       ├── rna/
│       │   ├── star.nf            # STAR
│       │   └── quantification.nf  # RSEM, featureCounts, R quantification
│       └── utilities/
│           ├── versions.nf        # Software Version
│           └── printSummary.nf    # Nextflow Summary
│
├── workflows/
│   ├── RNA/                    # Bulk RNA-seq workflow
│   │   ├── main.nf
│   │   └── nextflow.config
│   ├── ATAC/                   # Bulk ATAC-seq workflow
│   │   ├── main.nf
│   │   └── nextflow.config
│   ├── ChIP/                   # Bulk ChIP-seq workflow
│   │   ├── main.nf
│   │   └── nextflow.config
│   ├── scRNA/                  # 10x scRNA workflow
│   │   ├── main.nf
│   │   └── nextflow.config
│   ├── scATAC/                 # 10x scATAC workflow
│   │   ├── main.nf
│   │   └── nextflow.config
│   └── scMultiome/             # 10x multiome workflow
│       ├── main.nf
│       └── nextflow.config
│
└── examples/
    ├── test_RNA/
    │   └── run.sh
    ├── test_ChIP/
    │   └── run.sh
    ├── test_ATAC/
    │   └── run.sh
    ├── test_scRNA/
    │   └── run.sh
    ├── test_scMultiome/
    │   └── run.sh
    ├── test_scATAC/
    │   └── run.sh
    └── data/
        ├── scRNA_meta.csv
        ├── scMultiome_meta.csv
        ├── scATAC_meta.csv
        ├── ATAC_meta.csv
        ├── ChIP_meta.csv
        └── RNA_meta.csv

Examples

Each example directory contains:

run.sh — The run script for that workflow
../data/*.csv — Sample metadata CSV files

Running All Examples

# Bulk RNA
cd examples/test_RNA && bash run.sh

# ChIP-seq
cd examples/test_ChIP && bash run.sh

# Bulk ATAC
cd examples/test_ATAC && bash run.sh

# scRNA
cd examples/test_scRNA && bash run.sh

# scATAC
cd examples/test_scATAC && bash run.sh

# scMultiome
cd examples/test_scMultiome && bash run.sh

Modifying Examples

Edit run.sh in each directory to customize:

--input_csv — Path to your metadata CSV
--outdir — Output directory
--species — GRCh38 or mm10
-profile — docker or singularity

Output Structure

Each pipeline outputs to the specified --outdir:

Bulk RNA-seq

outdir/
├── bigwig/                         # Coverage tracks
│   └── {sample}_Aligned.sortedByCoord.out.bw
├── downloads/                      # SRA Files
├── quantification/                 # Quantification
│   ├── featurecounts/              # featurecounts Quantification
│   │   ├── gene_counts.csv
│   │   └── gene_counts_ensg.csv
│   └── rsem/                       # RSEM Quantification
│       ├── gene_counts.csv
│       ├── gene_counts_ensg.csv
│       ├── gene_fpkm.csv / gene_fpkm_ensg.csv
│       ├── gene_tpm.csv / gene_tpm_ensg.csv
│       └── transcript_*.csv
└── report/
    └── BioStream-Analysis-Report_multiqc_report.html

Bulk ChIP-seq

outdir/
├── bam/                            # Filtered, deduplicated BAM files
│   ├── {sample}.rmdup.sorted.bam
│   └── {sample}.rmdup.sorted.bam.bai
├── bigwig/                         # Coverage tracks
│   └── {sample}.bw
├── downloads/                      # SRA Files
├── peak/                           # MACS2 peak calls
│   ├── {sample}_peaks.narrowPeak
│   └── {sample}_summits.bed
└── report/
    └── BioStream-Analysis-Report_multiqc_report.html

Bulk ATAC-seq

outdir/
├── bam/                            # TN5-shifted, filtered BAM files
│   ├── {sample}.shifted.bam
│   └── {sample}.shifted.bam.bai
├── bigwig/                         # Coverage tracks
│   └── {sample}.bw
├── downloads/                      # SRA Files
├── peak/                           # MACS2 peak calls
│   ├── {sample}_peaks.narrowPeak
│   └── {sample}_summits.bed
└── report/
    └── BioStream-Analysis-Report_multiqc_report.html

All bulk workflows embed custom QC metrics (rRNA Ratio, Mito Ratio, NRF/PBC, FRiP) directly into the MultiQC HTML report.

Single-Cell Workflows (scRNA / scATAC / scMultiome)

outdir/
└── cellranger_count/
    └── {sample_id}/
        └── outs/
            ├── web_summary.html     # CellRanger interactive QC report
            ├── cloupe.cloupe        # Loupe browser file
            └── ...                  # count matrix, feature barcode matrix, etc.

License & Contact

License: Distributed under the MIT License. Feel free to use, modify, and distribute the platform for both academic and commercial applications.
Contact & Issues: For bug reports, feature requests, or collaboration inquiries, please open a GitHub Issue directly in this repository.

Name		Name	Last commit message	Last commit date
Latest commit History 1 Commit
common		common
examples		examples
fig		fig
workflows		workflows
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
bioStream.yaml		bioStream.yaml
build_references.sh		build_references.sh
download_refs.sh		download_refs.sh

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

bioStream

Table of Contents

Features

Installation

1. Clone the Repository

2. Pull Pre-built Docker Image

3. Install Nextflow

Reference Data Setup

Option A: Download Pre-built References (Recommended)

Option B: Build References from Scratch

Quick Start

1. Set Up Reference Path

2. Choose a Workflow

3. Prepare Your Input CSV

4. Run the Pipeline

Workflows

RNA — Bulk RNA-seq

ChIP — Bulk ChIP-seq

ATAC — Bulk ATAC-seq

scRNA — 10x Single-Cell RNA-seq

scATAC — 10x Single-Cell ATAC-seq

scMultiome — 10x Multiome (RNA + ATAC)

Directory Structure

Examples

Running All Examples

Modifying Examples

Output Structure

Bulk RNA-seq

Bulk ChIP-seq

Bulk ATAC-seq

Single-Cell Workflows (scRNA / scATAC / scMultiome)

License & Contact

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages