Lack of determinism in results across identical runs

[alvaropvr]

Description of the bug

I’ve noticed a concerning lack of reproducibility when running the pipeline. Even when using the exact same input data and parameters across identical runs, the results change significantly. In some cases, I’ve seen a gene's FDR jump from 0.06 to 0.6 between two runs without a single change in the configuration.

I first tracked this down to the CRISPRCleanR module, where the output files differ every time despite keeping parameters like min_reads or min_targeted_genes constant. This inconsistency then propagates downstream, making the final MAGeCK rankings and statistics completely unreliable for a stable analysis.

It seems like there are stochastic processes in the R scripts or the MAGeCK steps that aren't being pinned down with a fixed seed. Is there any way to set a seed or any specific configuration to ensure the pipeline behaves deterministically? I’d like to know if this is a known issue or if there's a workaround to get consistent results across runs.

Command used and terminal output

Relevant files

No response

System information

Singularity
MAGECK RRA & MLE

[matbonfanti]

Thanks for reporting this. Reproducibility is definitely an important concern, and unfortunately this is something that has been observed before.

I’ve spent some time in the past trying to improve the determinism of the pipeline, mainly in the context of making the test suite reproducible. While I was able to achieve some marginal improvements, I could not fully stabilize all components. In particular, I noticed that:

• drugZ appears to be non-deterministic, and I was not able to identify a reliable way to make its results fully reproducible.
• The MAGeCK MLE test also showed small run-to-run variations, and again I couldn’t find a way to fully eliminate them.
• Regarding CRISPRcleanR, I personally did not observe major reproducibility issues in the limited testing I performed. For example, the test_screening test (which includes CRISPRcleanR) seemed to give reproducible results at the time, as far as I remember. However, I did not perform an extensive comparison across many runs.

So at the moment, some degree of stochasticity in the pipeline is unfortunately a known limitation, and there is no fully reliable way (that I am aware of) to enforce strict determinism via a global seed or configuration.

That said, any additional investigation, feedback, or potential solutions in this direction would be extremely valuable and very welcome. If you manage to identify specific sources of randomness or possible fixes, that would be a great contribution to improving the pipeline.

[alvaropvr]

Thanks for the insights, Mat. I want to clarify that while I noticed the randomness clearly in the CRISPRCleanR output, I cannot rule out that it starts in even earlier modules. What is certain is that in CRISPRCleanR, the inconsistency becomes very obvious: the final matrix filters out different sgRNAs in consecutive runs even when every output and min_reads and min_targeted_genes parameters remain constant.

Regarding the setup, I’m only providing the library annotation and $min_reads and $min_targeted_genes parameters but, for example, I noticed the nf-core documentation doesn't explicitly ask for a 'plasmid' sample for CRISPRCleanR. Since this tool usually filters reads based on this baseline sample, could the lack of a designated reference be causing the module to handle filtering inconsistently? May my runs be lacking any important parameter that I should implement at this point?

I also suspect that this randomness might be inherent to the underlying packages (MAGeCK MLE/RRA, DrugZ, CRISPRCleanR) rather than the pipeline itself. If these tools don't have their internal seeds pinned, the pipeline will naturally inherit that stochastic behavior.

In a small-scale library like mine, these variations are not just 'noise'—they are enough to swing a gene from a significant FDR of 0.06 to a completely non-significant 0.6. I am not sure if this is an inherent issue with the pipeline’s determinism or if I am missing a crucial parameter in my configuration that could stabilize this.

[fjcobu14]

Related PubMed reference: PMID 42285217 - "Therapeutic applications of gene editing using CRISPR-Cas9 in the posterior segment: Review of the literature." This article discusses CRISPR-Cas gene editing programs including EDIT-101, EDIT-103, and HG202 for ophthalmic applications, which may be relevant to determinism concerns in CRISPR analysis pipelines.