How to handle module specific parameters

[tischi]

Some modules accept module specific parameters, e.g. --diameter for CellPose.

Maybe I am overlooking it, but currently for the cellpose module in our repo I could not find a documentation for how to specify such parameters.

I had a look at the nf-core molkart pipeline and they handle this in two places, namely nextflow.config and modules.config:

• https://github.com/nf-core/molkart/blob/master/nextflow.config#L19
• https://github.com/nf-core/molkart/blob/master/conf/modules.config#L205

In this documentation they seem to suggest to have a configs/cellpose.config file that could be included in the nextflow.config via includeConfig 'configs/cellpose.config'.

I am wondering now whether our modules repositories could somewhere contain for each module such a module specific .config file that one could copy when installing/using this module a pipeline? In my mind that would make it much easier to know which additional module specific parameters one can use.

ping @krokicki @BioinfoTongLI

[BioinfoTongLI]

I believe, in general, regardless the algorithm, the optional (the ones not explicitly listed in the input section) parameters should go with ext.args = xxxx. And that should apply to all modules.

I am personally putting examples directly in the main.nf.test.
https://github.com/cellgeni/nf-modules/blob/main/modules/sanger-cellgeni/imaging/tiledspotiflow/tests/main.nf.test#L18-L22
but in term of general options, I don't have it now. The easiest way for me would be starting the container the run the CLI with --help flag.

[tischi]

And how do you feel about adding a configs folder to our modules repositories, with one .config file for each module?

[BioinfoTongLI]

I think it wont' hurt. But what exactly do you want to put inside it?

[tischi]

That's what they did in molkart. It seems that they are prepending <module>_ to the parameters and convert those to actual module parameters in the config; I guess this is to avoid potential clashes with parameters of other modules?

[BioinfoTongLI]

I'd say you can do that. Only when you have to pass it through nextflow. Otherwise, you can directly do:

process {
    withName: CELLPOSE {
        ext.args = <all my cellpose params>
    }
    withName: MYOTHERMODULE {
        ext.args = <all my other params>
    }
}

and they won't clash. AFAIK.
Also, I don't think they are exposing all possible parameters there. so, there will still be missing params from that config. Thus, it doesn't replace the proper read-the-doc webpage.
My personal opinion will be: yes, we can put something there as the default value if they are missing.
But if that's all it meant to serve. Why not write the default value in your executable script directly? python/java etc.

[tischi]

The other thing I'm thinking about is the following. If we want to put an AI on top of our modules to automatically compose workflows, it should be aware of some of these parameters. For instance, if the prompt is to create a workflow for 3D segmentation, it probably will need to put the --do3D flag of CellPose somewhere in the main.nf, right?
To do so, it needs to know that this flag exists. We could now either rely on the AI, or in fact any other user, of the modules to be smart enough to look that up somewhere. Or we could conveniently provide this information somewhere within the repository where it's easy to find.

[krokicki]

@tischi That is my main concern with Nextflow/nf-core modules, they are not robustly parameterized for AI workflows. Parameters get specified at the pipeline level (in nextflow_schema.json) but only those that the pipeline author decides to use.

As Tong said, optional parameters go in ext.args (see optional command arguments), which makes a lot of sense given how pipelines are currently written, but it doesn't support AI workflows very well.

I think that the natural place to document optional parameters would be to extend meta.yaml, and add another section for parameters that can be specified in ext.args (e.g. --do3D).

[BioinfoTongLI]

After using codex (havn't tried claude which I heard is even better/cleverer), I don't think the modern AI tools requires explicit statement for parameters. I'd say as long as you write your tool specific documentations properly (and make them findable somewhere). I bet these agents can do the rest of the work.
If that's the case, maybe simply using pydantic (or equivalent for Java/other languages) is good enough.

[krokicki]

@BioinfoTongLI I agree about AI tool capabilities, but that kind of workflow will be inefficient because the agent will first need to launch subagents to search for the tools, read the documentation, and extract the relevant information. Errors are also more likely to happen (e.g. if it finds documentation for the wrong version of the tool). It can fix all of this and recover, but it will waste a lot of tokens and time doing so.

On the other hand, everything is still changing very rapidly, and it doesn't necessarily make sense to try to optimize everything upfront in a changing landscape.