How to create an "output table" when batch processing multiple images

[tischi]

Hi,

Let's say I have a folder with images:

images
  A.tif
  B.tif
  C.tif

And I use Nextflow to batch run CellPose on it and it creates a new folder with segmentations such that I now have

images
  A.tif
  B.tif
  C.tif
segmentations
  A_seg.tif
  B_seg.tif
  C_seg.tif

What I would like to have as an additional output is the following mobie-collection.csv table (specifically for convenient visualisation of the results as a MoBIE collection table but I think something like this is also useful in general):

datasetID	uri	type
A	images/A.tif	intensities
B	images/B.tif	intensities
C	images/C.tif	intensities
A	segmentations/A.tif	labels
B	segmentations/B.tif	labels
C	segmentations/C.tif	labels

Such that in total I would have:

mobie-collection.csv
images
  A.tif
  B.tif
  C.tif
segmentations
  A_seg.tif
  B_seg.tif
  C_seg.tif

Do you have suggestions regarding the best (most generic way) to achieve this?

I guess one could add another CREATE_OUTPUT_TABLE module at the end of the workflow?

Do you think there are better practices of achieving something similar?

Related to this I am also wondering whether the input could / should already by a table?

| datasetID | uri  | type |
| -------- | ------- | -------- |
| A | images/A.tif  | intensities |
| B | images/B.tif  | intensities |
| C | images/C.tif | intensities |

If so, how to create this input table? I guess that would be a step outside / prior to the actual Nextflow workflow?

ping @krokicki @BioinfoTongLI @jluethi

[krokicki]

@tischi Most nf-core pipelines use a spreadsheet ("sample sheet") as input, so there is a lot of tooling around this already, including for validation and parsing. This then becomes the meta argument inside the pipeline that we discussed.

As an example, I found it helpful to look at the RNASEQ pipeline.

They have a schema for the input CSV here:
https://github.com/nf-core/rnaseq/blob/3.23.0/assets/schema_input.json

It's used to parse the input here, and becomes the meta dictionary:
https://github.com/nf-core/rnaseq/blob/3.23.0/workflows/rnaseq/main.nf#L138

Another way to get started with this is the nf-core pipelines create command. Even if you don't end up using their output, you can use it to understand what's needed for sample sheet parsing.

So I would do that for input, and then add a CREATE_OUTPUT_TABLE at the end as you suggested.

[tischi]

I the workflow also outputs segmentation measurements, for MoBIE I would currently need:

datasetID	uri	type	labels_table_uri
A	images/A.tif	intensities
B	images/B.tif	intensities
C	images/C.tif	intensities
A	segmentations/A_labels.tif	labels	segmentations/A_measurements.csv
B	segmentations/B_labels.tif	labels	segmentations/B_measurements.csv
C	segmentations/C_labels.tif	labels	segmentations/C_measurements.csv

...however I wonder whether this could be cleaner:

datasetID	uri	type
A	images/A.tif	intensities
B	images/B.tif	intensities
C	images/C.tif	intensities
A	segmentations/A_labels.tif	labels
B	segmentations/B_labels.tif	labels
C	segmentations/C_labels.tif	labels
A	segmentations/A_measurements.csv	measurements
B	segmentations/B_measurements.csv	measurements
C	segmentations/C_measurements.csv	measurements

I think the disadvantage of the latter may be that it's not as explicit to which label image the table belongs. It can be only inferred from the dataset ID column. I think the advantage would be that the pattern would be to just add more rows to the table for all additional outputs, which I think is simpler than adding columns.

I know this might be very project specific and just my (MoBIE specific) problem, but if anyone has general opinions, I'm happy to hear :-)

[BioinfoTongLI]

I don't think we need to standardize at this detailed level. Especially if you choose to combine this in a separate later step.
It can be a Mobie specific modules which construct this csv just for MoBie. And other manifest-generators for other platforms