Skip to content

Feature request: sequential task execution that allows passing context between multiple tasks #96

Open

Description

@reesekneeland

Hello!

I am curious how exca might help facilitate multi-stage pipelines where multiple computationally intense operations must be run sequentially on the same data, for example, a pipeline that does data preprocessing, model training, and an analysis of the results. This order of operations would greatly benefit from intermediate caching (so multiple models don't each need to re-preprocess the data, and multiple analysis operations don't need to retrain the model), and my central question is: what the best way to implement such a pipeline is using the exca framework to take advantage of exca's caching functionality?

To layout some possible implementations I am thinking about to solve the problem:

Idea 1:

If tasks are able to be allocated inside of other tasks (an assumption I am not sure about, in my local testing it seems I can allocate "slurm" tasks within a "local" task, but am having issues nesting distributed slurm tasks), perhaps the whole pipeline could work in a nested manner, where the last operation constitutes the outer task (in this example the analysis step), and so the tasks for preprocessing and training execute within this task and return their outputs inside the execution stream for the current step.

The main problem I see with this approach is that the resources for the later steps must be allocated up front, and sit idle waiting for the initial pipeline steps to compute, which is not desirable. The implementation for this approach would also be quite clunky, as you would essentially have to write the pipeline backwards.

Idea 2:

After a step in the pipeline finishes executing (and caching), it would return the path to the cached CacheDict file (I'm not quite sure how to access this from the TaskInfra object or from within the task, but it definitely looks possible). The outer code execution stream would then take that path, and add it to the config for a future step, essentially telling it to "load this file". Future steps would manually initialize the CacheDict object to grab the data from the previous step.

The problems of this approach are that it relies on manually passing around the path to the cached data objects, modules would have the somewhat fragile interface of requiring one of these deep cache directory paths in order to function properly, and we must still perform the I/O operations of saving to and reading from the disk in between steps (this may be unavoidable when moving between different slurm instances though.)

Idea 2 seems the most straightforward for what I am trying to do, I understand that the obvious problem with allowing command line arguments in this framework (to easily pass data around) is that it makes it much more difficult to cache the output, as the output will change with different command line inputs and so any consistent caching system must also hash the command line arguments, which would be non-trivial when they contain complex data structures. Please let me know if this is something you have thought about or if you have any additional suggestions I might try!

Activity

  1. reesekneeland commented on Aug 14, 2025

    @reesekneeland
    Author

    Answered in the CCN Tutorial today, looks like Idea 1 is the standard way of doing it, you can nest tasks with infra allocations, and the outer tasks just hold their resources while they wait for the inner one to finish.

  2. reesekneeland commented on Aug 14, 2025

    @reesekneeland
    Author

    Reopening so this issue can serve as a proper feature request for native sequential task execution.

  3. changed the title [-]Question about passing context between multiple tasks[/-] [+]Feature request: sequential task execution that allows passing context between multiple tasks[/+] on Aug 14, 2025
  4. jrapin commented on Aug 19, 2025

    @jrapin
    Contributor

    but am having issues nesting distributed slurm tasks

    what were the issues? those work... at least in simple cases

    Idea 1

    Yes that's mostly what we do, out of convenience because we tend to allocate the resources necessary for step 1 (eg several GPU workers to process many images), wait for full completion and allocate resources for step 2 (eg multiple cpu workers to compute correlations) etc.

    Idea 2

    what would be a good API is not completely clear to me, passing around cache paths does not seem convenient (in some cases, there are several caches in the same file...). At some point we discussed some "pipeline" API within the team, but allocating resources becomes cumbersome, at least in the case of MapInfra as it's not clear if we should parallelize for each step, or all step together etc (and each step does not necessarily needs the same resources)

    Somehow that also reminds me of APIs such as in ray or dask.distributed that can spawn a subcluster of machines where data can be sent on the fly (and the cluster scales up and down on its own). I think that we could have some new infra that uses it, but it has drawbacks as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions