Skip to content

Add batch pipeline run creation endpoint#148

Open
Mbeaulne wants to merge 1 commit intomasterfrom
03-11-adds_bulk_pipeline_submission
Open

Add batch pipeline run creation endpoint#148
Mbeaulne wants to merge 1 commit intomasterfrom
03-11-adds_bulk_pipeline_submission

Conversation

@Mbeaulne
Copy link
Collaborator

@Mbeaulne Mbeaulne commented Mar 11, 2026

TL;DR

Added batch pipeline run creation functionality to allow creating multiple pipeline runs in a single API request.

What changed?

  • Added new /api/pipeline_runs/batch POST endpoint in the API router
  • Introduced BatchCreateRequest and BatchCreatePipelineRunsResponse data classes for handling batch requests and responses
  • Implemented create_batch method in PipelineRunsApiService_Sql that processes multiple pipeline run creation requests within a single database transaction
  • The batch creation method handles pipeline name extraction, execution node creation, annotations, and system annotation mirroring for each run in the batch
  • Added validation to enforce a maximum batch size of 100 runs and require at least one run per batch

How to test?

Send a POST request to /api/pipeline_runs/batch with a JSON payload containing an array of pipeline run specifications, each including root_task, optional components, and optional annotations fields. Verify that multiple pipeline runs are created successfully and returned in the response.

Why make this change?

This enables efficient creation of multiple pipeline runs simultaneously, reducing the number of API calls and database transactions needed when launching multiple related pipeline executions from the client. Bulk submission and paramater sweep are the motivation for this change.
👀 #1928 Add batch pipeline run submission with 100 run limit +156/-33
👀 #1927 Add feature flags for bulk argument upload and parameter sweep +207/-173
👀 #1903 Add parameter sweep dialog to OasisSubmitter +815/-10
👀 #1896 Refactor file extension extraction and consolidate test helpers +67/-124
👀 #1895 Add drag-and-drop file import and multi-format template export to OasisSubmitter +490/-205
👀 #1894 Add YAML file import support to task arguments dialog +17/-5
👀 #1892 Add JSON file import support to task arguments dialog +518/-17
👀 #1890 Add CSV import/export functionality to task arguments dialog +618/-11
#1889 Add bulk submission support to OasisSubmitter with comma-separated input values +428/-17

Copy link
Collaborator Author

Mbeaulne commented Mar 11, 2026

@Mbeaulne Mbeaulne force-pushed the 03-11-adds_bulk_pipeline_submission branch from 2be7906 to 90b3a27 Compare March 11, 2026 19:56
@Mbeaulne Mbeaulne changed the title Adds bulk pipeline submission Add batch pipeline run creation endpoint Mar 11, 2026
@Mbeaulne Mbeaulne marked this pull request as ready for review March 11, 2026 19:58
@Mbeaulne Mbeaulne requested a review from Ark-kun as a code owner March 11, 2026 19:58
@Mbeaulne Mbeaulne force-pushed the 03-11-adds_bulk_pipeline_submission branch from 90b3a27 to 0c04389 Compare March 11, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant