Skip to content

Add batch_id to pipeline run batches for tracking and filtering#150

Open
Mbeaulne wants to merge 1 commit into03-11-abstract_logic_from_create_and_bulk_createfrom
03-11-add_batch_id_to_annotations
Open

Add batch_id to pipeline run batches for tracking and filtering#150
Mbeaulne wants to merge 1 commit into03-11-abstract_logic_from_create_and_bulk_createfrom
03-11-add_batch_id_to_annotations

Conversation

@Mbeaulne
Copy link
Collaborator

@Mbeaulne Mbeaulne commented Mar 11, 2026

TL;DR

Added batch ID tracking to pipeline run batches, allowing users to identify and filter runs that were created together.

What changed?

  • Added batch_id field to BatchCreatePipelineRunsResponse that returns a unique identifier for each batch
  • Modified batch creation to generate a unique batch ID and inject it into the annotations of all runs in the batch
  • Added BATCH_ID as a new system annotation key with support for existence and equality predicates in filter queries
  • Enhanced the _mirror_system_annotations function to create searchable system annotations for batch IDs

How to test?

  • Create a batch of pipeline runs and verify the response includes a valid UUID batch_id
  • Check that all runs in a batch have the same batch_id in their annotations
  • Verify that different batches receive different batch IDs
  • Test filtering pipeline runs by batch ID using the new system annotation key

Why make this change?

This enables users to track and manage related pipeline runs that were submitted together as a batch, making it easier to monitor, filter, and operate on groups of runs that share a common context or purpose.

Copy link
Collaborator Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@Mbeaulne Mbeaulne changed the title Add batch id to annotations Add batch_id to pipeline run batches for tracking and filtering Mar 11, 2026
@Mbeaulne Mbeaulne marked this pull request as ready for review March 11, 2026 22:09
@Mbeaulne Mbeaulne requested a review from Ark-kun as a code owner March 11, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant