Add optional total_count field to ListPipelineJobsResponse#167
Open
Add optional total_count field to ListPipelineJobsResponse#167
Conversation
Mbeaulne
changed the title
Mbeaulne
force-pushed
the
03-13-add_total_counts_to_the_runs_endpoint
branch
from
d45706d to
9d010d9
Compare
Mbeaulne
changed the title
morgan-wowk
reviewed
Mar 13, 2026
| total_count = session.scalar( | ||
| sql.select(sql.func.count()) | ||
| .select_from(bts.PipelineRun) | ||
| .where(*where_clauses) |
Collaborator
There was a problem hiding this comment.
At a glance this looks like it will only return you the count for the current page. Can you confirm?
Collaborator
There was a problem hiding this comment.
Although, your tests seem to suggest otherwise
Collaborator
There was a problem hiding this comment.
Oh I see what's happening. This is based on limit + offset. Which will soon change:
https://app.graphite.com/github/pr/TangleML/tangle/131?ref=gt-pasteable-stack
I guess, let's adjust this solution accordingly
morgan-wowk
approved these changes
Mar 13, 2026
Ark-kun
reviewed
Mar 14, 2026
| ).all() | ||
| ) | ||
|
|
||
| total_count = None |
Contributor
There was a problem hiding this comment.
This looks problematic as this makes queries about 4000+x slower. Basically, we:
- Run same query twice
- The 2nd time we run it without any limits.
This is really bad for query performance.
As the DB grows, these queries become slower and slower. For some heavy queries like substring match (search by name), the slowdown will be immense.
Normal queries search for "First 10 matches", scanning the DB until 10 matches are found. But the new "total_count" query always scans the whiole DB (40000+ runs).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Added
total_countfield to theListPipelineJobsResponseto provide the total number of pipeline runs matching the query criteria when requested.What changed?
total_count: int | None = Nonefield toListPipelineJobsResponseclassinclude_total_count: bool = Falseparameter to thelistmethod inPipelineRunsApiService_Sqlinclude_total_countis True, the method executes a count query usingsql.func.count()with the same filter conditions as the main querytotal_countis populated in the response object when the parameter is enabledHow to test?
Run the existing test suite which now includes comprehensive tests for the
total_countfunctionality:test_list_total_count_not_included_by_default- verifies count is None by defaulttest_list_total_count_empty- verifies count is 0 when no runs existtest_list_total_count_matches_results- confirms count matches actual resultstest_list_total_count_with_pagination- ensures count remains consistent across paginated requeststest_list_total_count_with_filter- validates count respects filter conditionsWhy make this change?
This enhancement allows clients to optionally retrieve the total number of pipeline runs that match their query criteria, which is essential for implementing proper pagination controls and displaying accurate result counts in user interfaces. The feature is opt-in to avoid performance overhead when the total count is not needed.