Repository navigation
ci: refresh unit shard timings from five current 12-shard runs (held-out spread 10.2% → 3.8%) - #37393
Merged
Merged
Conversation
Replaces the October 25 five-shard eight-worker baseline with run 41403569437, which includes the November 1 test-time fixes. On an independent run's measurements the projected per-shard spread falls from 12.0% to 7.4%.
Contributor
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 14 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
A single run's per-file noise left this PR's own run 21.8% apart. The importer now accepts several complete runs and averages each file; held out, the averaged baseline projects 3.8% against 10.2% for the October 25 baseline.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This refreshes the unit shard balancer's timing baseline (
config/scripts/ci-shard-timings.json, unit section). It replaces the October 25 single-run baseline with a per-file average over five complete, green 12-shard full runs, all taken after #37362. The E2E weights are byte-identical.Why
unit fullPR runs. Each has all 12 shardspassed, 0 unhandled errors, one source SHA per run, attempt 1, and Node 24.22.2 / Bun 1.4.4.Why five runs instead of one
The first commit imported a single run (41403569437) with the existing importer. This PR's own CI run under that baseline came out at 1,244–1,515 worker-seconds (21.8% spread), worse than the old baseline. That one-off result was not a fluke of that run: per-file durations vary enough between runs that a single run's noise shifts whole shards. So
ci-unit-timing-import.mjsnow accepts several run directories and averages each file across them. Every run is still validated exactly as before. The runs must be distinct and share one runtime. A single directory produces the same output as before, and a unit test covers the averaging and both rejections.Sources: 41403569437, 41406098648, 41406030210, 41405874918, 41405380104.
Balance before and after
Each baseline assigns a run's files, and every shard is evaluated with that run's own per-file measurements (this reproduces the actual measured shard loads exactly).
Held-out run 41406120225 (not part of the average):
Leave-one-out over six post-fix runs (average the other five, evaluate on the held-out run): the averaged baseline gives a 1.9–8.6% spread, against 8.8–21.2% for the October 25 baseline. Runs from before #37362 aren't comparable, because their
pty-subprocess-*files still carried 5 s waits.Actual result in this PR's CI (run 41407550754, averaged baseline; assignment
baselineSha256matches this file)The actual spread is 1,276–1,413 worker-seconds (10.8%), with jobs taking 398–442 s. On this run's measurements the October 25 baseline would have projected 1,273–1,411 (10.9%), so on this run the two baselines come out even. Per-file run-to-run noise sets a floor of roughly 5–10% that no single baseline can remove. The gain shows up in the expected, not guaranteed, spread: across the seven post-fix runs, the averaged baseline's held-out spread is 1.9–10.8% (the 10.8% being this PR's own run), against 8.8–21.2% for the October 25 baseline. The worst cases improve most.