Skip to content

ci: refresh unit shard timings from five current 12-shard runs (held-out spread 10.2% → 3.8%) - #37393

Merged
nwparker merged 2 commits into
mainfrom
ci/refresh-unit-shard-timings
Nov 2, 2026
Merged

nwparker merged 2 commits into
mainfrom
ci/refresh-unit-shard-timings

Conversation

@nwparker

@nwparker nwparker commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

This refreshes the unit shard balancer's timing baseline (config/scripts/ci-shard-timings.json, unit section). It replaces the October 25 single-run baseline with a per-file average over five complete, green 12-shard full runs, all taken after #37362. The E2E weights are byte-identical.

Why

  • Wrong shape. The October 25 baseline came from 5 eight-worker shards (run 40276310369). PRs now run 12 four-worker shards.
  • Stale file list. It was missing 1,595 currently measured files, which were weighted at the median, and still listed 374 deleted files.
  • Stale weights. It predates the November 1 fixes that removed about 390 s of 5 s waits from specific files.
  • No main source. Main no longer runs PR Checks on push after Revert "ci: full checks on main pushes, changed-file lint on PRs, Slack alert when main breaks (#37282)" #37341, so the sources are green unit full PR runs. Each has all 12 shards passed, 0 unhandled errors, one source SHA per run, attempt 1, and Node 24.22.2 / Bun 1.4.4.

Why five runs instead of one

The first commit imported a single run (41403569437) with the existing importer. This PR's own CI run under that baseline came out at 1,244–1,515 worker-seconds (21.8% spread), worse than the old baseline. That one-off result was not a fluke of that run: per-file durations vary enough between runs that a single run's noise shifts whole shards. So ci-unit-timing-import.mjs now accepts several run directories and averages each file across them. Every run is still validated exactly as before. The runs must be distinct and share one runtime. A single directory produces the same output as before, and a unit test covers the averaging and both rejections.

node config/scripts/ci-unit-timing-import.mjs RUN_DIR... config/scripts/ci-shard-timings.json

Sources: 41403569437, 41406098648, 41406030210, 41405874918, 41405380104.

Balance before and after

Each baseline assigns a run's files, and every shard is evaluated with that run's own per-file measurements (this reproduces the actual measured shard loads exactly).

Held-out run 41406120225 (not part of the average):

Baseline Max (worker-s) Min (worker-s) Spread
October 25 (run 40276310369) 1,430.1 1,297.3 10.2%
Single run 41403569437 (first commit; actual CI) 1,515.0 1,243.7 21.8%
Five-run average (this PR) 1,387.1 1,336.4 3.8%

Leave-one-out over six post-fix runs (average the other five, evaluate on the held-out run): the averaged baseline gives a 1.9–8.6% spread, against 8.8–21.2% for the October 25 baseline. Runs from before #37362 aren't comparable, because their pty-subprocess-* files still carried 5 s waits.

Actual result in this PR's CI (run 41407550754, averaged baseline; assignment baselineSha256 matches this file)

Shard 1 2 3 4 5 6 7 8 9 10 11 12
Worker-s 1,276 1,402 1,364 1,366 1,391 1,277 1,349 1,366 1,328 1,363 1,413 1,336
Job s 398 429 417 420 428 402 424 434 413 423 442 413

The actual spread is 1,276–1,413 worker-seconds (10.8%), with jobs taking 398–442 s. On this run's measurements the October 25 baseline would have projected 1,273–1,411 (10.9%), so on this run the two baselines come out even. Per-file run-to-run noise sets a floor of roughly 5–10% that no single baseline can remove. The gain shows up in the expected, not guaranteed, spread: across the seven post-fix runs, the averaged baseline's held-out spread is 1.9–10.8% (the 10.8% being this PR's own run), against 8.8–21.2% for the October 25 baseline. The worst cases improve most.

Replaces the October 25 five-shard eight-worker baseline with run 41403569437,
which includes the November 1 test-time fixes. On an independent run's
measurements the projected per-shard spread falls from 12.0% to 7.4%.
@coderabbitai

coderabbitai Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 14 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Repository UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0a781f8b-1e23-498b-bbb2-2effc0194ae7
Commits

Reviewing files that changed from the base of the PR and between 3a00c74 and 8bbea82.

Files selected for processing: 4
  • config/scripts/ci-shard-timings.json
  • config/scripts/ci-shard-timings.md
  • config/scripts/ci-unit-timing-import.mjs
  • config/scripts/ci-unit-timing-reporter.test.mjs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

A single run's per-file noise left this PR's own run 21.8% apart. The importer
now accepts several complete runs and averages each file; held out, the
averaged baseline projects 3.8% against 10.2% for the October 25 baseline.
@nwparker nwparker changed the title ci: refresh unit shard timings from a current 12-shard run (projected spread 12.0% → 7.4%) ci: refresh unit shard timings from five current 12-shard runs (held-out spread 10.2% → 3.8%) Nov 2, 2026
@nwparker
nwparker merged commit 33e7582 into main Nov 2, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant