Skip to content

CI: refresh .test_durations from main's real timings (Part of #5133) - #5255

Merged
squid-protocol merged 1 commit into
mainfrom
churn/5133-20261016T1306Z
Oct 16, 2026
Merged

squid-protocol merged 1 commit into
mainfrom
churn/5133-20261016T1306Z

Conversation

@squid-protocol

Copy link
Copy Markdown
Owner

Part of #5133

What

.test_durations regenerated from the real per-test seconds of main run 38901664312 (ea93351d8, after #5247): the three test-times-part-* artifacts merged, 21549 tests, none unrecorded (the old file lacked 180 and held 13 stale ones).

Predicted per-part sums (least_duration, replayed offline against the real seconds)

split part 1 part 2 part 3
actual on main (balanced by the 2026-10-12 nightly cache) 906 s / 4.9 min 1278 s / 7.1 min 840 s / 4.8 min
old committed file, replayed 957 1070 999
refreshed file, replayed 1008 1008 1008

Replay used my own re-implementation of least_duration (sort descending, give each test to the lightest group); pytest-split is not installed in the box venv, so the real plugin's tie-breaking was not run. Wall scales by about 0.33 min per 100 s of test seconds, so the refreshed file predicts about 5.6 min per part, not the 7-8 min the brief gave as a target: there is no total left to spread, the suite is 3025 s of test seconds in all.

Ground truth (not fixable by this file)

mainframe-ground-truth.yml splits by explicit file lists, not durations. Summing this file over tests/cobol_mainframe/: part 1 (everything but two files, plus the ledger check) 1071 s, part 2 408 s (test_evidence_components.py has no entries here, so part 2 is understated). least_duration 2-way would be 740 / 740 s, about 7 min each. Matches 10.8 / 3.3 min wall.

Open items (workflow edits, not authorized here)

  1. ground-truth: replace the matrix tests: lists with --splits 2 --group N --splitting-algorithm least_duration (and put the ledger step on the lighter part).
  2. smoke-test full-suite-part balances by the newest test-durations-* cache (run 38247246921, 2026-10-12), not this file; the file is only the cache-miss fallback and the full-suite-gate input. The next nightly saves a fresh cache; until then the smoke parts stay at 4.9/7.1/4.8.

Sibling gaps

  • full-suite-gate OS matrix reads the committed file: gets the refresh.
  • Stale nightly durations cache feeding smoke (above).
  • ground-truth: not duration-based at all (above).

Gates

pr_gates --fast 6/6 pass; --ratchets 8/8 pass; time_ratchet.py check (reads slow marks from source): 0 over the limit, 7 warned.

Lessons for agents

  • time_ratchet.py check --times .test_durations skips the source slow-mark read and false-fails on module-scoped fixture cost; run it with no --times to check the committed file.
  • Test-times artifacts are {"seconds": {...}, "slow": ...}, not pytest-split format: take ["seconds"].

🤖 Generated with Claude Code

… seconds (Part of #5133)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@squid-protocol squid-protocol added the shepherd:merge Maintainer: merge when green (auto-merge, squash). Guide: docs/ci.md label Oct 16, 2026
@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

@squid-protocol
squid-protocol marked this pull request as ready for review October 16, 2026 13:14
@squid-protocol

Copy link
Copy Markdown
Owner Author

No description provided.

@squid-protocol
squid-protocol enabled auto-merge (squash) October 16, 2026 13:14
@squid-protocol
squid-protocol merged commit 29a8a58 into main Oct 16, 2026
43 checks passed
@squid-protocol
squid-protocol deleted the churn/5133-20261016T1306Z branch October 16, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

shepherd:merge Maintainer: merge when green (auto-merge, squash). Guide: docs/ci.md

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant