Repository navigation
ci: evidence refresh's Db2 job in 4 shards + a records leg (Part of #4863) - #4866
Merged
Merged
Conversation
… a ratchet job (Part of #4863) The Db2 job was the refresh's critical path (~34 min vs <=16 for every other job): 38 cases / 673 scenarios at ~9 s per scenario per slot on one runner that also hosts Db2, then the stale Db2 records re-proved after the sweep. Now 5 runners, each with its own Db2: proof_sweep --db2-only --shard I/4 --no-ratchet, and the records beside them. A new thin `db2` job merges the shards with `--aggregate --expect db2` (a missing shard fails it) and keeps the outputs and the `db2-results` artifact the refresh job takes; the refresh job passes every Db2 shard's sweep.json to the ledger update. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MhEx8PwSow3Pyfg6J4KZ97
Owner
Author
|
No description provided. |
2 similar comments
Owner
Author
|
No description provided. |
Owner
Author
|
No description provided. |
squid-protocol
marked this pull request as ready for review
October 10, 2026 23:51
squid-protocol
enabled auto-merge (squash)
October 10, 2026 23:51
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #4863 (step 6: a fast Db2 job).
Why. The Db2 job is the refresh's critical path. On the 10/10 nightly (run 38043242484) it took 34 min, against at most 16 for every other job. 25.5 min of that is the sweep: 38 cases / 673 scenarios at
--jobs 4on one 4-vCPU runner that also hosts Db2, about 9 s per scenario per slot (mostly the fixed cost of each Db2 CLP call and JVM start). After the sweep, it re-proves 4 records whose cases the sweep had just run (3.5 min). You can see this live on run 38094461851: every other job finished by 23:43, while Db2 kept the refresh waiting.Change (
evidence-refresh.ymlonly):db2-part: a 5-leg matrix, each leg with its own Db2. Four legs runproof_sweep.py --det-only --db2-only --shard I/4 --no-ratchet, balanced bydet_sweep_durations.json; the fifth leg re-proves the stale Db2 records in parallel with them.db2(new, thin): downloads the parts, thenproof_sweep.py --aggregate <shards> --expect db2 --skip-ledger-check. That is the same ratchet as before, applied to the merge, so a shard that died fails it. It keeps the job's old outputs (sweep_ok,sha) and thedb2-resultsartifact, so therefreshjob barely changes.refresh: hands every Db2 shard'ssweep.jsontodet_coverage_ledger.py update(it already takes several).Checked locally:
actionlintis clean (private venv).lane, db2-part, db2, sweep, records, refresh.sweep.json, split as the shards would split it: all 4 shards → "as expected", exit 0; shard 3 missing → 10 cases "no result", exit 1.pr_gates.py --fast: 5/5 pass.committed … is currentchecks on generated pages, which are already out of date on main since CICS: DOCUMENT CREATE / RETRIEVE in both runtimes and both translators (Part of #4769) #4858 and advisory in CI.Expected: each shard about 2.5 min of setup + its estates + about 6.5 min of sweep; the records leg about 7 min; the merge about 2 min. The Db2 path should drop from about 34 min to about 13–15, roughly level with the non-Db2 sweep shards, so the refresh ends in about 20 min instead of about 37. The cost is 5 runner starts and Db2 image pulls instead of 1.
Not done:
gitgalaxy/, which changes on nearly every merge, so it would rarely hit.Proving it on CI: this workflow runs nightly or on dispatch only. After merge, dispatch
gh workflow run "Evidence Refresh" -R squid-protocol/gitgalaxy(db2 = true) and compare thedb2-partanddb2timings with 34 min.🤖 Generated with Claude Code
https://claude.ai/code/session_01MhEx8PwSow3Pyfg6J4KZ97