Evidence refresh: re-measure translation every night (the reports' survey froze on 2026-10-07)

What happened. The evidence reports on main still show the det translator as it was on 2026-10-07 (survey at 3bcbb8c6, 143 commits and ~30 translator merges ago), while they print "stale: 0". Programs made whole since then (BNK1CAC, CREACC, BNKMENU, lgwebst5, ZECSPLT, COPAUS2C …) sit at L0 even though their cases are equal, often at 100% paragraph and branch coverage. blockers_history.jsonl also stopped at #4636 (10/07).

Where we really are (local survey of main cd3af8477, 2026-10-10, same corpora pins):

	committed report (translator of 10/07)	fresh survey (main today)
CICS programs whole	78 / 125 (non-burned 5 / 21)	97 / 125 (non-burned 7 / 21)
all programs whole	—	116 / 179 (non-burned 12 / 52), 26 refused whole
burned L0 / L1	42 / 4	22 / 9
burned L2+ / L3+ / L4	71 / 34 / 18	86 / 46 / 24
levels stale	0 shown	76 of 86 (harness + oracle moved after the 10/10 09:57Z refresh; tonight's nightly re-sweeps them)

Per estate (fresh): CardDemo 44: L0 13, L1 4, L2 12, L3 3, L4 12 · CBSA 31: L0 3, L1 4, L2 12, L3 6, L4 6 · GenApp 31: L0 0, L2 13, L3 13, L4 5 · Mortgage 6: L0 2, L1 1, L2 3 · zECS 5: L0 4, L4 1. L5: 0 (#4628 not measured).

Why. Three pieces that each work but don't talk to each other:

evidence_report.py freezes the translation survey into report.json's measured block; the bot runs --refresh --live, which re-uses it. The comment at evidence-refresh.yml:772 says the job lacks corpora, but the refresh job already fetches the pinned corpora (as do sweep / records / db2).
Blocker Baseline (blocker-baseline.yml) surveys exactly the burned corpora on main every night (green, 4–22 min), and only uploads an artifact. Nothing reads it.
Staleness (--check --live, the release gate) fingerprints harness / oracle / case / corpus inputs, not the translator the survey ran. A frozen survey is never "stale", so the release gate can't see it.

Plan, in order (each step independently safe)

Re-measure translation in the nightly refresh. In the refresh job, at the run's pinned SHA: cics_census.py survey --baseline --no-census --no-fetch --sha $SHA --root $GG_SCRATCH (~5 min on the corpora it already has), then build the reports from it (evidence_report.py --all --baseline --sha $SHA) instead of --refresh --live. Fix the comment at :772.
A survey can be stale. Store the translator fingerprint (det/, standards/cics, cobolrt sources) with the survey; when it doesn't match the tree, the levels that rest on it are L* like coverage (evidence report: keep a program's last measured level marked stale instead of dropping L3/L4 to zero on any harness change #4730), counted apart, and --check --live fails. The summary prints "translation measured at " next to the coverage date.
One survey per night, one writer. Fold Blocker Baseline into Evidence Refresh (or have it hand its survey over), and publish blockers.txt/json + a nightly blockers_history.jsonl line on the generated branch (CI v2: required checks + merge queue, generated data off main (one writer), prove only what moved, one PR workflow #4825 step 2), not by hand on main. Backfill 10/07 → today from the local caches (census-cache/ holds 08b43bb, 63c1a6a, 5abdf0c, 82918f0, 5242878, cd3af84).
Shorter daytime staleness. Every merge touching the harness or oracle stales all levels until 03:17Z. Options: the post-merge idle dispatcher (ci: PRs first -- post-merge checks run on the latest main when runners are idle (#4825) #4843) dispatches a refresh when harness / oracle / translator moved, or narrow the harness fingerprint to what each case reads (det sweep: copy forward verdicts by content key, prove only what moved, work-sized shards up to 20 #4814-style keys). Measure before choosing.
Release gate covers all three clocks: translation survey, coverage ledger, records — one SHA each, all current.

Not in CI (by rule): census (non-burned) counts stay a local run; only counts, never committed artefacts.

Acceptance: after one nightly run, the committed / generated report quotes a survey of that night's main; BNK1CAC-class programs show their real level; the history file has a line per night; a translator merge without a refresh makes --check --live fail.

🤖 Generated with Claude Code

ci: evidence refresh re-measures the translation every run (Part of #…

Step 6 (added 2026-10-10): a fast Db2 job. It is the refresh's critical path: 34 min vs ≤16 for every other job (nightly run 38043242484). It runs 38 cases / 673 scenarios at --jobs 4 on one 4-vCPU runner that also hosts Db2, about 9 s per scenario per slot (mostly CLP back-end starts and JVM starts), and then re-proves 4 records whose cases the sweep had just run (3.5 min); the 4 estates are generated one after another first (3 min).

Shard the Db2 sweep over 4 runners (proof_sweep.py --det-only --db2-only --shard I/4, merged with --aggregate --expect db2), each with its own Db2. Expected ~34 → ~10–12 min.
The stale Db2 records are re-proved on their own runner, in parallel with the shards.
Cache the generated estates between runs.
(Later, measure first) one long-lived Db2 session per case instead of a CLP process per reset/dump; content-keyed skips (det sweep: copy forward verdicts by content key, prove only what moved, work-sized shards up to 20 #4814) for Db2 cases.

ci: evidence refresh's Db2 job in 4 shards + a records leg, merged by…

tests: evidence-report tests no longer depend on the committed report…

Progress 2026-10-11.

Step 1 done: ci: evidence refresh re-measures the translation every run (Part of #4863) #4864. The refresh's refresh job surveys its own commit (6.3 min on CI) and builds the reports from it. Proved on run 38094461851.
Step 6 done: ci: evidence refresh's Db2 job in 4 shards + a records leg (Part of #4863) #4866. The Db2 job runs as 4 shards plus a records leg, merged by a ratchet job. On run 38101829778: shards 10.5–12.4 min, records 8.7 min, merge 29 s. Db2 path 35.5 → 17 min (including about 3 min waiting for a runner); whole refresh 48.5 → 28 min; "shards found: 4, sweep: as expected".
Follow-up: tests: evidence-report tests no longer depend on the committed reports' counts (Part of #4863) #4869. Three evidence-report tests asserted the committed reports' old counts, so the first fresh bot PR (evidence: refresh stale records and the evidence report #4867) failed them. They now assert only what they test, and pass on both old and new data.
Bot PR evidence: refresh stale records and the evidence report #4867 carries the re-measured reports. Burned estates: L0 22, L1 9, L2+ 86, L3+ 46, L4 24, plus the async credit-card estate's first report (9/10 at L4).

Left: step 2 (a survey can be stale, and the release gate fails on it), step 3 (one survey a night + history on generated), step 4 (shorter daytime staleness), step 5 (release gate over all three measurements), and step 6.4 (per-scenario Db2 overhead, measure first).

CI speed-ups merged 2026-10-11 (from a measured pipeline audit; reports in the orchestrator handoffs, pipeline-audit-2026-10-11):

CI: shard windows 3.10 and macOS, right-size full-suite-gate caps (Part of #4903) #4906: Full Suite Gate wall 38–41 → 24 min. Windows 3.10 runs as 3 even parts (13/15/15 min), macOS 3.11/3.12 as one leg each, caps right-sized.
Pin GALAXYSCOPE_MAX_WORKERS=1 in parallel scan tools (Fixes #4896) #4899: unicode gauntlet nested worker pools fixed: 182 s → 39 s, peak load 82.5 → 10.5, byte-identical output. Same pin in cultural_gauntlet and import_graph_accuracy.
galaxyscope: size pool from GG_JOB_CORES; pin parallel verifiers (Fixes #4902) #4907: galaxyscope sizes its pool from GG_JOB_CORES, and parallel tools give each child an equal share. A 2-leg bless takes 73 s at load 9.3, vs 76 s at load 15 before.
tooling: box-capacity skill + box_status.py (admission control, collision checks for agent fleets) #4882: heavy-run.sh fair core share plus box_status.py (box-capacity skill).
pipeline: timeouts on docker/javac/java/mvn calls, a Db2 pool deadline, workflow timeout-minutes; GnuCOBOL image cache in cics-crucible (#4904) #4908: timeouts on docker/javac/java/mvn calls, a 20-min Db2 pool deadline, workflow timeout-minutes, and the GnuCOBOL image cached in cics-crucible.
equivalence: runs after the first Maven run take the direct JVM path (Fixes #4910) #4914 (merging): equivalence fault runs skip Maven, about 1.9 s per run (readacct 78 → 55 s), with verdicts identical.
Fix nightly: hermetic stale-level evidence tests, ground-truth installs [full] #4912: nightly evidence-report tests made independent of the committed data (bot PRs were blocked by them), and Mainframe Ground Truth installs [full] (Mainframe Ground Truth nightly fails: feature_inventory --all-burned --check needs gitgalaxy[full] #4911).
In progress: cbsa-bnk1cac (Db2 case) stopped proving in the nightly — and PR CI never proves Db2 cases (det sweep --skip-db2) #4901, Db2 cases proven in PR CI when a PR moves them.
Measured, not worth doing: pipeline: compile shared C stubs and LD/UL loaders once per sweep, not per case (measure first) #4905 (stub/loader compile caches save <0.3 s per case).

For this issue: steps 1 (#4864) and 6 (#4866) are done. Open: steps 2 (survey staleness: a translator merge without a refresh fails --check --live), 3 (one survey per night plus a history line), 4 and 5. Steps 2 and 3 are queued next for the agent churn.

evidence report: a translation survey can be stale; the release gate …

ci: publish-blockers writes generated only from main; other refs get …

ci: fix YAML of the preview step name (#4863)

ci: one survey a night; blockers + history on generated (#4863 step 3) (

Closed: all planned steps have landed:

Step 1 (re-measure translation every run): PR ci: evidence refresh re-measures the translation every run (Part of #4863) #4864
Step 6 (Db2 sharding + ratchet): PR ci: evidence refresh's Db2 job in 4 shards + a records leg (Part of #4863) #4866
Decouple report tests from committed counts: PR tests: evidence-report tests no longer depend on the committed reports' counts (Part of #4863) #4869
Step 2 (translation survey staleness gate): PR evidence report: a translation survey can be stale; the release gate fails on it (Part of #4863) #4919
Step 3 (one survey per night; blockers and history published to generated branch): PR ci: one survey a night; blockers + history on generated (#4863 step 3) #4924
Evidence records and report refreshed: PR evidence: refresh stale records and the evidence report #4923

Navigation Menu

Evidence refresh: re-measure translation every night (the reports' survey froze on 2026-10-07) #4863

Plan, in order (each step independently safe)

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Evidence refresh: re-measure translation every night (the reports' survey froze on 2026-10-07) #4863

Description

Plan, in order (each step independently safe)

Activity

squid-protocol commented on Oct 10, 2026

squid-protocol commented on Oct 11, 2026

squid-protocol commented on Oct 11, 2026

squid-protocol commented on Oct 12, 2026

squid-protocol commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions