Benchmark history stalled 2026-08-12 → 08-31: a four-cause relay behind 100% green checks #1012

Symptom

benchmarks/ultrafuzzbench/history.json published no observation between 2026-08-12 and 2026-08-31 — 174 observations, newest run_timestamp 2026-08-12T01:03:09.686Z — while the Modal Eval Benchmarks workflow ran on nearly every push to main and reported success.

Root-caused 2026-08-31. The publication gate is not defective; it refused correctly throughout. The outage is a relay of four independent causes, which is why every previous single-cause fix (#560, #935) closed without restoring data flow.

The relay

The final phase, in detail

Smoke target very-liquid-vaults-foundry (rheo-xyz/very-liquid-vaults@e5038470) declares a transitive submodule lib/console3. That repository became temporarily private on 2026-08-27T03:01:48Z. The Modal sandbox clones cohort targets with no git credential, so git submodule update --init --recursive --depth 1 (packages/modal/src/public-worker.ts:1037) aborted in ~3 s with fatal: Authentication failed.

This is pre-model — preparePublicBenchmark runs before checkpointPublicModelWorkStart — hence model_work_started: false, exit_category: "unreachable", diagnostic_code: "dependency-unreachable", three identical pre-model retries, then recovery-budget-exhausted. 8 of that target's 9 submodules were unaffected; one unreachable dependency fails all three targets, because preparation is all-or-nothing.

Resolved out-of-band: the dependency is public again at the same commit 401bd220c343e885c336c406c4e959088e519acf, so the pinned gitlink resolves and no cohort re-pin or cohort_fingerprint change was required. Verified by reproducing the sandbox's exact anonymous recursive clone: SUBMODULE-UPDATE-RC=0.

Why it went unnoticed for 19 days

1. The soft-fail hatch. scripts/ci/describe-smoke-soft-fail.mjs:61-70 forgives five operational categories when mode === "smoke" && eventName === "push", so collect exits 0 and the check is green. Structural tell: cleanup_incomplete_run (gated on incomplete == 'true') ran in 29 of 29 green push runs since 08-13 — a 100% failure rate reported as 100% success.
2. The publication no-op. scripts/ci/classify-modal-benchmark-publication.mjs correctly reached ready=false, so every later step skipped and the job went green having emitted one stdout line and no annotation.
3. Nothing measured freshness. The repo contained zero cron: triggers, and eval history --check validated integrity and chart byte-equality but had no recency assertion — it exited 0 on 19.5-day-old data.

Also verified, to close off two plausible-but-wrong theories: #567 is innocent (its 08-13 date is coincidence; the benchmark-family rename is path-consistent), and the generation: 1 / 3 / 6 disagreement across artifacts is three correct counters on three different axes, not a bug.

Fixes

See the linked draft PR.

1. scripts/ci/verify-cohort-reachability.mjs (new) — anonymous pre-compute gate, wired into eval-benchmarks.yml before any image build or sandbox launch. Fails in ~2 s naming the exact unreachable URL, instead of burning a ~4 h paid run that reports success. Deliberately tokenless: an authenticated probe reads a private dependency fine and reports the cohort healthy. Exit 1 (named denial) blocks; exit 2 (inconclusive) warns and continues.
2. scripts/ci/assert-benchmark-history-freshness.mjs + .github/workflows/benchmark-history-freshness.yml (new) — the repo's first scheduled monitor, importing the canonical parser rather than re-deriving the rule.
3. eval history --max-age-days — opt-in recency assertion; default behaviour byte-identical, so the existing gate on main is unchanged.
4. Both silent publication no-ops now annotate (::warning:: + $GITHUB_STEP_SUMMARY) with no change to any ready/eligible verdict, now pinned by tests.
5. Worker diagnostics — the failure reason (git exited 1: fatal: Authentication failed for '…') was computed and discarded at the serialization boundary; it now reaches the collected artifact via the existing eval-failure-diagnostics channel. Also defuses two latent landmines where a single collector-illegal log line would have destroyed a pair's entire artifact set.

Acceptance criteria

• A cohort dependency that is not anonymously cloneable fails in seconds, before Modal spend, naming the URL.
• The publication honesty gate is unchanged and still fails closed.
• A stale history is detectable and alerts.
• A pre-model command failure records its reason in a collected artifact.
• One run where all three smoke pairs reach terminal_status=succeeded with a validated public-results.json.

Known remaining work

• Phase C is unmasked, not fixed. The WORKFLOW_SUBMISSION_FAILED defect has not been exercised since 08-27 because the lane died earlier in preparation. Expect it to resurface on the first run that reaches the eval stage.
• Merge-velocity cancellation is unaddressed. cancel-in-progress: ${{ github.event_name == 'push' }} keyed to the shared github.ref cancelled 43 of 62 push runs. Separately, 28 of those cancelled runs had already launched Modal sandboxes and then had cleanup_incomplete_run cancelled too — an unreported compute leak. Needs a trigger/concurrency decision (schedule + dispatch, or per-commit concurrency).
• resume-required is terminal. eval-benchmark-recovery.yml only terminates sandboxes; there is no resume-and-publish path.
• The results archive is unwritable by automation, so each observation's evidence is a 30-day artifact.