Repository navigation
Modal benchmark secret gates fail on benign logical run ids, breaking every smoke/full run #883
Description
What happened
The first smoke dispatch after re-enabling Modal Eval Benchmarks (run 32866658497) failed twice from one root cause:
- In the sandbox: the worker died with
dependency-unreachable/resume-required, zero node counts,model_work_started: true. - On the runner: the
collectjob's diagnostic pass crashed withError: Modal recovery lifecycle contains secret-like content(recovery-lifecycle.js:243viacollectModalBenchmark), so no diagnostics could even be retrieved.
Root cause
redactSecretsInText in default "all" mode runs the speculative high-entropy pass, which flags the benchmark's own identifiers:
"ci-32866658497-1-smoke-ultrafuzz-bench-deepseek" → <redacted>
"ultrafuzz-ci-32866658497-1-smoke-ultrafuzz-bench-d-18cb2eabbe80" → <redacted>
(kebab-case + digits = 3 character classes, ≥12 unique chars, Shannon entropy ≥ 4.3 — reproduced against main's build.)
Three fail-on-hit gates in packages/modal scan JSON that embeds lineage.logical_run_id / volume names with that default mode:
assertPublicEvalDiagnosticsContainsNoSecrets(public-eval-diagnostics.ts:206) — called bycreatePublicEvalDiagnosticsFromRuninside the sandbox worker; the throw failsbuildDiagnostics, so the worker finalizes asunreachable→dependency-unreachablewith empty counts. This is the primary run failure.assertModalRecoveryLifecycleContainsNoSecrets(recovery-lifecycle.ts:348) — crashescollecton the runner (the observedcollectstack trace).assertPublicBenchmarkFileContainsNoSecrets(public-bundle.ts:228) — would fail bundle collection/publication for the same reason once the earlier gates pass.
Why it was never caught
- The gates landed in feat: add strict producer-visible JSON contracts with release-lane fix #558 (Aug 13), after the last completed benchmark run (Aug 12); the workflow was disabled until today, so they never executed in CI with a real
ci-<run_id>-…logical run id. Test fixtures use low-entropy ids. - Replace hand-rolled vendor secret patterns with secretlint #865 introduced exactly the needed distinction —
SecretScanMode"positive-only"— and applied it to the artifacts secret gate with the rationale that fail-on-hit gates must not use speculative heuristics (Shannon entropy scores a versioned contract method above a GitHub token, Artifact secret gate fails closed on ordinary prose: entropy, 40-hex and keyword heuristics cannot separate credentials from identifiers #819). The three modal gates were not swept.
Fix
Align the three modal fail-on-hit gates with packages/artifacts/src/secret-gate.ts: keep the exact injected-secret-value check byte-for-byte, and run the generic scan as containsSensitiveSecrets(text, [], "positive-only"). Verified that every existing test fixture (ghp_…, sk-…) is still caught in positive-only mode and the benign run ids pass. Rewriting sanitizers (sanitizeWorkerDiagnosticMessage, state/ledger redaction) stay in "all" mode — they redact rather than fail, which is the documented correct pairing.
Every smoke/full/threat-model lane is broken on main until this lands; the in-flight push-triggered smoke runs will fail the same way.
Activity
fix(runtime): persist workflow input with positive-only secret scanni…