Skip to content

Modal benchmark secret gates fail on benign logical run ids, breaking every smoke/full run #883

Closed

Description

@aviggiano

What happened

The first smoke dispatch after re-enabling Modal Eval Benchmarks (run 32866658497) failed twice from one root cause:

  1. In the sandbox: the worker died with dependency-unreachable / resume-required, zero node counts, model_work_started: true.
  2. On the runner: the collect job's diagnostic pass crashed with Error: Modal recovery lifecycle contains secret-like content (recovery-lifecycle.js:243 via collectModalBenchmark), so no diagnostics could even be retrieved.

Root cause

redactSecretsInText in default "all" mode runs the speculative high-entropy pass, which flags the benchmark's own identifiers:

"ci-32866658497-1-smoke-ultrafuzz-bench-deepseek"                → <redacted>
"ultrafuzz-ci-32866658497-1-smoke-ultrafuzz-bench-d-18cb2eabbe80" → <redacted>

(kebab-case + digits = 3 character classes, ≥12 unique chars, Shannon entropy ≥ 4.3 — reproduced against main's build.)

Three fail-on-hit gates in packages/modal scan JSON that embeds lineage.logical_run_id / volume names with that default mode:

  • assertPublicEvalDiagnosticsContainsNoSecrets (public-eval-diagnostics.ts:206) — called by createPublicEvalDiagnosticsFromRun inside the sandbox worker; the throw fails buildDiagnostics, so the worker finalizes as unreachable → dependency-unreachable with empty counts. This is the primary run failure.
  • assertModalRecoveryLifecycleContainsNoSecrets (recovery-lifecycle.ts:348) — crashes collect on the runner (the observed collect stack trace).
  • assertPublicBenchmarkFileContainsNoSecrets (public-bundle.ts:228) — would fail bundle collection/publication for the same reason once the earlier gates pass.

Why it was never caught

Fix

Align the three modal fail-on-hit gates with packages/artifacts/src/secret-gate.ts: keep the exact injected-secret-value check byte-for-byte, and run the generic scan as containsSensitiveSecrets(text, [], "positive-only"). Verified that every existing test fixture (ghp_…, sk-…) is still caught in positive-only mode and the benign run ids pass. Rewriting sanitizers (sanitizeWorkerDiagnosticMessage, state/ledger redaction) stay in "all" mode — they redact rather than fail, which is the documented correct pairing.

Every smoke/full/threat-model lane is broken on main until this lands; the in-flight push-triggered smoke runs will fail the same way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions