[aviggiano]

Closes #332.

The corroboration hook added by #320 was gated on runError === undefined, but any failed or incomplete row makes ultrafuzz eval run exit 1 (eval/run.ts:63 → command-shared.ts:117-119), which runCommand turns into a throw. So the hook could never run for the case it was written for.

Proven on the production path, by driving a real 2-row matrix with rows failing pre-model:

	origin/main	this branch
HOOK CALLED WHEN eval run EXITS NONZERO	false	true
model_work_started after settle	true	false

runCommand and throwIfAborted already knew the distinction that matters but discarded it: a killed command and a self-chosen nonzero exit both surfaced as a bare OperationalDispositionError("unreachable"). The timeout/abort branch now raises PublicWorkerCommandInterruptedError, and the gate became "not an interruption" — a command that chose its exit code has finished writing; one that was killed may have launched work it never journaled and keeps its raised flag.

The flag is still only ever lowered, and only on a journal that positively accounts for every record.

Also: resolveTerminalReportPath between a successful launch and appendJsonLine is now total, and the previously silent catch writes one redacted line through sanitizeWorkerDiagnosticMessage.

Residual, flagged not fixed: with the gate relaxed, a genuinely partial journal becomes reachable — if runEvalSuite throws mid-matrix, rows still in flight never append. The principled close is reader-side (require the journal to account for every matrix.json row before answering none), but that flips a deliberately documented assertion and is a call for the maintainer, not the implementer.

Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com

Greptile Summary

The PR allows the public Modal worker to corroborate model-work state after an eval command completes with a nonzero exit while preserving the raised checkpoint when the command is interrupted.

• Distinguishes normal nonzero exits from timeout, abort, and signal termination.
• Ensures launched eval rows are journaled even when terminal-report-path enrichment fails.
• Logs sanitized corroboration-read failures and adds coverage for checkpoint and recovery decisions.

Confidence Score: 5/5

The PR appears safe to merge, with no unacknowledged actionable defects established in the changed behavior.

Completed nonzero eval exits now reach journal corroboration, while timeout, abort, and signal termination preserve the conservative model-work checkpoint; downstream report consumers remain able to resolve report paths from each run root.

Important Files Changed

Filename	Overview
packages/evals/src/runner.ts	Makes terminal-report-path enrichment best effort so a successfully launched row still reaches runs.jsonl.
packages/modal/src/public-worker.ts	Separates interrupted commands from completed nonzero exits, updates journal corroboration, and adds sanitized failure logging.
packages/modal/test/public-worker.test.ts	Adds coverage for nonzero exits, interruption handling, model-work recovery classification, and diagnostic redaction.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Start eval command] --> B{How did child terminate?}
  B -->|Timeout, abort, or signal| C[PublicWorkerCommandInterruptedError]
  C --> D[Keep model_work_started raised]
  B -->|Exit code 0 or nonzero| E[Read final eval journal]
  E --> F{Journal evidence}
  F -->|Launched or unknown| D
  F -->|No model work| G[Lower model_work_started]
  D --> H[Persist diagnostics and terminal status]
  G --> H

Loading

_{Reviews (1): Last reviewed commit: "fix(modal): keep an externally killed ev..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Evals
• Knowledge Base — Modal remote benchmark execution