A generation-0 failure before the durable run is linked makes every later generation unrecoverable #378

Impact

Any failure in generation 0 that happens after eval plan creates the eval run directory but before a
durable run is linked converts a transient error into a permanently unrecoverable run. Every
subsequent recovery generation dies in ~92 seconds without doing any work, so the entire
maxNoProgressGenerations budget (default 3, packages/modal/src/defaults.ts:15) burns down and the run
is lost. Recovery cannot succeed no matter how many generations are allowed.

Observed on R54 (aave-v4-v0012-main-issue368fix-r54-invariant-only, image from 0088f80).

Evidence

Generation 0 failed at workflow submission (an unrelated npm publish race, #377). Generations 1 and 2 then
each died in ~92s. They wrote no events at all, which is why the node-error channel and the volume's
event log were both silent — the failure is above them. The cause is only in the sandbox's own stderr:

worker terminated: Error: resume requires exactly one linked durable run, found 0
  at locateModalResumeWorkspace (/opt/ultrafuzz/packages/modal/dist/resume.js:77)
  at async Object.run (/opt/ultrafuzz/packages/modal/dist/worker.js:96)

The linkage record on the volume confirms it — one record, failed, and never linked:

control/.ultrafuzz/evals/runs/<eval-run-id>/runs.jsonl
  1 record: status=failed  ultrafuzz_run_id=None

so productRunIds.size === 0 and resume.ts:120 throws.

Root cause: the routing predicate disagrees with the resume precondition

worker.ts:144 chooses resume-vs-fresh with:

async function hasExistingEvalWorkspace(): Promise<boolean> {
  const evalRoot = path.join(WORK_ROOT, "control", ".ultrafuzz", "evals", "runs");
  return (await readdirIfExists(evalRoot)).length > 0;      // worker.ts:213
}

That directory is created by eval run (packages/evals/src/runner.ts:127 writes eval.json), not by
eval plan — eval plan is a dry run and writes nothing. So the directory appears once model work has
begun, but the ultrafuzz_run_id linkage is only recorded once a durable run actually exists. Between
those two moments the predicate reports "resumable" for a workspace that provably has nothing to resume, and
locateModalResumeWorkspace — which requires exactly one linked durable run — immediately throws.

That window is not narrow: it spans the whole of workflow compilation and submission, which is where
generation 0 died here.

The part that makes this clearly a bug rather than a missing feature: prepareWorkspace() already handles
this exact state. Its else branch (worker.ts:336) exists precisely for "a pre-model workspace is
already on disk" — it asserts target, control, ground-truth/findings.yml, the suite and the source
proof are present, then re-runs references sync, validate and eval plan. That branch is
unreachable whenever eval plan got far enough to create the runs directory, which is exactly when it
is needed.

So the two halves disagree about what "existing workspace" means:

• hasExistingEvalWorkspace → "the eval runs directory is non-empty"
• locateModalResumeWorkspace → "exactly one linked durable run exists"

Suggested fix

Route on the same condition the resume path requires: a workspace is resumable only if a durable run was
actually linked. With no linked run, fall through to prepareWorkspace(), whose non-empty branch already
does the right thing.

Two properties any fix must preserve, both worth explicit tests:

1. Never silently discard real progress. If a linked durable run exists, behaviour must not change —
   this must not become a path that quietly restarts a run that had done model work.
2. Distinguish "nothing linked" from "more than one linked." productRunIds.size > 1 is genuine
   corruption and must keep failing loudly; only the 0 case is safe to treat as not-yet-started, because
   0 linked runs means by construction that no durable run was ever recorded.

Acceptance criteria

• A regression test that fails without the fix: a workspace whose runs.jsonl holds only unlinked
  records routes to the pre-model path and does not throw exactly one linked durable run.
• A test that a linked durable run still resumes exactly as before.
• A test that productRunIds.size > 1 still throws.
• Recovery is shown to survive a generation-0 failure that occurs after eval plan.

Note on diagnosis

Worth recording because it cost time: all three generations failed at near-identical durations (93s, 92s,
94s), which made a single shared cause look obvious, and the one cause already in hand was the npm E404 from
generation 0. That inference was wrong — generations 1 and 2 never reached npm. Only the sandbox stderr
distinguished them. Neither the event log nor the node-error channel can see a failure this early, which is
the same visibility gap as #361.