Skip to content

Eval journal appends are not durable, so a killed sandbox can lose the durable-run link #382

Closed

Description

@aviggiano

Split out of #378 review. Not a blocker for that fix, but it is the reason the state #378 handles can occur
more often than it should.

The asymmetry

state.json is written durably — writeRunState -> writeJsonDurable -> writeFileDurable, which fsyncs,
renames atomically and fsyncs the directory (packages/artifacts/src/safe-paths.ts).

The eval journal is not. appendJsonLine (packages/evals/src/utils.ts:300) is a plain
fs.appendFileSync: no fsync, no directory sync.

So on an abruptly killed sandbox the durable run can be on disk while the row linking it is not, which is
exactly the "durable run present, link absent" state that made R54 unrecoverable.

Why this is worth fixing separately

The repo already has the durable primitive and already uses it for this file from the other side:
appendLineDurable (fsync, O_NOFOLLOW, directory fsync) is what resume.ts uses to write the repaired
link back. appendJsonLine simply predates it.

#378's fix makes the state recoverable. It does not make it rarer. Both are worth having: recovery
is what keeps a run alive, durability is what keeps the journal honest about what happened.

Scope note, so the fix is not mis-aimed

Do not assume every unlinked-run case is a lost append. R54's own journal row was
status=failed, ultrafuzz_run_id=None, which is precisely what launchEvalRow writes when the launch
fails (packages/evals/src/runner.ts:272-277) — the append succeeded and correctly recorded a failed
launch. So R54 demonstrates the launch-failed window, not the lost-append window.

The lost-append window is a distinct, additional way to reach the same on-disk shape, and it is the one this
issue is about.

Acceptance criteria

  • appendJsonLine (or at least the runs.jsonl call sites) durably appends: fsync the file and the
    containing directory.
  • A test that a torn/partial final line cannot silently drop a linked row — note resume.ts
    deliberately throws on a malformed line today, and that behaviour should be preserved, not softened.
  • Measure the cost before adopting it everywhere; this is on the per-row path and fsync is not free.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions