Rewritten published artifact strands runs: verified dependency digest mismatch fails fan-in forever with zero failed nodes #275

Activity

1. aviggiano commented on Aug 6, 2026

   

   aviggiano

   on Aug 6, 2026

   CollaboratorAuthor

   More actions

   Root cause confirmed empirically — my hypothesis above was wrong

   I read R43's workflow log and durable volume directly. The hydration hypothesis is refuted: the log shows restoredTaskCount=36 on all three resume attempts, so terminal-task restoration works. There is no DEPENDENCY_DEADLOCK either. The actual failure is an artifact-contract poison-pill.

   The real failure

   From .smithers/logs/ultrafuzz-…-trial-1.log on R43's volume, identically on every resume:

   ✗ prepare:property-specification-fanin (attempt 1): artifact-contract failure:
       artifact dependency has not passed verification property-specification-crytic
       for property-specification-fanin
     caused by: verified dependency artifact changed properties/crytic.json
   ✗ Run failed: Task failed: prepare:property-specification-fanin
   status: failed   code: SESSION_ERROR
   

   That is assertVerifiedDependency → the artifactSha !== entry.sha256 check in packages/runtime/src/templates/smithers/workflows/workflow.tsx:1253-1256.

   Verified byte-level evidence

   Verification marker <runRoot>/.ultrafuzz-verification/property-specification-crytic.json, compared against the artifacts on the volume:

   artifact	marker sha256	on-disk sha256	match
   properties/crytic.md	97b050b0…0538	97b050b0…0538	yes
   findings.json	37517e5f…b570	37517e5f…b570	yes
   properties/crytic.json	8cfef182…1328	19d3976b…3e9d (44704 bytes)	NO

   The marker is otherwise entirely well-formed: correct schema_version, correct attempt_id, artifact set exactly matching the declared outputs. Its publications[] array records the same old sha 8cfef182… for properties/crytic.json.

   A full walk of the run root finds exactly one copy of that artifact (artifacts/property-specification-crytic/properties/crytic.json). So this is not a canonical-vs-attempt copy divergence — a published artifact that is supposed to be immutable was rewritten in place after its verification marker was written, and neither the marker's artifacts[] nor its publications[] was updated.

   Why this is unrecoverable

   prepare:property-specification-fanin fails deterministically on every attempt. Because the failure is in the prepare wrapper rather than the node body, Ultrafuzz's durable state records property-specification-fanin as pending with last_error: null and zero failed nodes, while the workflow run is failed. Recovery then cannot make progress:

   • modalDurableRunNeedsResume sees remaining: 11 > 0 and resumes;
   • the resumed run re-fails on the same digest mismatch within seconds;
   • repairModalEvalRunRecord refuses to finalize (packages/modal/src/resume.ts:130-133) and the sandbox exits terminal-run-non-resumable.

   R43 burned three recovery generations this way (~108 s each) before the budget exhausted, having already spent ~83.3 M tokens / ~$482. No retry-failed path helps, because there is no failed node to reset.

   This is why the #273 approach cannot work: neither rewinding a frame nor transferring to a fresh lineage changes the on-disk sha of properties/crytic.json.

   Expected fix

   Two distinct defects, and both need addressing:

   1. Prevent the divergence. A published artifact must not be rewritable after its verification marker is written, or the marker must be rewritten atomically with it. Whatever rewrote properties/crytic.json bypassed the publication immutability guards (assertPublishedFileReplacementAllowed, assertPublishedBytes). That write path is the primary bug. Note the digest comparison itself is correct and was tightened by fix(runtime): block artifact handoffs after failed verification #252 — fix(runtime): block artifact handoffs after failed verification #252 is not at fault; it surfaced a pre-existing corruption that previously passed silently.
   2. Make the state recoverable. A marker/artifact divergence should invalidate that dependency's verification and re-run its verifier, not fail every dependent task forever with no failed node recorded. Today a single diverged byte permanently strands a run with 21 completed nodes.

   Regression tests should cover: a dependency whose published artifact sha no longer matches its marker must either be re-verified or surface as a failed node that --retry-failed can reset — never as an unrecoverable terminal run with zero failed nodes.

   Also observed in the same log, likely unrelated but worth a separate look:
   [smithers] worktree sync: git rebase origin/ultrafuzz-pinned failed (exit 128): fatal: invalid upstream 'origin/ultrafuzz-pinned'

   Evidence preserved at r43-workflow-log.txt alongside the R43 inspections. R43 was never stopped or restarted.
2. aviggiano

   changed the title [-]Resume does not restore succeeded nodes whose output rows are unmatched, deadlocking fan-in[/-] [+]Rewritten published artifact strands runs: verified dependency digest mismatch fails fan-in forever with zero failed nodes[/+] on Aug 6, 2026
3. aviggiano commented on Aug 6, 2026

   

   aviggiano

   on Aug 6, 2026

   CollaboratorAuthor

   More actions

   Narrowing: both publication guards are intact, so the rewrite bypassed them

   Further evidence, and it rules out the obvious suspects.

   The stranded artifact is legitimate, not corrupt. I pulled properties/crytic.json off R43's volume and validated it against its declared contract:

   • properties/crytic.json — 44704 bytes, valid JSON, ultrafuzz/property-lens@1 validation passes, 102 properties.
   • For scale, properties/certora.json from the same run is 52790 bytes with 113 properties.

   So the file the run is refusing to accept is a complete, contract-valid lens output. The marker's sha is stale; the artifact is fine.

   The marker could not have been stale when it was written. Tracing the verify path in packages/runtime/src/templates/smithers/workflows/workflow.tsx:

   • publishVerifiedArtifacts (:3610-3614) writes every publication through publishFileDurableExclusive.
   • publishFileDurableExclusive (packages/artifacts/src/safe-paths.ts:188-204) calls assertPublishedBytes(root, destination, bytes) when the destination already exists, so a pre-existing file with different bytes throws rather than being silently kept.
   • writeArtifactVerificationMarker (:3665-3704) computes its publications[] shas from the very same buffers, and runs after publication.

   Therefore at marker-write time, disk bytes and marker bytes were necessarily identical. Combined with the divergence now on disk, the artifact was rewritten after its verify task completed.

   Reconciliation is not the writer either. reconcileTargets (packages/runtime/src/artifact-reconciliation.ts:99-120) skips any target whose destination already exists, and copyRegularFileExclusive re-checks and returns false if the destination exists. It cannot overwrite a published artifact.

   So the writer is still unidentified, and it bypasses both the exclusive-publish guard and the exclusive-copy guard. I am deliberately not shipping a "prevent the rewrite" patch while that is true — a fix aimed at the wrong writer would be worse than no fix, which is the lesson of #273.

   Revised fix plan

   1. Instrument to identify the writer. Record an integrity re-check of every published artifact against its marker at each node handoff boundary, with the node id and timestamp, so the next occurrence names the writer and the window. Cheap, and it converts this from forensics into a single log line.
   2. Make the state recoverable independently of the cause. Today the failure lands on the prepare: wrapper of the dependent, so Ultrafuzz records zero failed nodes and --retry-failed has nothing to reset. A marker/artifact divergence should instead mark the dependency node failed, so the existing timetravel-reset machinery re-runs it and re-verifies it from scratch. That preserves verification strength — the artifact is re-verified by the real verifier, not waved through.
   3. Do not auto-repair by rewriting the marker from current bytes. The sha is the anti-tampering seal, and for a security tool whose artifacts are authored by the model under test, silently re-sealing changed bytes is the wrong trade even though in this instance the bytes are benign.

   Additional detail for whoever picks this up: the durable node timestamps for all eight property-specification-* nodes were rewritten into a 13-second window (08:59:39Z-08:59:52Z) with ~1.4 s durations, i.e. a burst re-finalization at the moment the last lens finished — a plausible window for the rewrite and a good place to point the instrumentation first.
4. aviggiano commented on Aug 6, 2026

   

   aviggiano

   on Aug 6, 2026

   CollaboratorAuthor

   More actions

   Task timeline from R43's workflow DB — verify runs ~43 min after the node, and the verified bytes no longer exist

   I pulled R43's smithers.db off the volume (18.5 MB, preserved) and dumped the attempt timeline. This is the most useful artifact yet.

   08:00:43-08:07:31  finished  node:property-specification-certora
   08:07:31-08:16:34  finished  node:property-specification-crytic       <- crytic agent writes crytic.json
   08:16:34-08:23:33  finished  node:property-specification-runtime-verification
   08:23:34-08:30:13  finished  node:property-specification-a16z
   08:30:14-08:37:36  finished  node:property-specification-recon
   08:37:37-08:45:15  finished  node:property-specification-aviggiano
   08:45:15-08:52:52  finished  node:property-specification-0kn0t
   08:52:52-08:59:38  finished  node:property-specification-josselin-feist
   08:59:39-08:59:40  finished  verify:property-specification-certora    <- all 8 verifies batch here
   08:59:41-08:59:42  finished  verify:property-specification-crytic     <- 43 min after crytic's node
   ...
   08:59:51-08:59:52  finished  verify:property-specification-josselin-feist
   09:41:14-09:41:19  FAILED    prepare:property-specification-fanin
   

   The schedule is prepare-all → run-all → verify-all. verify:property-specification-crytic ran 43 minutes after node:property-specification-crytic finished, and in that window the six later lens agents ran in the same workspace. That is a large window for the shared properties/ directory to drift.

   Node rows confirm the asymmetry that makes this unrecoverable:

   node	state
   node:property-specification-crytic	finished
   verify:property-specification-crytic	finished
   prepare:property-specification-fanin	failed
   node:property-specification-fanin	pending
   verify:property-specification-fanin	pending

   Only the prepare: wrapper is failed. No node: is failed, which is exactly why Ultrafuzz reports zero failed nodes and --retry-failed has nothing useful to reset.

   The verified bytes are gone

   I walked the entire volume subtree for the run. There is exactly one crytic.json anywhere, and it holds 19d3976b… — not the marker's 8cfef182…. No workspace mirror copy survives. So the verified content exists nowhere on the volume and R43 cannot be repaired by restoring it; the crytic node would have to re-run.

   Also relevant: the run has 41 timeline frames, which independently confirms that #273's fresh-lineage fallback was dead code for this run (previousSmithersTimelineFrame always returns a value).

   Writers eliminated so far

   • publishFileDurableExclusive — calls assertPublishedBytes on an existing destination, so it throws rather than silently keeping differing bytes (packages/artifacts/src/safe-paths.ts:201-204).
   • reconcileTargets / copyRegularFileExclusive — both skip when the destination exists (packages/runtime/src/artifact-reconciliation.ts:99-120).
   • smithers worktree sync — only fetch + rebase origin/<base> inside a separate worktree, and R43's rebase failed non-destructively (invalid upstream 'origin/ultrafuzz-pinned').
   • workspace-handoff patches — .ultrafuzz and artifacts are in SENSITIVE_SEGMENTS and excluded (packages/runtime/src/workspace-handoff.ts:11-21).

   Since disk necessarily equalled the marker at 08:59:42 (publication asserts it) and differs now, the rewrite happened after the verify burst, i.e. across the sandbox exit at ~09:01 and the three restore/restart cycles before prepare:fanin ran at 09:41. The restore/staging path is the remaining candidate and is where instrumentation should go first.

   Recommended fix, now better founded

   1. Shrink the window: run verify:<node> immediately after node:<node> rather than batching all verifies at the end, so a verified artifact is sealed while its workspace is still quiescent.
   2. Make it recoverable: on a marker/artifact divergence, reset node:<dependency> and verify:<dependency> so the dependency re-runs and is genuinely re-verified — one node re-run (~9 min for crytic) instead of stranding 21 completed nodes. Note this also requires clearing the now-inconsistent marker and artifact, since publishFileDurableExclusive would otherwise throw on re-publish.
   3. Keep refusing to re-seal the marker from current bytes.

   Evidence preserved: r43-smithers-db-snapshot-20260806T1105Z.db, r43-workflow-log.txt, r43-crytic-json-ondisk-19d3976b.json.
5. aviggiano

   added 2 commits that reference this issue on Aug 6, 2026

   fix(runtime): gate dependency recovery on a real on-disk divergence

   ...

   d623dca

   fix(runtime): repair a diverged dependency by re-verifying it

   ...

   e3d95d2
6. aviggiano commented on Aug 6, 2026

   

   aviggiano

   on Aug 6, 2026

   CollaboratorAuthor

   More actions

   ROOT CAUSE IDENTIFIED: the writer is our own lens sanitizer, and this is a regression from #269

   The writer I could not find earlier is sanitizeLensReferenceExpectationAuthority in packages/runtime/src/artifact-gates.ts:1570.

   // packages/runtime/src/artifact-gates.ts:1607-1611
   if (removed === 0) return [];
   
   writeJsonDurable(lensPath, {
     ...lens.value,
     properties
   });

   lensPath is safeResolveInside(artifactDir, lensOutput.path, "property lens output") — the canonical published artifact, i.e. exactly artifacts/property-specification-crytic/properties/crytic.json.

   Why every guard missed it

   It writes with writeJsonDurable, not publishFileDurableExclusive. So assertPublishedBytes — the guard that makes published artifacts immutable — is never consulted. That is why the earlier elimination of the publish path, artifact reconciliation, worktree sync and workspace handoff all came back clean: none of them was the writer.

   Call path, and why it lands after verification

   sanitizeLensReferenceExpectationAuthority ← verifyRequiredArtifactsForAttempt (artifact-gates.ts:184) ← workflow-sync.ts:2058, i.e. the runtime sync path, which runs after the workflow's verify:<node> task has already published the artifact and written its marker. The sanitizer then edits the bytes and leaves the marker untouched, so marker.sha256 and the file disagree from that moment on.

   It only writes when removed > 0, which is exactly why a single lens diverged while crytic.md and findings.json stayed consistent.

   This is a regression from #269

   $ git log --oneline -S sanitizeLensReferenceExpectationAuthority -- packages/runtime/src/artifact-gates.ts
   2b00c48 fix(runtime): sanitize unsupported lens reference expectations (#269)
   

   2b00c48 is an ancestor of 4af20bf, the main commit R43 was built from. That explains why R43 hit this and earlier runs did not.

   Correct fix is prevention, not recovery

   The sanitizer is a trusted runtime gate deliberately removing unauthorized reference-expectation IDs — unlike a model rewriting its own output, so keeping the marker consistent with its edit is legitimate rather than a tamper bypass. Options, in preference order:

   1. Sanitize before the verification boundary, so the verifier hashes the final bytes and no marker update is needed.
   2. Keep the marker consistent: after writing, refresh the sha256 for that path in both artifacts[] and publications[] of <runRoot>/.ultrafuzz-verification/<attemptId>.json.
   3. Do not mutate the published artifact at all — report the diagnostic and let the producing node fix it.

   A regression test should assert that after the sanitizer rewrites a lens, the dependent's assertVerifiedDependency still passes — i.e. marker and bytes agree.

   Consequence for PR #278

   #278 is recovery-only and does not address this. Adversarial review also found it converts the stall into a terminal run loss for the five invariant-suite nodes: rememberInvariantSuitePublications (workflow.tsx:3096-3098) reads invariantSuitePublicationSnapshots, a module-level in-process Map populated only while node:<D> executes (workflow.tsx:2400 via artifactAwareAgent.generate). Resetting only verify:<D> leaves the node memoized, so on a resumed run the Map is empty and the verifier throws invariant suite publication snapshot is unavailable — after clearArtifactVerificationMarker has already deleted the marker, with retries={0}. The recovery's own verify:<D> finished guard then blocks it from firing again. Same class at verifyInvariantLedgerSourceEvidence (workflow.tsx:3421, :3468-3481), which reads the live worktree.

   So #278 is being parked as a draft, and the fix should target the writer above. Separately, the verifier arguably ought to be a pure function of on-disk state — rebuilding the invariant-suite snapshot from invariant-suite-manifest.json instead of trusting in-process memory — which would make standalone re-verification viable in general.
7. aviggiano

   added 3 commits that reference this issue on Aug 6, 2026

   fix(runtime): report a lens marker digest that drifted from disk

   ...

   0c5e19a

   fix(runtime): keep the marker consistent when sanitizing a lens

   ...

   090931e

   fix(runtime): report a lens marker digest that drifted from disk

   ...

   b8f4947
8. aviggiano

   closed this as completedin #279on Aug 6, 2026
9. aviggiano

   added a commit that references this issue on Aug 7, 2026

   fix(runtime): re-seal the verification marker after normalizing a pro…

   ...

   94d3291