[aviggiano]

Summary

• fixes the recovery half of Rewritten published artifact strands runs: verified dependency digest mismatch fails fan-in forever with zero failed nodes #275
• resets a diverged dependency's verify: and node: tasks so it re-runs and is genuinely re-verified
• quarantines the inconsistent marker and every artifact it declares so the exclusive publish has a clean path on the re-run

The failure this recovers from

A published artifact that diverges from its verification marker fails the dependent's prepare: wrapper on every attempt. Because the failure lands on the wrapper rather than the node body, the durable state records the dependent as pending with last_error: null and zero failed nodes — so --retry-failed has nothing to reset and the run is stranded permanently.

R43 (aave-v4-v0012-main-issue270-r43-invariant-only) lost 21 completed nodes and ~83.3 M tokens / ~$482 this way. Byte-level evidence from its volume:

artifact	marker sha256	on-disk sha256	match
properties/crytic.md	97b050b0…0538	97b050b0…0538	yes
findings.json	37517e5f…b570	37517e5f…b570	yes
properties/crytic.json	8cfef182…1328	19d3976b…3e9d	NO

prepare:property-specification-fanin then failed identically on every recovery generation (artifact-contract failure: artifact dependency has not passed verification property-specification-crytic … verified dependency artifact changed properties/crytic.json) until the no-progress budget was exhausted.

Why reset the dependency rather than repair the marker

The sha is the anti-tampering seal, and in this product the artifacts are authored by the model under test. Re-sealing the marker from current bytes would let a post-verification rewrite through, so this deliberately does not do that. Resetting verify:<dep> and node:<dep> makes the real verifier run again over freshly produced bytes — one node re-run (~9 min for crytic) instead of stranding 21 completed nodes.

Quarantine rather than delete, because publishFileDurableExclusive → assertPublishedBytes throws existing-file-conflict if the re-run publishes different bytes over the stale file, and because the diverged bytes are the only remaining evidence of the divergence.

Scope: this is recovery, not prevention

The writer that rewrites an already-published artifact is not yet identified. Ruled out with source evidence in #275:

• publishFileDurableExclusive — assertPublishedBytes throws on differing bytes (packages/artifacts/src/safe-paths.ts:201-204, :440-445)
• reconcileTargets / copyRegularFileExclusive — both skip when the destination exists (packages/runtime/src/artifact-reconciliation.ts:99-120)
• smithers worktree sync — only fetch + rebase in a separate worktree, and R43's rebase failed non-destructively
• workspace-handoff — .ultrafuzz and artifacts are excluded via SENSITIVE_SEGMENTS

Since publication asserts byte equality, disk necessarily matched the marker when verify:property-specification-crytic completed at 08:59:42Z, so the rewrite happened afterwards, across the sandbox exit and three restore cycles before prepare:fanin ran at 09:41:14Z.

Known limitation

Resetting a dependency drops successful_nodes from 21 to 20 and back. The Modal overseer's progress ratchet is canonical.successful_nodes > next.successful_nodes (packages/modal/src/recovery.ts:458), so that round trip does not register as progress. If the rewrite recurs on every restart rather than being a one-off race, the no-progress budget could still exhaust. This is strictly better than a permanent hard stop, but it is not a guarantee, and prevention still matters.

Validation

• new regression proven red without the source change and green with it on this branch, so it is not vacuous
• pnpm --filter @ultrafuzz/runtime test
• pnpm --filter @ultrafuzz/runtime typecheck
• npx eslint packages/runtime/src/smithers.ts packages/runtime/test/runtime.test.ts
• npx prettier --check on both files
• git diff --check

R36, R37 and R43 were never stopped or restarted; R43's overseer is still running and its evidence is preserved.

Greptile Summary

The PR adds resume-time recovery for dependencies whose published artifacts no longer match their verification markers.

• Parses dependency verification failures from Smithers inspection output.
• Rechecks marker hashes against durable artifacts before acting.
• Resets the dependency verifier and records recovery evidence.
• Adds regression coverage for divergence, stale errors, in-flight resets, and active workflows.

Confidence Score: 3/5

This PR is not safe to merge because recovery can re-attest post-verification artifact modifications without rerunning the artifact-producing node.

Resetting only verify:<dependency> causes the verifier to delete the old marker, validate the existing changed bytes, and issue a new marker whenever those bytes remain contract-valid, defeating the anti-tampering guarantee.

Files Needing Attention: packages/runtime/src/smithers.ts

Security Review

The recovery resets only the verifier, so changed artifact bytes that remain contract-valid can be re-attested without rerunning their producing node. This defeats the verification marker’s anti-tampering boundary and must be corrected before merging.

Important Files Changed

Filename	Overview
packages/runtime/src/smithers.ts	Adds divergence recovery, but resetting only the verifier can re-attest post-verification artifact modifications that remain contract-valid.
packages/runtime/test/runtime.test.ts	Adds broad recovery-path coverage, but the fixtures do not test a diverged artifact whose modified bytes still satisfy its contract.

Sequence Diagram

sequenceDiagram
  participant Resume
  participant Marker
  participant Verifier
  participant Artifact
  participant Dependent
  Resume->>Marker: Detect recorded SHA mismatch
  Resume->>Verifier: Reset verify:dependency only
  Verifier->>Marker: Delete stale marker
  Verifier->>Artifact: Read changed bytes
  Verifier->>Verifier: Validate artifact contract
  Verifier->>Marker: Write marker for changed bytes
  Dependent->>Marker: Check new SHA
  Dependent->>Artifact: Accept changed bytes

Loading

Prompt To Fix All With AI

### Issue 1
packages/runtime/src/smithers.ts:975-976
**Verifier re-seals modified artifacts**

When a previously verified artifact is rewritten to different but contract-valid bytes, recovery resets only `verify:<dependency>`, which validates the modified bytes and replaces the old marker without rerunning the producing node. Dependent tasks then accept artifacts that were modified after their original verification, and the old marker evidence is lost.

**How this was verified:** The reset targets only the verifier, whose generated implementation deletes the old marker, validates the existing bytes, and writes a new matching marker.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (3): Last reviewed commit: "fix(runtime): repair a diverged dependen..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Knowledge Base — Runtime: Run Orchestration Lifecycle

[aviggiano]

Verified against R43's real bytes: the detection does reach the resume side

The main risk with this PR was that it might be dead code — that the prepare: task's failure never reaches smithers inspect --format json, so the detection would never fire in production. That is the failure mode that got #273 drafted instead of merged, so I checked it directly rather than assuming.

1. The error is persisted at the run level. From R43's preserved smithers.db snapshot, _smithers_runs.error_json is 5470 bytes and contains the signature:

{"name":"SmithersError","code":"SESSION_ERROR",
 "message":"Task failed: prepare:property-specification-fanin ...", ...}

with the nested cause carrying artifact-contract failure: artifact dependency has not passed verification property-specification-crytic for property-specification-fanin.

2. inspect surfaces it. buildInspectSnapshot parses r.errorJson and splices it in as run.error whenever present (@smithers-orchestrator/cli/src/index.js:1576-1595):

let error = undefined;
if (r.errorJson) { try { error = JSON.parse(r.errorJson); } catch {} }
...
run: { id, workflow, status, ..., ...(error ? { error } : {}) },

3. The parser extracts the right dependency from those exact bytes. Feeding R43's real error_json into the production payload shape {ok, data:{run:{..., error}}} and running the regex from this PR:

EXTRACTED DEPENDENCIES: ["property-specification-crytic"]

So for R43 this fix would have quarantined property-specification-crytic's marker and artifacts and reset verify: + node: for it, instead of re-failing prepare:property-specification-fanin on every generation until the budget exhausted. It is not dead code.

Note this also means steps[] carrying no per-task error field is irrelevant here — the signal arrives via run.error, which is why the detection scans the serialized snapshot rather than the step rows.

Validation status

• full detached pnpm --filter @ultrafuzz/runtime test: 272 pass / 0 fail, exit 0
• new regression proven red without the source change, green with it, on this branch
• prettier, eslint, git diff --check, typecheck: clean

[greptile-apps]

Verifier re-seals modified artifacts

When a previously verified artifact is rewritten to different but contract-valid bytes, recovery resets only verify:<dependency>, which validates the modified bytes and replaces the old marker without rerunning the producing node. Dependent tasks then accept artifacts that were modified after their original verification, and the old marker evidence is lost.

How this was verified: The reset targets only the verifier, whose generated implementation deletes the old marker, validates the existing bytes, and writes a new matching marker.

Knowledge Base Used: Runtime: Run Orchestration Lifecycle

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/smithers.ts
Line: 975-976

Comment:
**Verifier re-seals modified artifacts**

When a previously verified artifact is rewritten to different but contract-valid bytes, recovery resets only `verify:<dependency>`, which validates the modified bytes and replaces the old marker without rerunning the producing node. Dependent tasks then accept artifacts that were modified after their original verification, and the old marker evidence is lost.

**How this was verified:** The reset targets only the verifier, whose generated implementation deletes the old marker, validates the existing bytes, and writes a new matching marker.

**Knowledge Base Used:** [Runtime: Run Orchestration Lifecycle](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/runtime.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[aviggiano]

Closing unmerged. It targets #275, which was closed on 2026-08-06 by PR #279 (77da78a). The Greptile review scored it 3/5 with "not safe to merge… defeating the anti-tampering guarantee", and the body advertises a node: reset and artifact quarantine that the diff does not contain — the only quarantin string in it is a comment.

smithersSnapshotUnverifiedDependencies is genuinely useful and was proven against R43's real error_json; it is tracked for extraction in #324. The recovery path itself is #288's job.

Tracked in #327.