[aviggiano]

Root cause

Two defects in the generated Smithers workflow template, in the same mechanism.

#217 — an ancestor resurrects a source a later stage deleted. artifactAncestorNodeIds (packages/runtime/src/smithers.ts) walks dependsOn transitively, so every invariant stage's dependencyArtifactDirs is the full transitive ancestor closure, not just its declared parents. materializeInvariantSuiteFromDependencies iterated all of those dirs and selected every suite path present. The published manifest carried only schema_version, producer_node_id, producer_attempt_id and files — there was no deletion channel. recordInvariantSuiteTombstone only mutated a module-level Map keyed by the deleting stage's workspace root, so a descendant's lookup was always empty for a path a predecessor deleted. Net effect: handlers correctly omits a deleted file from its own artifact, and coverage re-selects it straight out of the setup artifact and copies it back into the workspace.

#219 — a restart between an invariant agent task and its verifier killed the run, or silently reverted it. invariantSuiteTombstones, invariantSuiteDependencySnapshots and invariantSuitePublicationSnapshots were plain module-level Maps with no persistence anywhere. The verifier is its own <Task retries={0}> depending on the agent task, while publication and the first verifyArtifacts run inside the agent-generate wrapper. So once the agent node is durably complete, a restart runs only the verifier, and rememberInvariantSuitePublications hard-threw invariant suite publication snapshot is unavailable on the empty Map. With retries: 0 that killed the run — the R31/R43 signature. Symmetrically, with the in-process snapshot lost, the post-agent preparation pass fell through to a fresh selection and re-copied ancestor bytes over the harness the agent had just authored.

A correction worth recording: the "mutated ancestor bytes" framing in the original triage is wrong. assertTaskInputs → assertVerifiedDependency → rememberExpectedInvariantSuitePublications already validates every ancestor manifest, digest-checks every ancestor suite file and cross-checks the verification marker under <runRoot>/.ultrafuzz-verification/ on every preparation. Ancestor byte integrity was never the gap, and this PR does not touch that path.

The fix

#217 — a typed, additive deletion channel. The published manifest gains tombstones, validated per entry through assertSafeInvariantSuitePath and bounded by assertInvariantSuiteTombstoneBudget. It is an additive v1 field, so manifests already on disk from in-flight runs stay readable (absent = empty set). inheritedInvariantSuiteTombstones reads the manifests of the task's declared dependencies and returns (union of their tombstones) minus (union of their files); a tombstoned path is skipped during selection, so it is neither republished to this stage's artifact nor copied into the workspace, and the effective set is republished so the channel accumulates down the chain.

Restricting tombstone authority to declared predecessors is deliberate: it is exactly right for the linear invariant chain in .ultrafuzz/topology.yml, and the minus files term is what keeps a deleted-then-re-added source alive. dependencyArtifactDirs has no reliable topological order, so an ordering-sensitive rule was rejected.

#219 — durable handoff provenance that can be superseded. The exact materialized handoff is recorded at <runRoot>/invariant-suite-handoffs/<attemptId>/handoff.json (attempt, producer, path, size, digest, directness, tombstones, and a fingerprint of every ancestor invariant-suite manifest it was derived from), written after the copy loop. loadInvariantSuiteDependencyHandoff rebuilds it, re-reads each recorded byte from the ancestor artifact and fails closed on mismatch, so a post-restart pass keeps the handoff it used instead of reverting authored sources. rememberInvariantSuitePublications rebuilds its expectation from the durable manifest the attempt already wrote rather than hard-throwing, and resolveInvariantSuiteDependencySnapshot treats an empty selection as the legitimate recorded outcome it is for the first invariant stage.

Per the design constraint: every recovery path reads disk. No recovery path reads an in-process Map (the PR #278 mistake), and the empty case is a diagnosable typed failure naming the missing file rather than an unconditional throw.

What the review round changed

The first three findings below were major and are the reason this PR has a second commit.

The record can now go stale, and staleness is not tampering. Two reviewers independently reproduced the same terminal path: setup publishes v1, handlers records its handoff, an operator legitimately re-executes setup (retry-task / timetravel / a new Modal execution generation calling replacePublishedDirectory) and it republishes v2 with a fresh, internally consistent manifest and verification marker. assertVerifiedDependency passes. Then the stale record's digest check threw artifact-contract failure: invariant suite dependency changed … — from a retries={0} preparation node, on every subsequent resume, because the record is durable and attemptId === stableBaseAttemptId(node.id) is stable for the life of the run root. On main the same situation self-healed by re-deriving. That is precisely the "converts a stall into TERMINAL loss" shape this PR exists to remove, reached from disk instead of from a Map, and the operator saw the tamper message for a legitimate upstream retry.

The record now binds itself to producers: [{attempt_id, manifest_sha256}]. A fingerprint mismatch means "this record describes a superseded handoff": it is discarded and materialization re-derives against the ancestor's new, already-verified bytes. The hard failure is retained for exactly the tampering case — ancestor manifest byte-identical, recorded suite bytes not. Ancestor byte integrity is already guaranteed by assertVerifiedDependency, so failing closed here was buying no integrity.

The early return no longer trusts that the copies are still on disk. returning as soon as a record loaded rested on "a record implies the copies landed", which is an assumption about worktree lifetime, not a checked invariant — loadInvariantSuiteDependencyHandoff verifies ancestor bytes and never looks at the workspace. If the process died between writing the record and captureInvariantSuiteWorkspaceSnapshot, and the worktree was then reclaimed or reconstructed, resume found no snapshot to restore, returned without copying, and the stage's agent ran against a workspace with no inherited harness and no error. Worse, captureInvariantSuiteWorkspaceSnapshot is write-once, so the empty workspace was immediately persisted as the authoritative snapshot and every later retry restored the loss.

reconcileInvariantSuiteWorkspace now repopulates from the record — gated on the durable workspace snapshot being absent (the only window in which nothing else can restore the worktree) and on each individual path being absent (so the post-agent pass never overwrites a source this attempt authored and never resurrects one it deleted). The early return also no longer skips the ancestors' implemented-property expectation check: a record binds the suite bytes, not the ancestors' ledgers, so that fail-closed check is re-run on the recovered path too.

A retry no longer publishes the previous attempt's deletions. invariantSuiteTombstones is keyed by realpathSync(workspaceRoot) and was never cleared. resetTaskArtifactsForRetry restores the durable workspace snapshot — putting a file the attempt-1 agent deleted physically back — but left the tombstone. On attempt 2 the file is unchanged, so it is not re-discovered, yet the manifest still carried it as a tombstone, and implement-properties and campaign then suppressed it permanently: the campaign workspace Recon fuzzes loses a harness file, the suite fails to build, and it reads as model quality. Before this PR the same stale drop was accidentally healed by the #217 resurrection; the fix removed the healing while leaving the staleness. resetTaskArtifactsForRetry now drops the attempt's tombstone set. Nothing is lost: tombstones are fully re-derived from the protected baseline on every companions pass. (Precondition: max_attempts > 1, which the shipped topology leaves at 1 but benchmarks/smoke-benchmark.yml and packages/topology/src/expand.ts support — latent, not live.)

The record's tombstones are now consumed, not merely written. All three reviewers noted the field was parsed, path-checked and budget-checked and then discarded by every caller, advertising durability the code did not provide. materializeInvariantSuiteCompanions now prefers the record's deletion channel over recomputing from the ancestor manifests: the record is the durable statement of what the handoff actually suppressed, and recomputation is the fallback for an absent or superseded record, where re-derivation is the correct answer anyway.

The "record is unavailable" diagnostic is side-effect free. It formatted its path by calling a helper that mkdirSyncs two directories and can throw unsafe invariant workspace snapshot root … — so in exactly the run-root states where diagnosability matters most, the operator got an unrelated path-safety error instead of "the record is missing, here is the file". The message now uses a pure path computation.

Regression tests and red/green evidence

packages/runtime/test/invariant-suite-handoff-durability.test.ts, 15 cases. All 15 pass on this branch. The tests drive the extracted template helpers with injected collaborators against a real filesystem, because workflow.tsx is a generated template with a __ULTRAFUZZ_TASK_SPECS__ substitution point and cannot be imported directly.

Against the pre-#217/#219 template (77da78a): 13 of 15 fail.

not ok  1 - #217 a source deleted at an invariant stage is not resurrected from an indirect ancestor
ok      2 - #217 a declared predecessor that still carries a tombstoned path outranks the tombstone
not ok  3 - #217 a path this stage deletes and re-creates in the same attempt is not published as a tombstone
not ok  4 - #217 the published invariant-suite manifest carries deletion tombstones
not ok  5 - #219 publication verification recovers from the durable manifest after a restart
not ok  6 - #219 restart recovery still fails closed on tampered or unavailable suite artifacts
not ok  7 - #219 post-restart preparation keeps the recorded handoff instead of reverting authored sources
not ok  8 - #219 the durable handoff record fails closed when a recorded dependency no longer matches
not ok  9 - #219 a legitimately republished ancestor supersedes the record instead of failing the stage closed
ok     10 - #219 a worktree lost before the durable workspace snapshot exists is repopulated from the record
not ok 11 - #219 the record never resurrects a source the agent deleted once the workspace snapshot is durable
not ok 12 - #219 the recovered path still fails closed on an ancestor property with no published suite source
not ok 13 - #219 the durable record's tombstones drive the deletion channel this stage publishes
not ok 14 - #217 a retry drops the previous attempt's tombstones so a restored source is not suppressed
not ok 15 - #217 an emptied test-tree source is tombstoned on both discovery paths
# tests 15 / # pass 2 / # fail 13

Failure 5 is the exact invariant suite publication snapshot is unavailable message that killed R31/R43. Failure 7 is the silent revert. Failure 1 is the #217 resurrection.

Cases 2 and 10 are green on the pristine tree, and that is expected, not a gap. Case 2 asserts a live source is not suppressed — on a tree with no deletion channel there is nothing to suppress. Case 10 asserts a lost worktree is repopulated — on a tree that always re-derives, it always was. Both pin rules that only exist here, so both are proved by mutation instead.

Every safety-critical clause was mutated in turn, and each mutation killed exactly one test — no mutation was silent, and no mutation killed a second test:

Mutation	Test that caught it
tombstone selection suppression removed (the original #217 bug)	#217 … not resurrected from an indirect ancestor
present-path subtraction removed from inheritedInvariantSuiteTombstones	#217 a declared predecessor that still carries a tombstoned path outranks the tombstone
published-path tombstone clearing removed from the manifest	#217 a path this stage deletes and re-creates in the same attempt …
ancestor-fingerprint staleness check removed	#219 a legitimately republished ancestor supersedes the record …
workspace reconciliation dropped from the recorded path	#219 a worktree lost before the durable workspace snapshot exists …
reconciliation's snapshot gate removed (made unconditional)	#219 the record never resurrects a source the agent deleted …
expectation re-check dropped from the recorded path	#219 the recovered path still fails closed on an ancestor property …
record tombstones no longer preferred over recomputation	#219 the durable record's tombstones drive the deletion channel …
resetTaskArtifactsForRetry tombstone clear removed	#217 a retry drops the previous attempt's tombstones …
Git-fallback emptied-source tombstone removed	#217 an emptied test-tree source is tombstoned on both discovery paths

The first review round was right that the over-suppression guard was vacuous as originally written: it put the tombstone on setup's manifest, but setup is an indirect ancestor of coverage, so inheritedInvariantSuiteTombstones skipped it at the directDependencies filter and the present subtraction was never reached — deleting either rule left the suite green. The tombstone now sits on handlers, the direct dependency, which is the only shape that reaches the subtraction, and the mutation table above is the proof. The emptied-source tombstone is now driven against a real temporary git repository, which is the one path that needs one.

Review findings

14 actionable findings across three lenses (correctness-regression, blast-radius-compatibility, durability-crash-restart); 5 major. Several were raised by more than one reviewer.

Addressed in code (13):

1. major ×2 (blast-radius + durability, both reproduced empirically) — durable record has no invalidation path, so a legitimately republished ancestor permanently kills a retries=0 node → ancestor manifest fingerprints; the record is superseded, not treated as tampering.
2. major (correctness) — early return skips materialization without checking the recorded files exist, and the resulting empty workspace is frozen into the write-once durable snapshot → reconcileInvariantSuiteWorkspace, gated on snapshot absence and per-path absence.
3. major ×2 (correctness + durability) — stale in-process tombstones from a prior attempt published as durable chain-wide deletions → cleared in resetTaskArtifactsForRetry.
4. major (correctness) — the claimed over-suppression guard test did not exercise the rule it claimed to pin; both implementations of it could be deleted with the suite green → test reshaped onto the direct dependency, plus a companion case for the published-path clearing. Both now mutation-proven.
5. minor ×2 (blast-radius + durability) — early return skips workspace materialization even when the durable workspace snapshot is absent → same reconciliation, which also preserves the anti-revert property.
6. minor (correctness) — the Git-fallback emptied-source tombstone had zero behavioural coverage → real-git-repo test covering both discovery branches.
7. nit ×2 (blast-radius + durability) — handoff.json tombstones written and validated but never read back → now consumed by materializeInvariantSuiteCompanions.
8. nit (correctness) — the "record is unavailable" message was built by a function that mkdirs and can throw a different error → pure path computation.

Accepted as a documentation correction rather than a code change (1):

• minor ×2 (blast-radius + durability) — "the new emptied-source tombstone on the Git-fallback path is unreachable, so the claimed extra Preserve invariant-suite deletions and renames across retries #217 gap is not closed there." The reviewers are right and the original claim was overstated. changedTestTreePaths has exactly one caller, materializeInvariantSuiteCompanions, which always passes invariantSuiteProtectedBaselinePath(task); captureInvariantSuiteBaseline writes that file on every preparation and it lives outside the model-controlled workspace, so the baseline branch always wins and the Git-fallback block is not reached for the five invariant nodes. The truncation case was already handled on the baseline branch before this PR. The line is kept as defence-in-depth for a caller that has no baseline, and it now has coverage, but it closes no gap that was open. Corrected here rather than by deleting the line, because one reviewer explicitly offered that resolution and the branch is otherwise silently wrong if it is ever reached.

Rejected: none.

Residual risk

• workflow.tsx is covered by neither pnpm -w typecheck (packages/runtime/tsconfig.json includes only src/**/*.ts) nor pnpm -w lint (the glob is .ts/.js/.mjs, not .tsx). Compensated by a standalone tsc --noEmit --strict --jsx react-jsx --types node over the pristine, first-commit and final versions of the template: all three produce a byte-identical 51-error profile, i.e. zero new type errors from either commit. The template's real type surface is still only fully bound at run time once __ULTRAFUZZ_TASK_SPECS__ is substituted.
• Upgrade boundary. The manifest gains a field, so its bytes change. Producer and consumer read the same file in the same pass, so this is self-consistent, but a run resumed across this upgrade reads pre-upgrade ancestor manifests as having zero tombstones — stages that already published get no retroactive deletion protection. Likewise a pre-upgrade handoff.json has no producers array and is discarded as unbindable, which re-derives (safe) rather than failing.
• Tombstone authority is restricted to declared dependencies. Correct for the linear invariant chain; for a hypothetical multi-parent invariant DAG a deletion is honoured only if no declared parent still carries the path.
• Cost. loadInvariantSuiteDependencyHandoff now re-digests the ancestor manifests on each call and is called from more than one site per pass. Bounded by the existing 512-file / 64 MiB suite budgets, but it is more I/O per preparation than before.
• inheritedInvariantSuiteTombstones throws rather than skipping on a structurally invalid direct-dependency manifest. assertVerifiedDependency fails closed on the same manifest earlier in prepareArtifactMirror, so this should be unreachable — but it is a stricter posture than the old continue.
• The end-to-end restart is inferred, not executed. All tests drive extracted template helpers against a real filesystem; none exercises a live Smithers run, a real controller restart, or the Modal cloud execution path. The retries: 0 verifier and the separate preparation node were verified in source.
• The test file imports typescript to erase type annotations from the extracted helpers. It is a root devDependency, resolving via the root node_modules exactly as tsc, prettier, eslint and @types/node already do for this package. A change to the install layout would break the test harness, not the product code.

Validation

Ran from /home/ubuntu/uf-fix-217-219; df -h / checked before every heavy gate (166–176G free on /, 66% used).

• pnpm -w run ci → exit 0 (format:check "All matched files use Prettier code style!", lint clean, build across all 13 projects, validate:release wrote its report).
• pnpm -w typecheck → exit 0, all 13 workspace projects.
• git diff --check → clean.
• New focused suite → 15/15 (was 2 pass / 13 fail against 77da78a).
• Full runtime node suite (node scripts/run-tests.mjs, all 14 files) → 306 tests, 0 fail, 15 skipped.
• packages/runtime/test/generated-workflow-verifier.test.ts → green. This matters: that suite extracts template helpers by source-text offsets, so all new code was placed outside its slice ranges.
• Mutation testing: 10 mutations, each killing exactly one test (table above). The template was restored byte-for-byte after each.

Merge order

This PR should merge before #272 (fix/prepare-verify-wrapper-failure-surface). Both are runtime fixes; #272 is the narrower change and touches packages/runtime/src/workflow-sync.ts rather than the template, so it rebases cleanly onto this.

Fixes #217

Fixes #219

Greptile Summary

The PR adds a durable deletion channel and disk-backed provenance for invariant-suite handoffs, including restart recovery and supersession when ancestors republish. One same-process retry path still bypasses supersession.

• Adds tombstones to invariant-suite manifests and propagates them through declared dependencies.
• Persists dependency selections, tombstones, and ancestor-manifest fingerprints in per-attempt handoff records.
• Rebuilds publication expectations after restart and reconciles lost worktrees before snapshot capture.
• Adds focused filesystem-backed regression coverage for deletion, restart, retry, and tampering behavior.

Confidence Score: 4/5

The PR should not merge until same-process retries revalidate ancestor manifest fingerprints and re-derive superseded dependency handoffs.

A retained module-level dependency snapshot causes legitimate upstream retries to either terminate downstream preparation when bytes change or silently omit new paths and tombstones, bypassing the recovery behavior this change introduces.

Files Needing Attention: packages/runtime/src/templates/smithers/workflows/workflow.tsx

Important Files Changed

Filename	Overview
packages/runtime/src/templates/smithers/workflows/workflow.tsx	Adds durable tombstones and handoff provenance, but the populated in-memory snapshot branch bypasses ancestor-republication supersession.
packages/runtime/test/invariant-suite-handoff-durability.test.ts	Adds extensive extracted-helper regression tests, but does not cover ancestor republishing while the same process retains a dependency snapshot.
CHANGELOG.md	Documents the deletion-channel and durable-handoff behavior introduced by the runtime template changes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Materialize dependency suite] --> B{In-memory snapshot exists?}
  B -- Yes --> C[Check only previously selected bytes]
  C --> D[Return without fingerprint comparison]
  B -- No --> E[Load durable handoff]
  E --> F{Ancestor manifest fingerprints match?}
  F -- Yes --> G[Reuse recorded handoff]
  F -- No --> H[Re-derive selection and tombstones]
  D -. Upstream retry is missed .-> I[Stale suite or terminal mismatch]

Loading

Prompt To Fix All With AI

### Issue 1
packages/runtime/src/templates/smithers/workflows/workflow.tsx:3019-3029
**In-memory snapshot bypasses supersession**

When an upstream invariant stage republishes during a same-process retry or timetravel, this branch reuses the retained dependency snapshot without comparing manifest fingerprints. Changed bytes terminate downstream preparation, while added paths or tombstones are silently missed, leaving the downstream suite stale or resurrecting a deleted source.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "fix(runtime): make the durable invariant..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Knowledge Base — Runtime: Run Orchestration Lifecycle

[greptile-apps]

In-memory snapshot bypasses supersession

When an upstream invariant stage republishes during a same-process retry or timetravel, this branch reuses the retained dependency snapshot without comparing manifest fingerprints. Changed bytes terminate downstream preparation, while added paths or tombstones are silently missed, leaving the downstream suite stale or resurrecting a deleted source.

Knowledge Base Used: Runtime: Run Orchestration Lifecycle

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/runtime/src/templates/smithers/workflows/workflow.tsx
Line: 3019-3029

Comment:
**In-memory snapshot bypasses supersession**

When an upstream invariant stage republishes during a same-process retry or timetravel, this branch reuses the retained dependency snapshot without comparing manifest fingerprints. Changed bytes terminate downstream preparation, while added paths or tombstones are silently missed, leaving the downstream suite stale or resurrecting a deleted source.

**Knowledge Base Used:** [Runtime: Run Orchestration Lifecycle](https://app.greptile.com/monad-foudnation/-/custom-context/knowledge-base/monad-developers/ultrafuzz/-/docs/runtime.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.