Skip to content

fix(runtime): keep the verification marker consistent when sanitizing a property lens - #279

Merged
aviggiano merged 2 commits into
mainfrom
fix/lens-sanitizer-verification-marker
Aug 6, 2026
Merged

aviggiano merged 2 commits into
mainfrom
fix/lens-sanitizer-verification-marker

Conversation

@aviggiano

@aviggiano aviggiano commented Aug 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The bug

sanitizeLensReferenceExpectationAuthority (packages/runtime/src/artifact-gates.ts) rewrites the published property lens in place:

if (removed === 0) return [];

writeJsonDurable(lensPath, { ...lens.value, properties });

lensPath resolves inside the canonical artifact directory. Two consequences:

  1. It writes with writeJsonDurable, not publishFileDurableExclusive, so assertPublishedBytes — the guard that makes published artifacts immutable — is never consulted. That is why the exclusive-publish path, artifact reconciliation, worktree sync and workspace handoff all looked innocent when I audited them.
  2. It runs from verifyRequiredArtifactsForAttempt (artifact-gates.ts:184) on the runtime sync path (workflow-sync.ts:2058), which can execute after the workflow's verify:<node> task has already published the artifact and sealed its digest in <runRoot>/.ultrafuzz-verification/<attemptId>.json.

From that moment the marker describes bytes that no longer exist. Every dependent's assertVerifiedDependency fails, and because the failure lands on the prepare: wrapper rather than a node body the durable state records no failed node, so --retry-failed has nothing to reset. The run is stranded permanently.

It only writes when removed > 0, which is exactly why a single lens diverged in R43 while its sibling artifacts stayed consistent.

Regression from #269

$ git log --oneline -S sanitizeLensReferenceExpectationAuthority -- packages/runtime/src/artifact-gates.ts
2b00c48 fix(runtime): sanitize unsupported lens reference expectations (#269)

2b00c48 is an ancestor of 4af20bf, the main commit R43 was built from, which is why R43 hit this and earlier runs did not.

R43 evidence

Run aave-v4-v0012-main-issue270-r43-invariant-only. Marker vs the only copy on the durable volume:

artifact marker sha256 on-disk sha256 match
properties/crytic.md 97b050b0…0538 97b050b0…0538 yes
findings.json 37517e5f…b570 37517e5f…b570 yes
properties/crytic.json 8cfef182…1328 19d3976b…3e9d NO

The workflow DB's verification row records the same stale digest, and the on-disk file is complete and contract-valid (102 properties). prepare:property-specification-fanin then failed identically on all three recovery generations until the no-progress budget was exhausted — 21 completed nodes and ~83.3 M tokens / ~$482 lost.

The fix

The sanitizer is trusted runtime policy stripping unauthorized reference-expectation IDs — not a model editing its own output — so keeping the marker consistent with its edit is legitimate rather than a tamper bypass. After writing, refresh the recorded sha256 for just that one path in both artifacts and publications, leave every other entry and the rest of the marker untouched, and surface it as an ARTIFACT_VERIFICATION_DIGEST_REFRESHED warning.

When no marker exists yet the verifier has not run and will hash the sanitized bytes itself, so there is nothing to do.

Validation

  • new regression proven red without the source change and green with it
  • asserts the untouched sibling entry keeps its digest and that primary survives
  • asserts a second pass is idempotent — no rewrite, no re-report — because a sanitizer that re-fired on every sync would be its own loop
  • pnpm --filter @ultrafuzz/runtime test, typecheck, eslint, prettier, git diff --check

Related

#278 attempted to recover from this state and is parked as a draft: independent review found that resetting the dependency's verifier is guaranteed to throw for the five invariant-suite nodes (rememberInvariantSuitePublications reads an in-process Map populated only while the node runs) after clearArtifactVerificationMarker has already deleted the marker, and that timetravel unconditionally deletes frames, durability snapshots and VCS tags after the target — --no-deps does not gate it. Fixing the writer is the correct layer. A follow-up worth considering separately: make the workflow verifier a pure function of on-disk state so standalone re-verification is viable at all.

Greptile Summary

The PR updates property-lens sanitization so verification-marker digests follow sanitized artifact bytes and adds explicit detection of pre-existing marker drift.

  • Refreshes the affected path in both marker artifact collections after sanitization.
  • Reports unexplained marker drift without re-sealing it.
  • Adds regression coverage for digest refresh, preserved sibling metadata, idempotency, and drift detection.

Confidence Score: 4/5

The PR does not yet appear safe to merge because an interruption between the artifact and marker writes can still leave the run permanently unable to restore a consistent verification marker.

The artifact and marker remain separate durable writes, and after the artifact has been sanitized, every later pass takes the no-removal branch that reports stale-marker drift without repairing it.

Files Needing Attention: packages/runtime/src/artifact-gates.ts

Important Files Changed

Filename Overview
packages/runtime/src/artifact-gates.ts Adds targeted verification-marker digest refresh and drift diagnostics to the property-lens sanitizer.
packages/runtime/test/artifact-gates.test.ts Adds focused tests covering marker refresh, unchanged entries, idempotency, and matching or drifted digests.

Reviews (2): Last reviewed commit: "fix(runtime): report a lens marker diges..." | Re-trigger Greptile

Context used:

Comment thread packages/runtime/src/artifact-gates.ts
aviggiano and others added 2 commits August 6, 2026 12:18
This is the writer behind #275, and it is a regression from #269.

`sanitizeLensReferenceExpectationAuthority` rewrites the published property lens
in place with `writeJsonDurable`, not `publishFileDurableExclusive`, so the
immutability guard that `assertPublishedBytes` provides is never consulted. It
runs from `verifyRequiredArtifactsForAttempt` on the runtime sync path, which can
execute after the workflow verifier has already published the artifact and sealed
its digest in a verification marker. The marker then describes the
pre-sanitization bytes forever.

Every dependent's `assertVerifiedDependency` fails from that point on, and
because the failure lands on the `prepare:` wrapper rather than a node body the
durable state records no failed node, so `--retry-failed` has nothing to reset.
Aave run R43 lost 21 completed nodes and ~83M tokens to exactly this: the marker
recorded `8cfef182…` for `properties/crytic.json` while the only copy on the
volume was `19d3976b…`, and `prepare:property-specification-fanin` then failed
identically on all three recovery generations.

The sanitizer is trusted runtime policy stripping unauthorized reference
expectation IDs, not a model editing its own output, so the marker must follow
its edit rather than contradict it. Refresh the recorded sha256 for just that one
path in both `artifacts` and `publications`, leave every other entry untouched,
and report it as a warning diagnostic. When no marker exists yet the verifier has
not run and will hash the sanitized bytes itself, so there is nothing to do.

Fixes #275

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review findings on the first commit.

The refresh only ran on the write path, so a marker that ended up describing
stale bytes without this gate having rewritten anything never repaired and never
surfaced. That is reachable: the verifier can hash the lens and publish its
marker while an earlier pass is mid-rewrite, and afterwards there is nothing left
to sanitize, so the early return skipped the refresh entirely. Downstream it
shows up only on a `prepare:` wrapper, which records no failed node and cannot be
retried.

Report that case as an error instead of re-sealing it. Re-sealing a change the
runtime did not make would be indistinguishable from laundering an unexplained
change past verification, whereas an error makes the node fail visibly so
--retry-failed can act on it. Neither the marker nor the artifact is touched.

Also record `previous_sha256` on the refresh diagnostic, since it rewrites a
security-relevant attestation and the audit trail should show both sides.

The marker fixture now carries the `contract` and `contract_digest` fields a real
marker has, and the test asserts they survive the refresh untouched, so it
demonstrates the property the change exists for rather than only that a JSON
field moved. Added coverage for the drift path and for a matching marker.

Refs #275

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rewritten published artifact strands runs: verified dependency digest mismatch fails fan-in forever with zero failed nodes

1 participant