No supported way to re-seal workflow control evidence: one divergent file permanently disables product-level status and reporting

Summary

There is no supported way to re-seal or re-bless workflow control evidence. Once a run's sealed control files diverge for any reason, WORKFLOW_CONTROL_EVIDENCE_INVALID is permanent: ultrafuzz status refuses the run for the rest of its life, so no state sync, no completion gate, and no final report can ever be produced — even while the underlying Smithers run continues to execute and finish work.

Evidence

verifyWorkflowControlSnapshot compares the published snapshot's control files byte-for-byte (packages/runtime/src/workflow-integrity.ts:291):

throw new Error(`sealed workflow control file changed: ${controlFileLabel(key)}`);

which start-run.ts converts into the terminal diagnostic:

code:    WORKFLOW_CONTROL_EVIDENCE_INVALID
message: sealed workflow control file changed: workflow

In an 18-hour local run this fired from a single changed file. Verifying the seal by hand:

all 44,345 sealed execution_files hashed clean — zero mismatches
the only divergence was the published snapshot's .smithers/workflows/<run>.tsx:

seal expects    f37c4eb6…  2007734 bytes
generated copy  f37c4eb6…  2007734 bytes   ← still byte-identical to the seal
snapshot copy   96c367e8…  2008029 bytes   ← diverged

So the damage was one file out of 44,346, and the pristine bytes were still sitting on disk. There is still no way to recover, because sealWorkflowControlFiles refuses to write a second seal (workflow-integrity.ts:233):

if (pathEntryExists(paths.integrityPath)) {
  throw new Error("workflow control seal already exists");
}

There is no --reseal, no --accept-control-change, no generation bump. The run is bricked at the product level regardless of how repairable the underlying divergence is.

Why the divergence happened

Because agentic nodes compile with retries: 0 (filed separately), the only available way to grant a failed node another attempt was to patch the sealed snapshot's workflow. Smithers itself accepts this via smithers up --force --accept-workflow-change, and the run genuinely resumed and kept finishing nodes. Ultrafuzz then had no matching concept, so the product layer was locked out of a run that was still alive and progressing.

That asymmetry is the core defect: Smithers has a supported "I acknowledge the workflow changed" path and Ultrafuzz does not.

Impact

One divergent file permanently forfeits product-level status, sync, completion gating, and final reporting for that run.
The failure is silent in the wrong direction: smithers status reports a healthy running run while ultrafuzz status refuses it, so a run can look fine and be unfinishable.
Long runs are the ones most likely to need mid-flight recovery and the most expensive to discard, so the policy bites hardest exactly where it costs most.

Proposal

Add a supported re-seal path that preserves auditability rather than discarding it:

ultrafuzz reseal <run-id> (or --accept-control-change on the existing resume path) that recomputes the seal, bumps the control generation, and appends a provenance record: what changed, old and new digests, timestamp, and operator-supplied reason.
Keep the strict default. Re-sealing should be explicit, logged, and visible in the final report as a deviation, never implicit.
Report and validation output should surface "this run's control evidence was re-sealed N times" so a re-sealed run is never mistaken for a pristine one.
Consider a --dry-run that reports exactly which control files diverged and whether pristine bytes are still recoverable, which would have reduced the diagnosis here from hours to seconds.

Acceptance criteria

A run whose control evidence diverged can be returned to a verifiable state without discarding completed work.
Re-sealing bumps the generation and records an auditable provenance entry.
Re-sealed runs are explicitly marked as such in status and report output.
A diagnostic names the specific divergent control files instead of only the first one it hits.

Second independent reproduction, and a fix that does not require re-sealing

Corroborating this from a different run. Same failure, different digests, so it is not specific to one
workspace:

seal expects    a070815c…  1576623 bytes
generated copy  a070815c…  1576623 bytes   <- still byte-identical to the seal
snapshot copy   e00842a9…  1576639 bytes   <- diverged

All eight sealed control files verify clean by hand — graph, expanded_graph, graph_fingerprint,
config, tasks, input, workflow, evidence_workflow — and the only divergence is the published
snapshot's generated workflow. Diffing the pristine copy against the snapshot showed the whole divergence
was four call sites renamed to a helper that is actually declared in the generated workflow, i.e. exactly
the "patch the sealed snapshot to grant a failed node another attempt" scenario described here. Four
renames × 4 characters = the 16-byte delta.

So this reproduces the described mechanism precisely, including the asymmetry: smithers status reported
running-healthy throughout while ultrafuzz status refused the run.

The part I think is missing from the proposal

The proposal here is a re-seal path, and that is worth having. But there is a second, independent defect
that a re-seal would only paper over: status never needed execution-grade authority in the first
place.

status  ->  getRunHealth (state-export.ts)
        ->  readLinkedWorkflowEvidence (start-run.ts)
        ->  acquireWorkflowControlLock
            verifyWorkflowControlSnapshot          <- throws
            materializeWorkflowExecutionSnapshot
            plan / data-governance / link-journal cross-binding

readLinkedWorkflowEvidence is the authority a start/resume legitimately needs: it takes the control
lock, calls reconcilePendingWorkflowRunLink, materializes the execution snapshot, and cross-binds the
plan, data governance, and link journals. A status read needs none of it. It needs the workflow run ID
and the run layout so it can ask the runner for a status summary — which is why smithers status keeps
working when ultrafuzz status does not.

Two consequences worth separating from the re-seal discussion:

A read-only command acquires acquireWorkflowControlLock and runs reconcilePendingWorkflowRunLink,
which mutates link state. Neither belongs on a status read, independent of any seal question.
Decoupling status from execution authority restores observability without ever blessing a
divergence. Re-sealing, by design, marks tampered evidence as acceptable; keeping status strict-but-
readable does not. If both land, the re-seal path can stay rare and deliberate rather than being the
only way to see a run again.

Suggested split, keeping this issue's scope intact:

status (and any other read-only surface) resolves the run link without execution authority, and
surfaces seal divergence as a warning in its output rather than replacing the output with an
error. start / resume keep today's hard failure unchanged.
status no longer takes the control lock or reconciles run links.
The re-seal path as proposed above, for returning a run to genuinely verifiable state.

Happy to take the first item, since it is small and does not touch the trust model.

Navigation Menu

Repository navigation

No supported way to re-seal workflow control evidence: one divergent file permanently disables product-level status and reporting #674

Summary

Evidence

Why the divergence happened

Impact

Proposal

Acceptance criteria

Second independent reproduction, and a fix that does not require re-sealing

The part I think is missing from the proposal

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

No supported way to re-seal workflow control evidence: one divergent file permanently disables product-level status and reporting #674

Description

Summary

Evidence

Why the divergence happened

Impact

Proposal

Acceptance criteria

Activity

aviggiano commented on Aug 19, 2026

Second independent reproduction, and a fix that does not require re-sealing

The part I think is missing from the proposal

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions