Preserve artifact authority and dynamic ancestry across continuation #1142
Description
Suggested labels: bug, priority:P0, recovery, artifacts
Problem
Continuation can lose manifests, verification markers, or dynamic fan-out membership. A later output replacement can also invalidate authority for a consumer that was already admitted. Downstream verification then rejects artifacts that were previously accepted.
Generic reproduction
- Run a workflow with dynamic fan-out followed by aggregation.
- Interrupt the controller after some dynamic nodes verify successfully.
- Resume and regenerate one upstream output.
- Observe missing ancestors, missing verification authority, or
dependency authority changed after admission.
Expected behavior
Verified generations are immutable. Consumers retain the exact generation admitted for their attempt, and continuation reconstructs the same dynamic ancestry.
Proposed fix
- Store each artifact generation in an immutable attempt directory.
- Publish the current verified generation through an atomic pointer.
- Never delete the last verified manifest before its replacement verifies.
- Persist dynamic fan-out membership in the durable graph.
- Add a repair command that rebuilds manifests and verification indexes from immutable records.
Acceptance criteria
- Controller loss during dynamic fan-out can be resumed through final aggregation.
- A consumer's admitted authority cannot change mid-attempt.
- All successful dynamic ancestors survive continuation.
- Failed replacement does not destroy the prior verified generation.
This issue intentionally omits the audited repository and all audit results.
Activity
docs(runtime): say what lock-free dynamic expansion relies on and wha…
test(runtime): cover the identity check a published expansion now rel…
fix(runtime): published dynamic expansions are the lock-free authorit…
aviggiano commentedon Sep 29, 2026
Resolved by #1163 (with #1169 and #1192 for the recovery side).
The verified failure paths were in
loadOrCreateDynamicExpansion, which treated the published expansion manifest as a cache to re-prove on every render:DYNAMIC_EXPANSION_CHANGEDforever);.expansion.lockleft by a killed process blocked every render.#1163 makes the published manifest the lock-free authority for fan-out membership: the source is read only when the manifest is created, and the lock is removed. After a reset regenerates a source, the fan-out keeps its published items. An explicit
--retry-failedof the source itself still re-expands.Deliberately not built: immutable per-attempt generation directories, an atomic current-generation pointer and a repair command (as drafted in #1154). Reviewers found that store checked a copy no consumer reads, while weakening the existing fail-closed admission check.