Repository navigation
Make run status authoritative, bounded, and read-only #1145
Description
Suggested labels: bug, priority:P0, cli, state-machine
Problem
A dead controller can leave a run marked running. Different persisted sources can describe the same workflow as finished, succeeded, and failed. The status command can hang and may update state or accounting timestamps while inspecting the run.
Generic reproduction
- Kill a detached controller without a clean shutdown.
- Wait for its lease to expire.
- Run
ultrafuzz status <run-id>repeatedly. - Compare output and file hashes before and after inspection.
Expected behavior
Status returns quickly without mutation. Reconciliation is explicit and results in one coherent state.
Proposed fix
- Separate
statusfromreconcile. - Add bounded gateway and lock timeouts.
- Define one authoritative transition table.
- Reconcile an expired controller lease to
interruptedor another explicit state. - Atomically clear controller leases at terminal transition.
Acceptance criteria
- Status completes within a documented bound.
- Repeated status calls do not alter run-file hashes.
- A dead controller cannot leave an indefinitely running state.
- Workflow, event, lease, and CLI state agree after explicit reconciliation.
This issue intentionally omits the audited repository and all audit results.
Activity
test(runtime): stop the observation test pinning a finished run's act…
fix(runtime): make run synchronization single-writer, idempotent, bou…
aviggiano commentedon Sep 29, 2026
Resolved by #1180.
ULTRAFUZZ_RUNNER_QUERY_TIMEOUT_MS, capped at 600 s), and a timeout becomes a diagnostic instead of a hang.state.jsonforstatusor for a finished run.WORKFLOW_SYNC_IN_PROGRESSand writes nothing, so observers never race.statusreports the runner's health even when synchronization fails.ultrafuzz resume <run-id>. Verified on a real campaign: after SIGKILLing the engine and supervisor,statusreportedorphanedwith that hint within ~50 s, andultrafuzz resumerecovered the run.Kept by design:
statusstill pumps synchronization. It is the only process that advancesstate.jsonand enforcesworkflow_deadline_seconds(#1110), so a read-only status would make both worse.aviggiano commentedon Oct 15, 2026
Post-closure repository-neutral observation from UltraFuzz v0.1.4:
keep_workspaces = trueand roughly 8.1 GB of retained workspaces took about 80 minutes.The current
unstablecode has bounded observe-only synchronization, so this may already be fixed after the release. Please retain a regression fixture with large retained workspaces and assert that status latency is independent of workspace byte size/file count, or that the documented observation deadline returns runner health with a stale-local-state warning.This was a P2 monitoring roadblock rather than a run-ending failure because a lower-level read-only workaround existed.