FABPUB: WriterGenerationLease.v1 carries no holder liveness — orphaned lease files block partition rotation's DRAINING forever (4 orphans on claw's omniagent-plus namespace) #820

Activity

1. ViperJuice

   added

   bugSomething isn't workingSomething isn't working

   on Sep 9, 2026
2. ViperJuice commented on Sep 9, 2026

   

   ViperJuice

   on Sep 9, 2026

   CollaboratorAuthor

   More actions

   This is now load-bearing for an operator procedure, so recording what it forced.

   Lane D5 of #789 needs four orphaned generation-leases/*.json removed before the omniagent-plus partition can be rotated — the drain counts lease files, so an orphan blocks it exactly like a live writer. Writing the removal procedure ran straight into this issue, and the cross-vendor board refused three drafts of it before one held.

   What does not work, established during review:

   • Process scans cannot authorise the removal, individually or jointly. acquire (live.py:725-752) takes exclusive() only for the duration of the write, then emits {schema, generation, nonce} and returns — no pid, no retained flock, no retained descriptor. So a broker embedded in a process whose argv names neither phase-loop nor run-train, whose cwd is outside the worktree, and which holds no open descriptor under the namespace, escapes the argv scan, the /proc/*/cwd scan and the /proc/*/fd scan at once.
   • A start-time cutoff does not work either, which is the subtler one and cost a review round. "Stop every process older than the newest lease" looks sound and is not: a broker can take a lease and then fork a detached child that outlives it, and that child starts after the lease whose use it inherits. It is outside the selected set by construction, it survives kill -TERM -<pgid> of its parent's group, and it is invisible to ps --ppid once reparented.

   What the runbook therefore requires (docs/fabpub-partition-rotation-runbook.md §6, landed in #821): a maintenance reboot with the launchers disabled, and nothing weaker. A reboot kills every userspace process regardless of pgid, start time, or argv, so it dominates every selection rule. The three scans remain, demoted to confirming the boundary held rather than establishing it. Proceeding without a reboot is defined as a deviation requiring an explicit recorded acceptance that a surviving holder cannot be ruled out.

   Verified during review that the reboot is a complete boundary for this state and not merely a strong one: findmnt puts both the broker container and the FABPUB authority root on local ext4 with no network filesystem mounted, so no off-host process can hold one of these leases. That check is host-specific — it would have to be redone anywhere the state lives on shared storage.

   Why this issue still matters. A reboot is a heavy price for "we cannot tell whether a JSON file has an owner." A liveness token in WriterGenerationLease.v1 — a pid plus a start-time or boot-id, or a held flock — would make orphan detection decidable and reduce this to a scan. Until then, every future rotation on any host pays the same cost.

   Bounding the residual, since it came up: if a holder did survive, deleting its lease destroys the drain's evidence but does not let it corrupt the retired generation silently. Its writes are fenced by the DRAINING latch and require_current_generation, and by the successor's generation nonce after the flip; a predecessor-digest change between attestation and drain refuses the ceremony fail-closed. One reviewer ran eleven require_current_generation states against that claim and found a third fence the runbook does not name (_require_active_generation_root, live.py:822-838) — it understates rather than overstates.

   🤖 Generated with Claude Code

   https://claude.ai/code/session_01Rv2aKsUWdEKoWfB5PTpD1B
3. ViperJuice commented on Sep 11, 2026

   

   ViperJuice

   on Sep 11, 2026

   CollaboratorAuthor

   More actions

   Follow-up from the non-reboot investigation on September 11. No live lease removal, rotation, reboot, process shutdown or source change was performed.

   New host evidence: Claw already booted on September 10 at 00:42:38 UTC (kernel boot identity and wtmp agree). The two retained agent-harness leases have September 4 birth/ctime values and byte-match the authenticated September 8 recovery backup. These are a different namespace from the four omniagent-plus leases in this issue's original report. The earlier explanation that a new reboot was necessarily the next step omitted this existing boundary.

   Why that does not establish safe cleanup: a fresh process can explicitly reconstruct WriterGenerationLease(latch, persisted_generation, persisted_nonce) and pass validate_lease without modifying file bytes or ctime. This was reproduced only in a scratch namespace. Normal acquire() does create a new nonce and normal close releases only its own record; no normal adoption caller was found in the scanned runtime. That source trace is narrower than proving no current handle exists across all callers/restores.

   A second boundary matters: an injected pause after GitHubBrokerAdapter's generation validation, followed by scratch lease release and DRAINING, still reaches the fake push and fake gh pr create when resumed. There is no held generation lock spanning the external effect, and PR creation has no additional generation check. No real git/gh provider command ran. This rules out freeze/rotate/thaw as a sound workaround. Existing A17 flip-fence and A26 concurrent-store-writer tests both pass, so those local-store controls do not cover this provider scheduling gap.

   Verification used broker live.py at default branch 88833fff1cce8ab1076f3be8a1596d875ddb5f20, byte-identical to the installed task broker, four scratch lease assertions, the separate injected provider-gap probe, and the two unchanged store-fencing tests. The same provider-gap probe also passed against the unchanged default-branch source via PYTHONPATH, independently of pending r7 packaging fixes. No full-suite rerun or weakened frozen test.

   Native four-seat investigation review is terminal: Codex/Gemini/Grok each PARTIALLY AGREE; Fable returned empty DEGRADED/claude_tui_stalled. All cleanup/quiescence receipts verify. No recovery approval follows.

   Prospective scope: a repository-scoped publisher gate spanning external effects and child lifetime, acquisition exclusion during recovery, inheritance-aware liveness evidence, and an explicit legacy-v1 migration proof. PID-only metadata, timestamp-only reaping, or a new-runtime install alone cannot retroactively establish custody for the current records. Keep the agent-harness#789 D4 attestation, D5 fleet and no-hidden-retry constraints unchanged. The investigation does not request or record acceptance of residual risk.
4. ViperJuice commented on Sep 11, 2026

   

   ViperJuice

   on Sep 11, 2026

   CollaboratorAuthor

   More actions

   Update: the blocking instance is cleared, but the design gap this issue names is not

   The four leases described above were removed and the rotation completed on 2026-09-10 (generation 0 → 1 ACTIVE, receipt LegacyRepositoryPartitionReceipt.v3). Full record on #789.

   How orphanhood was decided matters more than that it was, because this issue's point is that it cannot be decided from the lease bytes:

   • all four lease files carried mtimes of 09-04 and 09-05, before the 2026-09-10 00:42:39 boot;
   • PID 1 started at the boot, so no process on the host predated it;
   • the broker directory is on local ext4 (df -T → ext4 on /), so no remote holder was possible.

   No holder could exist. That is a boot-boundary argument, not a liveness check — it works only because the machine restarted, and it tells you nothing on a running host. The files were backed up before removal, and the removal ran under a guard that re-read writer state in the same command as the mutation (leases=4, writers=0 at the instant of the delete).

   Why this should stay open

   The gap is unchanged: acquire (live.py:725-752) writes JSON with no pid, no flock, and no retained fd, so a lease still cannot be distinguished from an orphan by any means available to the broker. The next blocked rotation on a host that has not rebooted faces exactly the situation this issue describes, with no reboot to appeal to.

   Worth recording as prior art for whoever fixes it: a reboot is a sound decision procedure once, and only because the kernel guarantees no process survives it. Anything weaker — mtime age, a /proc scan, absence of a matching command line — is not, since a detached child can inherit a lease and start after any scan.

   Everything above is read-only observation plus the one recorded mutation; no change to the lease format or the drain logic.

   🤖 Generated with Claude Code

   https://claude.ai/code/session_01JcnSE4cpsabF41aAbjpM1g
5. ViperJuice commented on Sep 12, 2026

   

   ViperJuice

   on Sep 12, 2026

   CollaboratorAuthor

   More actions

   The bounded synchronous-effect repair is implemented and independently reviewed locally. This does not close #820 or authorize legacy lease cleanup, rotation, publication recovery, or a Claw reboot.

   The new outer latch section covers complete generation validation and each existing synchronous git push / gh pr create call separately. Local probes and confirmation reads remain outside it. A completed drain between the calls refuses PR creation; if terminal recording is also refused under DRAINING, the actual BrokerService retains its in-flight evidence and unsealed owner.

   Evidence against the explicit pending #789 dependency at 6e2b2a2901aaa225b39d986deab5927fbbbce909:

   • Before source changes: 22 new failures, 138 passes, no errors or skips. All 133 existing regression nodes passed. The independent process probe proved the generation lock was free during mutations; both service lease arrangements reached PR-create after an acknowledged drain.
   • After repair, the identical approved native command passed 160 tests, with no failures or skips. The runner's sealed verification artifact validates successfully.
   • Three isolated mutation controls failed at the intended assertions: removal of the held section (4 nodes), removal of the create guard (2), and moving validation before the lock (1). The new test module is byte-identical across RED, GREEN and mutants; all existing selected test/guard files match the composed base.
   • Fresh native Grok, Fable, Codex and Gemini reviews each returned usable AGREE on the exact source candidate. Exact-input, no-tools, transport, cleanup and physical raw-retention checks passed. Source-review result SHA256: 68fca4a88a86ce139d43cf947a53717ebb023ac61b53dc744a4b30c249f5f51e.

   The owned worktree is agent-harness-820-effect-fence-20260912, branch codex/v10-publisher-effect-fence-820-20260912. It remains uncommitted and unpushed to this repository because the normal publication-owner recovery gate remains blocked. There is no new PR or landing. Encrypted private recovery preserves the candidate, all test outcomes and complete provider sessions; original raw files are retained.

   Limits remain explicit: hung synchronous calls can block same-namespace activity indefinitely; parent death, exec/fork descendants and persisted-v1 handle adoption require a separate custody/admission contract. The #833 malformed-routing compatibility path is unchanged. #834 is another agent's separate unmerged change and has not been incorporated or approved here. The source-mode installed contract-floor warning is retained; these checks do not establish production or downstream V10 acceptance.
6. ViperJuice commented on Sep 13, 2026

   

   ViperJuice

   on Sep 13, 2026

   CollaboratorAuthor

   More actions

   The bounded synchronous-effect repair is now committed locally at
   71c4ba73c1df8315e6fbe0d6fbc76546ad300bea on
   codex/v10-publisher-effect-fence-820-20260912; the owned worktree is clean.

   Before committing, I reverified that the implementation, test, runbook and plan
   bytes still match the prior four-vendor approved candidate. The sole subsequent
   manifest delta records native completion metadata. The original native 160-pass
   result, six frozen test hashes and all review/evidence hashes still match.
   Scoped secret scanning passed. No unchanged test suite was rerun.

   The exact six-file commit, verified Git bundle, audit and receipts have been
   encrypted in the existing private recovery repository at
   ff30b62bedc70f18b6e39a57ad3f9b12ec6d016a; independent restoration verified all
   219 files in that checkpoint. All original data remain intact, including the
   unsuccessful first bundle command and its corrected named-ref invocation.

   Public source remains unpushed, with no new PR or merge. This local commit
   does not resolve the permanent agent-harness#827 publication ambiguity, prove
   provider-child custody, authorize lease removal or maintenance, repair
   agent-harness#833, or establish V10 acceptance. A later landing candidate needs
   its exact review/evidence gates. The issue remains open.
7. ViperJuice

   added a commit that references this issue on Oct 15, 2026

   Merge origin/main into the agent-harness#820 effect fence (agent-harn…

   ...

   b5e725d