Skip to content

worktree-allocator.js's isProcessAlive PID check is meaningless across fleet-mode's separate containers #7085

Description

@JSONbored

Context

packages/loopover-miner/lib/worktree-allocator.js is a "fleet-mode" store per its own header comment:
"durable local bookkeeping for which worktree paths are allocated to which fleet attempts." Per
packages/loopover-miner/DEPLOYMENT.md, fleet mode runs multiple separate CONTAINERS, each its own OS process
with its own independent PID namespace by default, sharing one bind-mounted data volume.

reclaimOrphanedAllocations (worktree-allocator.js:126-139), run on every openWorktreeAllocator() call
(i.e. at the startup of ANY miner process that opens this shared store), decides whether an active slot is
orphaned by calling isProcessAlive(row.owner_pid) (lines 74-85), which resolves via process.kill(pid, 0)
a check against the CALLING process's OWN PID namespace. owner_pid itself is stored as a bare
process.pid (line 148, 204, 211) with no host/container qualifier recorded alongside it.

This is unsound the moment two different containers share the store, which fleet mode's shared-data-volume
design makes a real, documented scenario: a PID recorded by container A has no relationship to the PID space
container B observes when it later opens the same store and runs its own isProcessAlive check. Two concrete
failure modes follow directly:

  • A truly-dead worker's PID from container A can coincidentally match a PID that IS alive in container B's own
    namespace (e.g. container B's own init process at PID 1) — isProcessAlive wrongly reports "alive", so a
    genuinely orphaned slot is never reclaimed and permanently wastes a worktree slot.
  • A live worker's PID in container A does not exist in container B's namespace — isProcessAlive wrongly
    reports "dead" the moment container B starts up (or any other container opens the shared store), so
    reclaimOrphanedAllocations steals the slot out from under a STILL-RUNNING attempt in container A, freeing
    (and making reassignable) a worktree directory that attempt is actively writing to.

This is also a convention gap relative to this package's own established pattern: every sibling shared-lease
store uses AGE-based expiry, not PID-liveness, for exactly this reason — claim-ledger.js's
sweepExpiredClaims/DEFAULT_MAX_CLAIM_AGE_MS and portfolio-queue-expiry.js's sweepStuckItems/
DEFAULT_MAX_LEASE_MS both reclaim a lease once it is simply too OLD, never by checking whether a recorded PID
is alive — a scheme that works correctly regardless of which container observes it. worktree-allocator.js is
the one store in the package's local-store family that instead uses a PID-liveness check, and it is the wrong
check for the exact multi-container deployment this file's own header comment says it exists to serve.

Requirements

  • worktree-allocator.js's orphan-reclaim logic MUST NOT rely solely on cross-process isProcessAlive(pid)
    liveness for deciding whether an active slot is orphaned.
  • Add an age-based fallback/replacement matching the package's own established convention (mirror
    portfolio-queue-expiry.js's DEFAULT_MAX_LEASE_MS/sweepStuckItems shape): an active slot whose
    allocated_at timestamp is older than a configurable max-lease threshold MUST be reclaimed regardless of
    what isProcessAlive reports for its owner_pid.
  • The existing same-container isProcessAlive check MAY be retained as an additional, faster same-host signal
    (a slot can still be reclaimed immediately if its owner PID is confirmed dead within the SAME PID namespace),
    but it MUST NOT be the only signal a cross-container caller relies on — the age-based check MUST independently
    guarantee eventual reclaim even when isProcessAlive gives a wrong answer due to PID-namespace mismatch.
  • reclaimOrphanedAllocations MUST accept an injectable nowMs (mirroring sweepStuckItems's own signature)
    and a configurable max-lease duration (mirroring worktree-allocator.js's own maxConcurrency option
    pattern), defaulting to a value at least as generous as portfolio-queue-expiry.js's DEFAULT_MAX_LEASE_MS
    so a legitimately long-running attempt is never prematurely reclaimed.

Deliverables

  • reclaimOrphanedAllocations in packages/loopover-miner/lib/worktree-allocator.js reclaims a stale
    active slot by age (allocated_at older than a max-lease threshold), independent of isProcessAlive's
    PID-namespace-dependent verdict.
  • A test simulating the cross-container false-negative case: an active slot whose recorded owner_pid
    does not exist in the calling process's namespace (i.e. isProcessAlive returns false) but whose
    allocated_at is recent, asserting it is NOT reclaimed prematurely once the age-based guard is added.
  • A test simulating the cross-container false-positive case: an active slot whose recorded owner_pid
    happens to collide with a live PID in the calling process's namespace, but whose allocated_at is older
    than the max-lease threshold, asserting it IS reclaimed by the new age-based path.

Test Coverage Requirements

99%+ Codecov patch coverage on every changed line and branch in worktree-allocator.js, including both new
age-threshold branches (still-within-lease vs. past-lease) crossed with both isProcessAlive outcomes, plus the
two regression tests above.

Expected Outcome

worktree-allocator.js's orphan reclaim converges on the same age-based, container-topology-agnostic
convention every sibling shared-lease store (claim-ledger.js, portfolio-queue-expiry.js) already uses, so
fleet mode no longer risks either permanently stranding a dead worker's worktree slot or prematurely stealing a
live worker's slot out from under it based on a PID-namespace collision that was never a meaningful signal
across separate containers.

Links & Resources

  • packages/loopover-miner/lib/worktree-allocator.js:74-85 (isProcessAlive), :126-139
    (reclaimOrphanedAllocations) — the code to change.
  • packages/loopover-miner/lib/portfolio-queue-expiry.js (DEFAULT_MAX_LEASE_MS/sweepStuckItems) — the
    age-based convention to mirror.
  • packages/loopover-miner/lib/claim-ledger.js (sweepExpiredClaims/DEFAULT_MAX_CLAIM_AGE_MS) — the same
    age-based convention, a second sibling precedent.
  • packages/loopover-miner/DEPLOYMENT.md — fleet mode's shared-data-volume, multi-container deployment shape.

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions