Skip to content

fix(miner): reclaim orphaned worktree allocations on every acquire() - #8918

Merged
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
oktofeesh1:claude/issue-8859-worktree-orphan-reclaim-per-acquire
Jul 26, 2026
Merged

fix(miner): reclaim orphaned worktree allocations on every acquire()#8918
loopover-orb[bot] merged 1 commit into
JSONbored:mainfrom
oktofeesh1:claude/issue-8859-worktree-orphan-reclaim-per-acquire

Conversation

@oktofeesh1

Copy link
Copy Markdown
Contributor

Summary

  • `reclaimOrphanedAllocations` previously ran exactly once, inside `openWorktreeAllocator()`. A long-lived fleet worker that opens its allocator once and calls `.acquire()` for hours never re-swept a peer that crashed mid-lease after open — only a process restart re-triggered reclaim.
  • `acquire()` now calls `reclaimOrphanedAllocations` at its start, on every invocation, matching the sibling `portfolio-queue-manager.ts`'s `claimNextBatch()`'s existing per-claim `sweepStuckItems(...)` sweep pattern.
  • Reworked the allocator's `nowMs` handling into a `getNowMs()` closure (was a captured value) so production callers get a genuinely fresh `Date.now()` on every reclaim call — both at open time and on every `acquire()` — while a test's injected `nowMs` stays frozen across both, preserving the existing age-based-reclaim test suite's determinism.

Fixes #8859.

Test plan

  • Added a regression test simulating a peer's allocation orphaned via a second connection to the same store file after the allocator was already open, asserting the very next `acquire()` call (on the same, already-open allocator instance — no `openWorktreeAllocator()` reopen) reclaims it.
  • `npx tsc -p packages/loopover-miner/tsconfig.json --noEmit` — 0 errors
  • `npx vitest run test/unit/miner-worktree-allocator-lease-expiry.test.ts test/unit/miner-worktree-allocator.test.ts test/unit/miner-worktree-allocator-collisions.test.ts` — all 38 tests passing
  • `npm run test:ci` (full local gate) — green
  • `npm audit --audit-level=moderate` — pre-existing dev-only eslint/minimatch advisory only, unrelated to this change (no lockfile diff)

reclaimOrphanedAllocations previously only ran once inside
openWorktreeAllocator(), so a long-lived fleet worker never re-swept a
peer's crashed/orphaned allocation after open -- only a process
restart re-triggered reclaim. acquire() now calls it at the start of
every invocation, matching portfolio-queue-manager.ts's claimNextBatch
per-claim sweep pattern.

Reworked the allocator's nowMs handling into a getNowMs() closure so
production callers get a genuinely fresh Date.now() on every reclaim
(open-time and every acquire()), while a test's injected nowMs stays
frozen across both -- preserving the existing age-based reclaim
suite's determinism.

Refs JSONbored#8859.
@oktofeesh1
oktofeesh1 requested a review from JSONbored as a code owner July 26, 2026 13:26
@superagent-security

Copy link
Copy Markdown
Contributor

Superagent didn't find any vulnerabilities or security issues in this PR.

@codecov

codecov Bot commented Jul 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.61%. Comparing base (8a3d2c9) to head (30f92a5).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8918      +/-   ##
==========================================
- Coverage   93.84%   90.61%   -3.24%     
==========================================
  Files         803       97     -706     
  Lines       80084    22610   -57474     
  Branches    24277     3917   -20360     
==========================================
- Hits        75157    20487   -54670     
+ Misses       3562     1945    -1617     
+ Partials     1365      178    -1187     
Flag Coverage Δ
backend 100.00% <100.00%> (+4.86%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
packages/loopover-miner/lib/worktree-allocator.ts 100.00% <100.00%> (ø)

... and 706 files with indirect coverage changes

@loopover-orb loopover-orb Bot added the gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier. label Jul 26, 2026
@loopover-orb

loopover-orb Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Tip

✅ LoopOver review result - approve/merge recommended

Review updated: 2026-07-26 13:31:13 UTC

2 files · 1 AI reviewer · no blockers · readiness 98/100 · CI green · clean

✅ Suggested Action - Approve/Merge

  • safe to merge

Review summary
This PR moves the per-acquire orphan reclaim sweep from a one-time open() call into a call at the start of every acquire(), matching the sibling portfolio-queue-manager sweep pattern, and refactors the captured nowMs into a getNowMs() closure so production gets a fresh Date.now() on each call while tests stay deterministic. The logic is correct: the closure evaluates Number.isFinite(options.nowMs) once at setup and returns either a fixed test value or a fresh Date.now() call each invocation, preserving both the frozen-clock test contract and the fresh-clock production behavior. The added regression test faithfully simulates a peer overwriting the active row via a second DB connection after open, and asserts the same long-lived allocator instance self-heals on the next acquire() without a reopen — this is a real, non-fabricated reproduction of the bug described in #8859.

Nits — 4 non-blocking
  • nit: acquire() now runs a full table scan + reclaim pass (SELECT all active rows, per-row UPDATE) on every single call, which is a minor throughput cost for a high-frequency allocator under heavy acquire() churn — not a blocker since worktree acquisitions are inherently low-frequency (one per attempt), but worth a one-line comment noting the tradeoff was accepted.
  • nit: worktree-allocator.ts:387-389's reclaim call re-derives getNowMs() on every acquire() but the surrounding markActive.run() below still uses `new Date().toISOString()` directly instead of getNowMs()-derived time, so a test injecting nowMs won't see that same frozen clock reflected in a freshly allocated slot's allocated_at — worth confirming this asymmetry is intentional.
  • Consider documenting in the reclaimOrphanedAllocations JSDoc that it's now called on every acquire(), not just once at open, since a future reader auditing call frequency will only find the comment inline at the two call sites.
  • The test's peer connection UPDATE (test/unit/miner-worktree-allocator-lease-expiry.test.ts) directly bypasses the store's own release()/acquire() API to simulate crash-orphaning — that's appropriate here since it's specifically testing the same-host dead-pid path, but a short comment cross-referencing why a raw DatabaseSync bypass is necessary (vs. reusing an allocator helper) would help future readers understand it's deliberate.

Decision drivers

  • ✅ Code review — No blockers (1 reviewer)
  • ✅ Gate result — Passing (No configured blocker found.)
Context & advisory signals — never blocks the verdict
Signal Result Evidence
Linked issue ✅ Linked #8859
Related work ✅ No active overlap found No same-issue or scoped active PR overlap found.
Change scope ✅ 20/20 Low review scope from cached public metadata (1 linked issue).
Validation posture ✅ 25/25 PR body includes validation/test evidence.
Contributor workload ✅ 10/10 Author activity: 44 registered-repo PR(s), 36 merged, 0 issue(s).
Contributor context ✅ Confirmed Gittensor contributor oktofeesh1; Gittensor profile; 44 PR(s), 0 issue(s).
Improvement ✅ Minor risk: clean · value: minor · LLM: moderate
Linked issue satisfaction

Addressed
The diff adds reclaimOrphanedAllocations to the start of acquire() using a getNowMs() closure for a fresh timestamp each call, matching the sibling per-claim sweep pattern requested, and includes a test simulating an allocation orphaned after allocator-open that is reclaimed on a subsequent acquire() without reopening.

Review context
  • Author: oktofeesh1
  • Role context: outside_contributor
  • Public audience mode: oss maintainer
  • Lane context: Repository is configured for direct PR review.
  • Public profile languages: JavaScript, MDX, Python, TypeScript
  • Official Gittensor activity: 44 PR(s), 0 issue(s).
  • PR-specific overlap: none found.
Contributor next steps
  • Keep the PR focused and include validation evidence before maintainer review.
Signal definitions
  • Related work = same linked issue, overlapping active PRs, or title/path similarity.
  • Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
  • Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
  • Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
  • Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.
🧪 Chat with LoopOver

Ask LoopOver a question about this PR directly in a comment — grounded only in the same cached, public-safe facts shown above, never a new claim.

  • @loopover ask &lt;question&gt; answers contribution-quality Q&A with source citations and freshness.
  • @loopover chat &lt;question&gt; answers in natural prose from cached decision-pack facts via local inference (maintainer/collaborator; read-only).
  • A plain-language @loopover mention with a real question is routed to the closest matching read-only command automatically — no exact syntax required.

Full command reference: https://loopover.ai/docs/loopover-commands

🧪 Experimental — new and may change.

🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed


💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by LoopOver, a quiet PR intelligence layer for OSS maintainers.

  • Re-run LoopOver review

@loopover-orb loopover-orb Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LoopOver approves — the gate is satisfied and CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gittensor:feature Gittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ams: worktree orphan reclaim only runs once at allocator-open time, not per-acquire

1 participant