Skip to content

fix(worktree): reclaim a prepared checkout whose discard failed in-process - #17899

Merged
nwparker merged 1 commit into
mainfrom
nwparker/worktree-preparation-discard-leak
Sep 1, 2026
Merged

nwparker merged 1 commit into
mainfrom
nwparker/worktree-preparation-discard-leak

Conversation

@nwparker

@nwparker nwparker commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor
Files Added Deleted Net
Test 1 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​209 0 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​209
Prod 2 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​101 $\color{#cf222e}{\Huge{\mathbf{−}}}$​4 $\color{#1a7f37}{\Huge{\mathbf{+}}}$​97

Problem

Pre-existing leak, found while investigating #17792. The speculative-preparation feature itself is unchanged.

src/main/worktree-create-preparation.ts keeps at most WORKTREE_CREATE_PREPARATION_LIMIT = 3 prepared checkouts, evicting the oldest and expiring each after a 5-minute TTL. Both paths funnel into discardEntry, which was:

await discardPreparedWorktree(entry.repoPath, entry.preparedPath, entry.options).catch(() => {})

The failure was swallowed with nothing recorded. The only other reclaim path is cleanupStalePreparations, which skips every preparation still owned by a live process:

if (!lockOwnerPid || isProcessAlive(lockOwnerPid)) {
  continue
}

So a discard that failed inside the running app — a held file handle, a slow AV scanner, a busy directory — stranded both the scratch directory under .orca-preparing/ and its locked Git worktree registration until the app was restarted. Every later composer open walked past it, because the pid in the lock reason was its own and still alive. The 3-entry limit does not bound this: evicted entries leave the in-memory map whether or not their discard succeeded, so leaked directories accumulate across a session.

What this does

New src/main/worktree-preparation-discard-retry.ts records a discard that failed, keyed by host (repo path + WSL distro) and prepared path, and retries it later. A record is dropped as soon as its discard succeeds, and after 3 attempts.

When the retry actually runs. retryPendingPreparationDiscards has exactly one caller: the first statement inside cleanupStalePreparations. That function runs only when prepareWorktreeCreateForRepo starts a fresh preparation — it is skipped entirely by the if (existing) return existing.ready short-circuit. So the trigger is not "the next composer open"; it is the next composer open that starts a new preparation for that host, i.e. one whose repo / workspace root / base branch / distro key is not already covered by a live cached preparation. Re-opening the composer on the same repo and base inside the 5-minute TTL will not fire a retry. In the leak's own scenario this is close to immediate, because the leak is produced by eviction or TTL expiry, both of which free the key.

Where it sits relative to listWorktreeGraph. It is void-called before listWorktreeGraph, and not awaited, so it starts concurrently with the stale-preparation scan and may still be running during the mkdir and worktree add that follow. It is deliberately not sequenced ahead of the scan: claimPreparedWorktree awaits entry.ready, which awaits cleanupStalePreparations, so awaiting the retry would put a worktree unlock + worktree remove --force pair — each bounded by WORKTREE_REMOVAL_REGISTRATION_TIMEOUT_MS (30 s) — directly in front of the user's Create click, in exactly the Windows busy-handle failure mode this change targets, on the one code path whose purpose is latency. The retry never rejects and nothing downstream reads its result.

Two other properties are worth stating, because each was a defect in an earlier draft:

  • A record is claimed, not just deleted on success. discardPreparationWithRetry removes the key from the registry on entry. Two composer opens for the same host can overlap now that the retry is non-blocking, and without the claim both would run the same worktree remove concurrently.
  • Enrolment is unconditional, including for a preparation whose own checkout failed. prepareWorktreeCreateCheckout does self-discard in its failure path, but through performDiscardPreparedWorktree(...).catch(() => {}) — best-effort, exactly as fallible as the discard this change is fixing. A worktree add that succeeded and a reset --hard that then failed on a busy handle leaves a registered, locked worktree whose self-discard also failed; skipping enrolment there would carve the marquee Windows case out of the fix. The cost of enrolling a path that is genuinely already gone is 2 background unlock + remove --force pairs that fail with "is not a working tree" — acceptable precisely because the retry no longer blocks anything.

Cleanup is not made more aggressive. It still refuses to touch a live-pid preparation it discovers via listWorktreeGraph; only paths this process created and already tried to discard are retried. That preserves the fail-safe direction — a preparation directory is speculative scratch, so leaving one is wasted disk, while deleting the wrong thing destroys user work.

The give-up branch is no longer silent: after the third failed attempt it logs [worktree-create] prepared checkout could not be discarded in 3 attempts; <path> stays registered until this process exits.

What changes for users

Platform Delta
macOS No change in the common case: discards succeed, the registry stays empty, and retryPendingPreparationDiscards is a no-op over an empty map. If a discard fails, the stranded preparation is reclaimed when the next fresh preparation for that repo starts, instead of at app restart. No change to the awaited create path.
Linux No change, same as macOS.
Native Windows Same as macOS, and the main beneficiary: a transient handle on a checkout is the realistic way git worktree remove --force fails, and it is also the way prepareWorktreeCreateCheckout's own self-discard fails.
WSL on Windows Same as macOS. Retries are scoped per distro through the host key (repo path + wslDistro) and each record carries its own AddWorktreeOptions, so a failure in one distro never triggers work against another. Pinned by a test; deleting the distro from the key fails it.
SSH No change. prepareWorktreeCreateForRepo returns early when repo.connectionId is set, so no preparation is ever created on an SSH host and no record can exist.
Relay / remote wire No change. Same early return; no RPC params, stream frames, opcodes, or published content are touched, so there is nothing to capability-negotiate.
Folder workspace No change. isFolderRepo(repo) returns early on the same line as the SSH check.
GitHub / GitLab / other providers No change. Nothing provider-facing is touched.

What this does NOT do

  • Does not change the speculative-preparation feature, its 3-entry limit, or its 5-minute TTL.
  • Does not add latency to the awaited create path. The retry is fire-and-forget; deleting the call site fails 5 tests, so the call is pinned, and nothing awaits its result. It does add background work — see Costs.
  • Does not retry on every composer open. See "When the retry actually runs" above; a stranded path can sit unretried while a live cached preparation keeps covering the composer's selection.
  • Does not widen cleanupStalePreparations to reclaim live-pid preparations it finds on disk. That would race destructively with finalizePreparedWorktree: claimPreparedWorktree removes the entry from the live map before worktree move runs, so a concurrent scan would see a locked, live-pid, unmapped preparation mid-create and delete it.
  • Does not enroll the consumePreparedWorktreeCreate fallback discard. finalizePreparedWorktree self-cleans through removeFailedFinalization, so that path usually discards an already-removed worktree.
  • Does not surface anything in the UI. The give-up path writes one main-process console.warn; there is no toast, badge, or notification.
  • Does not check the filesystem before enrolling. An fs probe would need a WSL round trip to be correct on that host, for a case whose cost is 2 background Git pairs.
  • No new Git subcommands or options; the retry reuses the existing discardPreparedWorktree (worktree unlock + worktree remove --force), well inside the Git 2.25 baseline.
  • Does not touch resolveGitMetadataPath or any resolver nullability. discardPreparationWithRetry returns Promise<void> and never rejects, so the entry.ready chain gains no new rejection path.

Costs and residual risk

Things this makes worse, named:

  • Background Git work at preparation-start time. When a record exists, starting a preparation now kicks off one worktree unlock + worktree remove --force pair per stranded path for that host, running concurrently with the stale scan, the mkdir, and the worktree add. discardPreparedWorktree wraps itself in runWithGitReadCacheInvalidation, which clears in-flight Git read caches around the call, so a status read racing that window is re-taken. This is CPU/IO contention, not blocking, and it occurs only in the already-degraded case — but it is work main did not do at all in this scenario.
  • Unconditional enrolment can chase a path that is already gone. When entry.ready rejected before worktree add ever ran (a mkdir or base-ref failure), the recorded path does not exist; its 2 retries fail with "is not a working tree" and cost 2 background Git pairs plus 2 cache-invalidation rounds. Deliberate trade against carving out the busy-handle case where the checkout's own self-discard failed.
  • A fire-and-forget promise outlives its trigger. If the app quits between the preparation start and the retry finishing, the retry is abandoned mid-flight. Same end state as today (the directory persists to the next launch, where the dead-pid branch reclaims it), but the abandonment is now something this change can cause rather than only inherit.
  • Give-up is still abandonment, just a logged one. After 3 failed attempts the record is dropped and the directory plus its locked registration persist — visible in git worktree list — until the process exits. That is today's behavior; this change makes it the floor rather than the default, and adds the diagnostic line that was missing.
  • Unbounded-in-principle registry. pendingDiscards is bounded only by how many discards can fail, at 3 attempts each. In practice the source is capped by the 3-entry preparation limit plus the TTL, and each record is 4 strings; there is no explicit size cap.
  • One new module. worktree-preparation-discard-retry.ts (71 lines) rather than inlining, which also keeps worktree-create-preparation.ts under the 300-line max-lines cap. No max-lines disable was added.

Verification

Diff: 3 files, +310/-4.

  • src/main/worktree-create-preparation.test.ts: 18 passed — 11 pre-existing, 7 new. All 7 are platform-agnostic and run everywhere (the WSL scoping test drives wslDistro through the mocked git options, so it needs no WSL host).
  • With the related suites — worktree-create-preparation-wsl-root, git/worktree-create-preparation-real-git, shared/worktree/create-preparation, git/worktree-mutation-route-invalidation, runtime/orca-runtime-create-base-prefetch — 6 files / 42 tests passed.
  • npx oxfmt --write then npx oxlint clean (exit 0) on all three changed files.

Mutation verification — 10 mutations, each applied alone and reverted, every one killed.

# Mutation Tests killed
1 Delete void retryPendingPreparationDiscards(cleanupKey) 5
2 PREPARATION_DISCARD_ATTEMPT_LIMIT → Number.POSITIVE_INFINITY 1 (stops retrying…)
3 Delete .filter((target) => target.hostKey === hostKey) 2 (retries only the leaked paths…, scopes retries to the WSL distro…)
4 Drop ${options.wslDistro ?? ''} from preparationHostKey 1 (scopes retries to the WSL distro…)
5 Delete pendingDiscards.clear() from the reset 3
6 Delete the in-flight settle loop from the reset 2 (drops recorded discards…, settles an evicted preparation's discard…)
7 Delete the give-up console.warn 1 (stops retrying…)
8 Delete the claim pendingDiscards.delete(pendingKey(target)) 2 (retries a discard that failed…, stops retrying…)
9 Re-narrow discardEntry to skip enrolment when entry.ready rejected 1 (retries a failed checkout whose own self-discard…)
10 trackPreparationDiscard(discardEntry(entry)) → bare void discardEntry(entry) 1 (settles an evicted preparation's discard…)

Every one of the 7 new tests is killed by at least one mutation of the production hunk it claims to cover, so none is vacuous. Notably: stops retrying a preparation that never becomes removable is an exact call-count plus warn-message assertion rather than the bare negative it started as; scopes retries to the WSL distro whose preparation leaked (mutation 4) was added because the distro component of the host key was previously unpinned; and settles an evicted preparation's discard before the reset drops the registry (mutation 10) pins that _resetWorktreeCreatePreparationsForTests waits for a background eviction discard that is still parked on an unfinished checkout, instead of clearing the registry and letting the failure repopulate it into the next test.

With all production code restored: 18/18 pass, working tree clean.

…ocess

Speculative create-preparation evicts entries past the 3-entry limit and the
5-minute TTL, and both paths swallowed a failed `discardPreparedWorktree`.
The only other reclaim path, `cleanupStalePreparations`, skips any preparation
whose lock-reason pid is still alive, so a discard that failed inside the
running app stranded its scratch checkout and its locked worktree registration
until restart.

Record the failed discard keyed by host (repo path + WSL distro) and prepared
path, and retry it the next time a fresh preparation starts for that host.
The retry is kicked off before `listWorktreeGraph` but never awaited, so it
runs alongside the stale scan and the `worktree add` instead of sitting in
front of the user's create. It is capped at 3 attempts and warns when it gives
up. Enrolment is unconditional: `prepareWorktreeCreateCheckout` self-discards
on failure, but only best-effort, so a checkout that failed on a busy handle
can strand the same registration.
@coderabbitai

coderabbitai Bot commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 10 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: 1a69057b-e7ef-40dd-bfe4-08c36aa6e7b1

📥 Commits

Reviewing files that changed from the base of the PR and between c475d23 and 4062a3d.

📒 Files selected for processing (3)
  • src/main/worktree-create-preparation.test.ts
  • src/main/worktree-create-preparation.ts
  • src/main/worktree-preparation-discard-retry.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

  • Retry registry — new src/main/worktree-preparation-discard-retry.ts records a discard that failed in-process, keyed by host (repo path + WSL distro) and prepared path, and retries it on the next fresh preparation for that host, bounded at 3 attempts with a console.warn give-up.
  • Discard wiring — discardEntry now routes through discardPreparationWithRetry (enrolling unconditionally, including a failed checkout whose own self-discard also failed), and eviction/expiry discards are fire-and-forget via trackPreparationDiscard so the test reset can settle them.
  • Retry trigger — retryPendingPreparationDiscards is void-called at the top of cleanupStalePreparations, so reclaim work never adds latency to the awaited create path; the claim (delete-before-discard) prevents overlapping retry passes from double-running a path.
  • Tests — 7 new platform-agnostic tests cover live-pid retry, per-host scoping, WSL distro scoping, the 3-attempt bound + warn, failed-checkout enrollment, and the reset-settle semantics. 18/18 pass; typecheck clean.

The trade-offs the description names (fire-and-forget background Git work, unconditional enrolment chasing already-gone paths, bounded-in-practice registry) are all accurate reads of the code, and the fail-safe direction — only retry paths this process created and already tried to discard, never widening stale cleanup to live-pid preparations — is preserved.

Pullfrog  | View workflow run | Using DeepSeek Pro (free via Pullfrog for OSS) | 𝕏

@nwparker
nwparker merged commit f2db24b into main Sep 1, 2026
29 checks passed
@nwparker
nwparker deleted the nwparker/worktree-preparation-discard-leak branch September 1, 2026 09:39
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…ocess (stablyai#17899)

Speculative create-preparation evicts entries past the 3-entry limit and the
5-minute TTL, and both paths swallowed a failed `discardPreparedWorktree`.
The only other reclaim path, `cleanupStalePreparations`, skips any preparation
whose lock-reason pid is still alive, so a discard that failed inside the
running app stranded its scratch checkout and its locked worktree registration
until restart.

Record the failed discard keyed by host (repo path + WSL distro) and prepared
path, and retry it the next time a fresh preparation starts for that host.
The retry is kicked off before `listWorktreeGraph` but never awaited, so it
runs alongside the stale scan and the `worktree add` instead of sitting in
front of the user's create. It is capped at 3 attempts and warns when it gives
up. Enrolment is unconditional: `prepareWorktreeCreateCheckout` self-discards
on failure, but only best-effort, so a checkout that failed on a busy handle
can strand the same registration.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant