Repository navigation
fix(remote): stop a colliding path key, a dead conflict state, and a live-PTY removal from losing tabs #17948
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
154 changes: 154 additions & 0 deletions
154
src/renderer/src/hooks/remote-workspace-deferred-placement-retry.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,154 @@ | ||
| import { afterEach, describe, expect, it, vi } from 'vitest' | ||
| import type { RemoteWorkspaceObservedSnapshot } from '../../../shared/remote-workspace-types' | ||
| import { createDeferredSnapshotPlacementRetries } from './remote-workspace-deferred-placement-retry' | ||
| import type { RemoteWorkspaceSnapshotPlacementStore } from './remote-workspace-snapshot-placement' | ||
| import { | ||
| appState, | ||
| flush, | ||
| owner, | ||
| snapshot | ||
| } from './__tests__/remote-workspace-target-sync-test-harness' | ||
|
|
||
| /** `appState()` carries worktree `repo-a::/remote/work`, so this path is placeable at once. */ | ||
| const PLACEABLE_PATH = '/remote/work' | ||
|
|
||
| function placementStore(): RemoteWorkspaceSnapshotPlacementStore { | ||
| const state = appState() | ||
| return { getState: () => state, subscribe: () => () => {} } | ||
| } | ||
|
|
||
| const rejections: unknown[] = [] | ||
| const onUnhandled = (reason: unknown): void => { | ||
| rejections.push(reason) | ||
| } | ||
| process.on('unhandledRejection', onUnhandled) | ||
| afterEach(() => { | ||
| rejections.length = 0 | ||
| }) | ||
|
|
||
| describe('createDeferredSnapshotPlacementRetries', () => { | ||
| it('swallows a getSnapshot rejection instead of leaving it unhandled', async () => { | ||
| const applySnapshot = vi.fn(async () => {}) | ||
| const retries = createDeferredSnapshotPlacementRetries({ | ||
| store: placementStore(), | ||
| getCurrentAuthority: () => owner, | ||
| getSnapshot: () => Promise.reject(new Error('relay dropped')), | ||
| applySnapshot | ||
| }) | ||
|
|
||
| retries.watch(owner, [PLACEABLE_PATH]) | ||
| await flush() | ||
| await new Promise((resolve) => setTimeout(resolve, 0)) | ||
|
|
||
| expect(rejections).toEqual([]) | ||
| // A failed pull is unverifiable, so the target is left on `conflict` rather than re-applied. | ||
| expect(applySnapshot).not.toHaveBeenCalled() | ||
| retries.stop() | ||
| }) | ||
|
|
||
| it('swallows an applySnapshot rejection instead of leaving it unhandled', async () => { | ||
| const retries = createDeferredSnapshotPlacementRetries({ | ||
| store: placementStore(), | ||
| getCurrentAuthority: () => owner, | ||
| getSnapshot: async () => snapshot(4), | ||
| applySnapshot: () => Promise.reject(new Error('relay dropped')) | ||
| }) | ||
|
|
||
| retries.watch(owner, [PLACEABLE_PATH]) | ||
| await flush() | ||
| await new Promise((resolve) => setTimeout(resolve, 0)) | ||
|
|
||
| expect(rejections).toEqual([]) | ||
| retries.stop() | ||
| }) | ||
|
|
||
| it('bounds an apply that keeps re-arming the same unplaced paths', async () => { | ||
| // Guards the test itself: without a bound the chain never yields and the run would hang. | ||
| const CHAIN_GUARD = 50 | ||
| let pulls = 0 | ||
| const retries = createDeferredSnapshotPlacementRetries({ | ||
| store: placementStore(), | ||
| getCurrentAuthority: () => owner, | ||
| getSnapshot: async (): Promise<RemoteWorkspaceObservedSnapshot> => { | ||
| pulls += 1 | ||
| return snapshot(4 + pulls) | ||
| }, | ||
| applySnapshot: async () => { | ||
| // The apply reports the same still-unplaced path, which arms the next watch. | ||
| if (pulls < CHAIN_GUARD) { | ||
| retries.watch(owner, [PLACEABLE_PATH]) | ||
| } | ||
| } | ||
| }) | ||
|
|
||
| retries.watch(owner, [PLACEABLE_PATH]) | ||
| for (let tick = 0; tick < CHAIN_GUARD * 2; tick += 1) { | ||
| await flush() | ||
| } | ||
|
|
||
| expect(pulls).toBeLessThan(CHAIN_GUARD) | ||
| expect(pulls).toBe(3) | ||
| retries.stop() | ||
| }) | ||
|
|
||
| it('lets a chain that keeps placing rows run past the stalled bound', async () => { | ||
| // Five placeable paths, one fewer unplaced each round: that chain is converging and is already | ||
| // bounded by the set emptying, so the stall bound must not cut it off at 3. | ||
| const state = appState({ | ||
| worktreesByRepo: { | ||
| 'repo-a': [1, 2, 3, 4, 5].map((n) => ({ | ||
| id: `repo-a::/remote/w${n}`, | ||
| repoId: 'repo-a', | ||
| hostId: 'ssh:target-a' | ||
| })) | ||
| } | ||
| }) | ||
| const allPaths = [1, 2, 3, 4, 5].map((n) => `/remote/w${n}`) | ||
| let pulls = 0 | ||
| let remaining = allPaths.length | ||
| const retries = createDeferredSnapshotPlacementRetries({ | ||
| store: { getState: () => state, subscribe: () => () => {} }, | ||
| getCurrentAuthority: () => owner, | ||
| getSnapshot: async (): Promise<RemoteWorkspaceObservedSnapshot> => { | ||
| pulls += 1 | ||
| return snapshot(4 + pulls) | ||
| }, | ||
| applySnapshot: async () => { | ||
| remaining -= 1 | ||
| retries.watch(owner, allPaths.slice(0, remaining)) | ||
| } | ||
| }) | ||
|
|
||
| retries.watch(owner, allPaths) | ||
| for (let tick = 0; tick < 40; tick += 1) { | ||
| await flush() | ||
| } | ||
|
|
||
| // One pull per round until the unplaced set empties -- five, not the stall bound of three. | ||
| expect(pulls).toBe(5) | ||
| retries.stop() | ||
| }) | ||
|
|
||
| it('does not spend the chain budget on watches armed outside a retry', async () => { | ||
| let pulls = 0 | ||
| const retries = createDeferredSnapshotPlacementRetries({ | ||
| store: placementStore(), | ||
| getCurrentAuthority: () => owner, | ||
| getSnapshot: async (): Promise<RemoteWorkspaceObservedSnapshot> => { | ||
| pulls += 1 | ||
| return snapshot(4 + pulls) | ||
| }, | ||
| applySnapshot: async () => {} | ||
| }) | ||
|
|
||
| // Each of these is a fresh host snapshot arrival, not a continued chain. | ||
| for (let arrival = 0; arrival < 6; arrival += 1) { | ||
| retries.watch(owner, [PLACEABLE_PATH]) | ||
| await flush() | ||
| await flush() | ||
| } | ||
|
|
||
| expect(pulls).toBe(6) | ||
| retries.stop() | ||
| }) | ||
| }) |
129 changes: 129 additions & 0 deletions
129
src/renderer/src/hooks/remote-workspace-deferred-placement-retry.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| import type { DirectSshAuthority } from '../../../shared/ssh-types' | ||
| import type { RemoteWorkspaceObservedSnapshot } from '../../../shared/remote-workspace-types' | ||
| import { directSshAuthoritiesEqual } from './direct-ssh-reconnect-tokens' | ||
| import { | ||
| waitForSnapshotWorktreePlacement, | ||
| type RemoteWorkspaceSnapshotPlacementStore | ||
| } from './remote-workspace-snapshot-placement' | ||
|
|
||
| /** How long a conflicted target keeps watching the catalog for the rows it could not place. The | ||
| * catalog for a remote host often only fills in when the user opens the worktree, which is minutes | ||
| * after connect, and `conflict` has no other exit: it suppresses uploads and holds terminal | ||
| * authority at `unverifiable` until a reconnect or an unsolicited host push happens to arrive. */ | ||
| const DEFERRED_SNAPSHOT_PLACEMENT_TIMEOUT_MS = 600_000 | ||
|
|
||
| /** A retry's own apply can report paths it still could not place, which arms the next watch. If the | ||
| * catalog reports those paths placeable, that watch fires at once and the chain never yields. | ||
| * | ||
| * Counted only while the unplaced set stops shrinking: a chain that keeps placing rows is | ||
| * converging and is already bounded by that set emptying, so cutting it short would strand the | ||
| * target on `conflict` for no reason. A chain that re-arms on the same or a larger set is the | ||
| * spinning case, and that is what this bounds. */ | ||
| const MAX_STALLED_DEFERRED_PLACEMENT_CHAIN = 3 | ||
|
|
||
| export type DeferredSnapshotPlacementRetryDeps = { | ||
| store: RemoteWorkspaceSnapshotPlacementStore | ||
| getCurrentAuthority: (targetId: string) => DirectSshAuthority | null | ||
| getSnapshot: (targetId: string) => Promise<RemoteWorkspaceObservedSnapshot | null> | ||
| applySnapshot: (targetId: string, snapshot: RemoteWorkspaceObservedSnapshot) => Promise<void> | ||
| } | ||
|
|
||
| export type DeferredSnapshotPlacementRetries = { | ||
| /** Empty `worktreePaths` retires the target's outstanding watch instead of arming one. */ | ||
| watch: (authority: DirectSshAuthority, worktreePaths: readonly string[]) => void | ||
| stop: () => void | ||
| } | ||
|
|
||
| /** | ||
| * Re-pull once the local catalog can place the host paths an apply had to drop. | ||
| * | ||
| * Why a fresh pull rather than replaying the snapshot already held: that snapshot is only evidence | ||
| * of what the host had when it was taken. The host owns execution state, so the answer acted on has | ||
| * to be its current one — see docs/reference/ssh-execution-boundary.md. A pull that comes back | ||
| * empty or fails is left alone: it is `unverifiable`, and moving the target off `conflict` on it | ||
| * would re-authorise uploads and sleeping-agent resume from a picture known to be incomplete. | ||
| */ | ||
| export function createDeferredSnapshotPlacementRetries( | ||
| deps: DeferredSnapshotPlacementRetryDeps | ||
| ): DeferredSnapshotPlacementRetries { | ||
| const watchers = new Map<string, AbortController>() | ||
| /** Consecutive non-shrinking re-arms, and the set size they stalled at. Absent means no chain. */ | ||
| const chains = new Map<string, { stalledDepth: number; unplacedCount: number }>() | ||
| const applying = new Set<string>() | ||
| let stopped = false | ||
|
|
||
| const watch = (authority: DirectSshAuthority, worktreePaths: readonly string[]): void => { | ||
| // Always retire the previous watch: an apply that placed everything passes no paths, and that | ||
| // is exactly when the outstanding watch is obsolete. | ||
| watchers.get(authority.targetId)?.abort() | ||
| watchers.delete(authority.targetId) | ||
| // A watch armed from outside a retry is a fresh snapshot arrival, not a continued chain. | ||
| const previous = applying.has(authority.targetId) ? chains.get(authority.targetId) : undefined | ||
| if (stopped || worktreePaths.length === 0) { | ||
| chains.delete(authority.targetId) | ||
| return | ||
| } | ||
| const stalledDepth = | ||
| previous && worktreePaths.length >= previous.unplacedCount ? previous.stalledDepth + 1 : 0 | ||
| if (stalledDepth >= MAX_STALLED_DEFERRED_PLACEMENT_CHAIN) { | ||
| // Leave the target on `conflict`: the paths are not becoming placeable by re-pulling. | ||
| chains.delete(authority.targetId) | ||
| return | ||
| } | ||
| chains.set(authority.targetId, { stalledDepth, unplacedCount: worktreePaths.length }) | ||
| const controller = new AbortController() | ||
| watchers.set(authority.targetId, controller) | ||
| const isCurrent = (): boolean => | ||
| !stopped && | ||
| watchers.get(authority.targetId) === controller && | ||
| directSshAuthoritiesEqual(deps.getCurrentAuthority(authority.targetId), authority) | ||
| void (async () => { | ||
|
pullfrog[bot] marked this conversation as resolved.
Resolved
|
||
| try { | ||
| const placed = await waitForSnapshotWorktreePlacement( | ||
| deps.store, | ||
| authority, | ||
| worktreePaths, | ||
| isCurrent, | ||
| controller.signal, | ||
| DEFERRED_SNAPSHOT_PLACEMENT_TIMEOUT_MS | ||
| ) | ||
| if (!placed || !isCurrent()) { | ||
| return | ||
| } | ||
| const snapshot = await deps.getSnapshot(authority.targetId) | ||
| if (!snapshot || snapshot.revision <= 0 || !isCurrent()) { | ||
| return | ||
| } | ||
| applying.add(authority.targetId) | ||
| try { | ||
| await deps.applySnapshot(authority.targetId, snapshot) | ||
| } finally { | ||
| applying.delete(authority.targetId) | ||
| } | ||
| } catch { | ||
| // `getSnapshot` is an IPC call that rejects when the relay drops, and the apply can reject | ||
| // with it. A failed pull is `unverifiable`, and the documented behaviour is to leave the | ||
| // target on `conflict` rather than act on a picture known to be incomplete -- so swallow it | ||
| // here instead of surfacing an unhandled rejection. | ||
| } finally { | ||
| // A still-current controller means the apply did not re-arm, so the chain ends here. | ||
| if (watchers.get(authority.targetId) === controller) { | ||
| watchers.delete(authority.targetId) | ||
| chains.delete(authority.targetId) | ||
| } | ||
| } | ||
| })() | ||
| } | ||
|
|
||
| return { | ||
| watch, | ||
| stop: () => { | ||
| stopped = true | ||
| for (const controller of watchers.values()) { | ||
| controller.abort() | ||
| } | ||
| watchers.clear() | ||
| chains.clear() | ||
| } | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Keep the retry budget separate from fresh snapshot arrivals.
applying.has(authority.targetId)only shows that an apply is pending. It does not show that thiswatchcall came from that apply. If a fresh host snapshot arrives while an earlierapplySnapshotis pending for the same target, this branch reuses the oldstalledDepth. At depth two with the same path count, line 68 discards the fresh watch. The target then stays inconflictwithout a recovery attempt.Split fresh
watchcalls from internal retry re-arms, or pass explicit retry provenance. Add a test for a fresh arrival that overlaps a stalled apply.