Repository navigation
fix(terminal): add replacement policies for repeated same-handle stream - #12003
Conversation
Recovery logic strengthened with replacement-policy tiers (reuse/prefer-replacement/require-replacement) and bounded same-handle end cycles. Prevents infinite flapping by capping reuse attempts and inventory wait windows. Tracks ready evidence to reattach from prior snapshots when inventory becomes unavailable.
a0d217b to
012228c
Compare
📝 WalkthroughWalkthroughRemote PTY recovery now uses graded replacement policies and bounded host-session inventory polling. The transport can reuse a same-handle subscription, prefer a replacement, or require a replacement after repeated stream endings. Recovery preserves prior ready evidence when inventory becomes temporarily unavailable. Input errors affect the terminal only when the failed handle remains current. Lifecycle operations reset recovery state. Tests cover inventory failures, handle replacement, stale input, repeated stream endings, condemned handles, and recovery bounds. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (5)
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts (2)
1450-1453: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueExtract the effective-policy expression into a helper.
The expression
stricterReplacementPolicy(replacementPolicy, getRecoveryReplacementPolicy(previousHandle))appears at Lines 627, 674, 1430, 1450, and 1472. Each site must re-read the current policy after anawait, so caching is not safe, but a helper keeps the freshness and removes the duplication.♻️ Proposed helper
+ const effectiveReplacementPolicyFor = ( + previousHandle: string, + replacementPolicy: HostHandleReplacementPolicy + ): HostHandleReplacementPolicy => + stricterReplacementPolicy(replacementPolicy, getRecoveryReplacementPolicy(previousHandle))
1509-1513: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueConsider the effective policy in the early return.
Line 1509 strengthens the sticky policy for the current handle. Line 1510 then tests the
replacementPolicyargument only. If a required replacement is already sticky for this handle and a later caller passes'reuse', the early return does not apply and the transport re-arms inventory polling whilestopWaitingForPublishedHandleis active. Reading the effective policy matches the stated intent.♻️ Proposed change
strengthenRecoveryReplacementPolicy(handle, replacementPolicy) - if (replacementPolicy === 'require-replacement' && stopWaitingForPublishedHandle) { + if (getRecoveryReplacementPolicy(handle) === 'require-replacement' && stopWaitingForPublishedHandle) { // Why: once recovery is handed to accepted snapshots, repeated sends to the stale handle must not re-arm inventory RPCs. return }src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts (3)
2054-2062: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDispose the transport at the end of the test.
The neighboring tests call
transport.destroy?.()before restoring real timers. This test and the test at Line 2221 leave the transport active, which keeps its recovery timers and handle subscription registered after the test finishes. Add the same cleanup for symmetry.♻️ Proposed change
emitSnapshot(latestSubscribePayload().streamId, 'same handle reattached') expect(transport.isConnected()).toBe(true) + transport.destroy?.() } finally { vi.useRealTimers() }
2099-2100: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuePrefer invariant assertions over magic poll-count windows.
The bounds
> 25and< 40encode the current backoff schedule and the attach timeout. Similar magic windows appear at Lines 2046, 2050, and 2154. IfHOST_SESSION_ATTACH_POLL_MSor the replacement poll ceiling changes, these tests fail with no indication of the real cause. The invariant that matters is already asserted at Line 2102: no further inventory calls after the cap. Consider deriving the bounds from the constants or dropping the upper magic numbers.
2734-2747: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse the
subscribedTerminalHandleshelper.This block re-implements the helper added at Line 66. Call the helper instead.
♻️ Proposed change
- const subscribedTerminals = subscriptionSendBinary.mock.calls - .map((call) => decodeTerminalStreamFrame(call[0])) - .flatMap((frame) => { - if (frame?.opcode !== TerminalStreamOpcode.Subscribe) { - return [] - } - const payload = decodeTerminalStreamJson<{ terminal: string }>(frame.payload) - return payload ? [payload.terminal] : [] - }) - expect(subscribedTerminals).toEqual([ + expect(subscribedTerminalHandles()).toEqual([ 'terminal-stale', 'terminal-replacement', 'terminal-replacement' ])
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 82bdc872-274a-40b3-b6d1-02668c05abb0
📥 Commits
Reviewing files that changed from the base of the PR and between b04c695 and a0d217b33a717e294159a56eee3a2fb74765b1ad.
📒 Files selected for processing (2)
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.tssrc/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
There was a problem hiding this comment.
🧹 Nitpick comments (4)
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts (4)
2114-2164: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDistinguish this test from the near-duplicate at Line 2609.
The test at Line 2609 uses almost the same title ("reuses prior ready evidence when the trailing inventory poll fails") and asserts the same outcome. Only the failure trigger differs: this test fails the second and later inventory calls by call index, while the other test fails only the short-timeout poll. Rename both titles to state the trigger, or merge them into one parameterized test. Distinct titles keep failure output unambiguous.
♻️ Suggested rename
- it('reattaches from prior ready evidence when the trailing inventory poll fails', async () => { + it('reattaches from prior ready evidence when inventory fails after the first poll', async () => {
2527-2541: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert an observable effect of the strengthened policy.
The test name states that a sticky replacement policy is applied. The assertions only check that
hostListCallsdoes not grow during a 1 second advance and that the handle set is unchanged. That result also holds when the backoff delay is simply longer than 1 second, so the test can pass without the policy being strengthened.Advance the timers until the next inventory poll runs, then assert that the transport rejects the same
terminal-oldhandle and does not resubscribe it. That assertion fails if the policy reverts toreuse.
2734-2742: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReuse the new
subscribedTerminalHandles()helper. This PR addssubscribedTerminalHandles()at Line 66, but two sites still decode Subscribe frames inline.
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts#L2734-L2742: replace the inline decode andflatMapblock withconst subscribedTerminals = subscribedTerminalHandles().src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts#L2040-L2043: remove the localsubscribeCount()helper and usesubscribedTerminalHandles().length.
2547-2607: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMerge the two repeated same-handle flap tests. Both tests drive two same-handle end/resubscribe cycles, then a third end, and assert three
terminal-flappingsubscriptions followed by thedisconnectedphase. Only the runtime environment id and a few extra assertions differ.
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts#L2547-L2607: keep this test as the single scenario and add theisConnected()assertion from the other test.src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts#L2166-L2219: remove this test, or narrow it to an assertion that the other test does not cover.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: fca02e6d-06d0-4092-87bf-c25933e5ef05
📥 Commits
Reviewing files that changed from the base of the PR and between a0d217b33a717e294159a56eee3a2fb74765b1ad and 012228c.
📒 Files selected for processing (2)
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.tssrc/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
Greptile SummaryThis PR refactors the remote PTY transport's reconnection strategy from a binary
Confidence Score: 4/5The change is safe to merge. The core policy-tier refactor is correct and well-tested; the two findings are edge cases that do not cause incorrect terminal behavior in normal flapping scenarios. The policy enum refactor, stale-handle guards, and same-handle reuse counter are all logically sound and covered by nine new tests. Two issues remain: the inventory window counter resets on every new recovery epoch so HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY=2 never enforces a true 2-window cap, and resetSameHandleEndReuse() on transport close allows the same-handle cycle counter to be cleared by network drops, which could delay when require-replacement kicks in for a flapping session. Files Needing Attention: src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts — specifically the beginResubscribeInventoryWindow epoch-reset logic and the onTransportClose call to resetSameHandleEndReuse.
|
| Filename | Overview |
|---|---|
| src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts | Adds three-tier replacement-policy enum, same-handle end-reuse counter, and inventory-window retry cap. The policy tier refactor and stale-handle error guards are correct; two issues: the inventory window counter always resets per epoch so HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY=2 is functionally equivalent to unlimited until recovery timeout, and resetSameHandleEndReuse() on transport close allows bypassing the per-cycle reuse cap. |
| src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts | Adds 9 new integration-style tests covering inventory retry, same-handle reuse cycles, prior-evidence reattach, stale-send race conditions, and replacement policy propagation. Tests are thorough and correctly validate the intended behaviors, though the two recovery windows test name is misleading as the actual stop is driven by the recovery engine timeout, not the window counter. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Stream onEnd fired] --> B[replacementPolicyAfterWebStreamEnd]
B --> C{Same handle? Within timeout? Count < LIMIT?}
C -- First 2 cycles prefer-replacement --> D[scheduleResubscribeAfterTransportClose]
C -- After 2 cycles require-replacement --> D
D --> E[waitForResubscribeHostSessionHandle poll session.tabs.list]
E --> F{Inventory result}
F -- Different handle found --> G[rebindRemoteTerminalHandle]
F -- Same handle and prefer-replacement wait for timeout --> H[finishBoundedWait return lastReadyHandle]
F -- Same handle and require-replacement --> I[Return undefined do not subscribe]
F -- Inventory fails inventoryWindow less than 2 --> J[throw recoverable error recovery.schedule retry]
G --> K[subscribeToHandle sameHandleEndRecovery=false]
H --> L[subscribeToHandle sameHandleEndRecovery=true]
L --> M[onSubscribed recordSameHandleEndReuse increment count]
I --> N[stopWaitingForPublishedHandle listener waits for snapshot]
J --> O{Recovery timeout exhausted?}
O -- No --> D
O -- Yes --> P[phase: disconnected]
K --> Q[connected]
M --> Q
Comments Outside Diff (1)
-
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts, line 1719-1746 (link)onTransportClosereset allows bypassing the same-handle end-cycle capresetSameHandleEndReuse()is called insideonTransportClose, which runs on network-level transport drops. If the pattern isonEndto reuse (count = 1) toonTransportCloseto counter reset toonEndto reuse (count = 1) the counter never accumulates past 1, sorequire-replacementis never reached via thesameHandleEndReuseCount >= HOST_SESSION_SAME_HANDLE_END_REUSE_LIMITpath.A session that alternates clean stream-ends with transport closes (e.g. due to a flapping connection alongside a rapidly-cycling PTY process) would never be capped, defeating the protection this PR intends to provide. Consider whether the transport-close reset should be gated on the transport being non-recoverable (i.e. only reset on
!recoverable).
Reviews (1): Last reviewed commit: "fix(terminal): add replacement policies ..." | Re-trigger Greptile
| const beginResubscribeInventoryWindow = (recoveryEpoch: number): number => { | ||
| if (resubscribeInventoryEpoch !== recoveryEpoch) { | ||
| resubscribeInventoryEpoch = recoveryEpoch | ||
| resubscribeInventoryWindows = 0 | ||
| } | ||
| resubscribeInventoryWindows += 1 | ||
| return resubscribeInventoryWindows | ||
| } |
There was a problem hiding this comment.
HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY cap never takes effect
resubscribeInventoryWindows is reset to 0 every time resubscribeInventoryEpoch !== recoveryEpoch, which happens on every retry because each retry receives a new nextEpoch from recovery.schedule. As a result inventoryWindow is always 1 at the check site, so 1 < HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY (2) is always true.
In practice this means: inventory failures always throw the recoverable error and keep retrying until the recovery system's own timeout exhausts — never because the window cap was reached. The constant value of 2 is indistinguishable from 100; only the value 1 would change behavior (preventing any inventory retry).
The "caps unavailable host inventory at two recovery windows" test passes because the recovery engine's own scheduling limit stops things at ~2 x 15 s, not because of this constant. The constant as written is dead code for its named purpose.
…am (stablyai#12003) Recovery logic strengthened with replacement-policy tiers (reuse/prefer-replacement/require-replacement) and bounded same-handle end cycles. Prevents infinite flapping by capping reuse attempts and inventory wait windows. Tracks ready evidence to reattach from prior snapshots when inventory becomes unavailable.
Summary
reuse/prefer-replacement/require-replacement) and bounded same-handle end cycles.Test plan