Skip to content

fix(terminal): add replacement policies for repeated same-handle stream - #12003

Merged
AmethystLiang merged 1 commit into
mainfrom
getting-reconnecting-to-remote-runtime
Aug 1, 2026
Merged

AmethystLiang merged 1 commit into
mainfrom
getting-reconnecting-to-remote-runtime

Conversation

@AmethystLiang

Copy link
Copy Markdown
Contributor

Summary

  • Strengthen remote runtime PTY recovery with replacement-policy tiers (reuse / prefer-replacement / require-replacement) and bounded same-handle end cycles.
  • Cap reuse attempts and inventory wait windows to prevent infinite reconnect flapping.
  • Track ready evidence so sessions can reattach from prior snapshots when inventory becomes unavailable.

Test plan

  • Unit tests for remote-runtime PTY transport replacement policies and same-handle recovery paths
  • Manually verify remote runtime terminal reconnect after host session end with same handle
  • Confirm no reconnect flapping when inventory repeatedly fails

Recovery logic strengthened with replacement-policy tiers (reuse/prefer-replacement/require-replacement) and bounded same-handle end cycles. Prevents infinite flapping by capping reuse attempts and inventory wait windows. Tracks ready evidence to reattach from prior snapshots when inventory becomes unavailable.
@AmethystLiang
AmethystLiang force-pushed the getting-reconnecting-to-remote-runtime branch from a0d217b to 012228c Compare August 1, 2026 20:36
@coderabbitai

coderabbitai Bot commented Aug 1, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Remote PTY recovery now uses graded replacement policies and bounded host-session inventory polling. The transport can reuse a same-handle subscription, prefer a replacement, or require a replacement after repeated stream endings. Recovery preserves prior ready evidence when inventory becomes temporarily unavailable. Input errors affect the terminal only when the failed handle remains current. Lifecycle operations reset recovery state. Tests cover inventory failures, handle replacement, stale input, repeated stream endings, condemned handles, and recovery bounds.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description includes a summary and unit-test plan, but it omits the required Screenshots, AI Review Report, Security Audit, and Notes sections. Add all required template sections, complete the testing checklist, and document visual impact, AI review coverage, security risks, and platform-specific notes.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the main change: replacement policies for repeated same-handle terminal stream recovery.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (5)
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts (2)

1450-1453: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Extract the effective-policy expression into a helper.

The expression stricterReplacementPolicy(replacementPolicy, getRecoveryReplacementPolicy(previousHandle)) appears at Lines 627, 674, 1430, 1450, and 1472. Each site must re-read the current policy after an await, so caching is not safe, but a helper keeps the freshness and removes the duplication.

♻️ Proposed helper
+  const effectiveReplacementPolicyFor = (
+    previousHandle: string,
+    replacementPolicy: HostHandleReplacementPolicy
+  ): HostHandleReplacementPolicy =>
+    stricterReplacementPolicy(replacementPolicy, getRecoveryReplacementPolicy(previousHandle))

1509-1513: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Consider the effective policy in the early return.

Line 1509 strengthens the sticky policy for the current handle. Line 1510 then tests the replacementPolicy argument only. If a required replacement is already sticky for this handle and a later caller passes 'reuse', the early return does not apply and the transport re-arms inventory polling while stopWaitingForPublishedHandle is active. Reading the effective policy matches the stated intent.

♻️ Proposed change
     strengthenRecoveryReplacementPolicy(handle, replacementPolicy)
-    if (replacementPolicy === 'require-replacement' && stopWaitingForPublishedHandle) {
+    if (getRecoveryReplacementPolicy(handle) === 'require-replacement' && stopWaitingForPublishedHandle) {
       // Why: once recovery is handed to accepted snapshots, repeated sends to the stale handle must not re-arm inventory RPCs.
       return
     }
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts (3)

2054-2062: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Dispose the transport at the end of the test.

The neighboring tests call transport.destroy?.() before restoring real timers. This test and the test at Line 2221 leave the transport active, which keeps its recovery timers and handle subscription registered after the test finishes. Add the same cleanup for symmetry.

♻️ Proposed change
       emitSnapshot(latestSubscribePayload().streamId, 'same handle reattached')
       expect(transport.isConnected()).toBe(true)
+      transport.destroy?.()
     } finally {
       vi.useRealTimers()
     }

2099-2100: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Prefer invariant assertions over magic poll-count windows.

The bounds > 25 and < 40 encode the current backoff schedule and the attach timeout. Similar magic windows appear at Lines 2046, 2050, and 2154. If HOST_SESSION_ATTACH_POLL_MS or the replacement poll ceiling changes, these tests fail with no indication of the real cause. The invariant that matters is already asserted at Line 2102: no further inventory calls after the cap. Consider deriving the bounds from the constants or dropping the upper magic numbers.


2734-2747: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use the subscribedTerminalHandles helper.

This block re-implements the helper added at Line 66. Call the helper instead.

♻️ Proposed change
-      const subscribedTerminals = subscriptionSendBinary.mock.calls
-        .map((call) => decodeTerminalStreamFrame(call[0]))
-        .flatMap((frame) => {
-          if (frame?.opcode !== TerminalStreamOpcode.Subscribe) {
-            return []
-          }
-          const payload = decodeTerminalStreamJson<{ terminal: string }>(frame.payload)
-          return payload ? [payload.terminal] : []
-        })
-      expect(subscribedTerminals).toEqual([
+      expect(subscribedTerminalHandles()).toEqual([
         'terminal-stale',
         'terminal-replacement',
         'terminal-replacement'
       ])

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 82bdc872-274a-40b3-b6d1-02668c05abb0

📥 Commits

Reviewing files that changed from the base of the PR and between b04c695 and a0d217b33a717e294159a56eee3a2fb74765b1ad.

📒 Files selected for processing (2)
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (4)
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts (4)

2114-2164: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Distinguish this test from the near-duplicate at Line 2609.

The test at Line 2609 uses almost the same title ("reuses prior ready evidence when the trailing inventory poll fails") and asserts the same outcome. Only the failure trigger differs: this test fails the second and later inventory calls by call index, while the other test fails only the short-timeout poll. Rename both titles to state the trigger, or merge them into one parameterized test. Distinct titles keep failure output unambiguous.

♻️ Suggested rename
-  it('reattaches from prior ready evidence when the trailing inventory poll fails', async () => {
+  it('reattaches from prior ready evidence when inventory fails after the first poll', async () => {

2527-2541: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Assert an observable effect of the strengthened policy.

The test name states that a sticky replacement policy is applied. The assertions only check that hostListCalls does not grow during a 1 second advance and that the handle set is unchanged. That result also holds when the backoff delay is simply longer than 1 second, so the test can pass without the policy being strengthened.

Advance the timers until the next inventory poll runs, then assert that the transport rejects the same terminal-old handle and does not resubscribe it. That assertion fails if the policy reverts to reuse.


2734-2742: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Reuse the new subscribedTerminalHandles() helper. This PR adds subscribedTerminalHandles() at Line 66, but two sites still decode Subscribe frames inline.

  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts#L2734-L2742: replace the inline decode and flatMap block with const subscribedTerminals = subscribedTerminalHandles().
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts#L2040-L2043: remove the local subscribeCount() helper and use subscribedTerminalHandles().length.

2547-2607: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Merge the two repeated same-handle flap tests. Both tests drive two same-handle end/resubscribe cycles, then a third end, and assert three terminal-flapping subscriptions followed by the disconnected phase. Only the runtime environment id and a few extra assertions differ.

  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts#L2547-L2607: keep this test as the single scenario and add the isConnected() assertion from the other test.
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts#L2166-L2219: remove this test, or narrow it to an assertion that the other test does not cover.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: fca02e6d-06d0-4092-87bf-c25933e5ef05

📥 Commits

Reviewing files that changed from the base of the PR and between a0d217b33a717e294159a56eee3a2fb74765b1ad and 012228c.

📒 Files selected for processing (2)
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts

@greptile-apps

greptile-apps Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Greptile Summary

This PR refactors the remote PTY transport's reconnection strategy from a binary requireReplacement boolean to a three-tier HostHandleReplacementPolicy enum (reuse / prefer-replacement / require-replacement), and adds a bounded same-handle end-reuse counter to prevent infinite reconnect flapping when a stream terminates and reattaches to the same handle repeatedly.

  • Policy tier refactor: waitForResubscribeHostSessionHandle now accepts the policy enum, tracks lastReadyHandle as prior evidence for prefer-replacement fallback, and gates subscription on require-replacement + same-handle checks; all callers updated symmetrically.
  • Same-handle end-cycle cap: replacementPolicyAfterWebStreamEnd escalates from prefer-replacement (first 2 cycles) to require-replacement (cycle 3+) using sameHandleEndReuseCount; recordSameHandleEndReuse is wired into onSubscribed only for same-handle recoveries.
  • Inventory window retry cap: beginResubscribeInventoryWindow / HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY are introduced to limit inventory retries; the counter resets on each new recovery epoch, so it always reads 1 at the check site — the actual retry limit comes from the recovery engine's own timeout.

Confidence Score: 4/5

The change is safe to merge. The core policy-tier refactor is correct and well-tested; the two findings are edge cases that do not cause incorrect terminal behavior in normal flapping scenarios.

The policy enum refactor, stale-handle guards, and same-handle reuse counter are all logically sound and covered by nine new tests. Two issues remain: the inventory window counter resets on every new recovery epoch so HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY=2 never enforces a true 2-window cap, and resetSameHandleEndReuse() on transport close allows the same-handle cycle counter to be cleared by network drops, which could delay when require-replacement kicks in for a flapping session.

Files Needing Attention: src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts — specifically the beginResubscribeInventoryWindow epoch-reset logic and the onTransportClose call to resetSameHandleEndReuse.

Important Files Changed

Filename Overview
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts Adds three-tier replacement-policy enum, same-handle end-reuse counter, and inventory-window retry cap. The policy tier refactor and stale-handle error guards are correct; two issues: the inventory window counter always resets per epoch so HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY=2 is functionally equivalent to unlimited until recovery timeout, and resetSameHandleEndReuse() on transport close allows bypassing the per-cycle reuse cap.
src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.test.ts Adds 9 new integration-style tests covering inventory retry, same-handle reuse cycles, prior-evidence reattach, stale-send race conditions, and replacement policy propagation. Tests are thorough and correctly validate the intended behaviors, though the two recovery windows test name is misleading as the actual stop is driven by the recovery engine timeout, not the window counter.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Stream onEnd fired] --> B[replacementPolicyAfterWebStreamEnd]
    B --> C{Same handle? Within timeout? Count < LIMIT?}
    C -- First 2 cycles prefer-replacement --> D[scheduleResubscribeAfterTransportClose]
    C -- After 2 cycles require-replacement --> D
    D --> E[waitForResubscribeHostSessionHandle poll session.tabs.list]
    E --> F{Inventory result}
    F -- Different handle found --> G[rebindRemoteTerminalHandle]
    F -- Same handle and prefer-replacement wait for timeout --> H[finishBoundedWait return lastReadyHandle]
    F -- Same handle and require-replacement --> I[Return undefined do not subscribe]
    F -- Inventory fails inventoryWindow less than 2 --> J[throw recoverable error recovery.schedule retry]
    G --> K[subscribeToHandle sameHandleEndRecovery=false]
    H --> L[subscribeToHandle sameHandleEndRecovery=true]
    L --> M[onSubscribed recordSameHandleEndReuse increment count]
    I --> N[stopWaitingForPublishedHandle listener waits for snapshot]
    J --> O{Recovery timeout exhausted?}
    O -- No --> D
    O -- Yes --> P[phase: disconnected]
    K --> Q[connected]
    M --> Q
Loading

Comments Outside Diff (1)

  1. src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts, line 1719-1746 (link)

    P2 onTransportClose reset allows bypassing the same-handle end-cycle cap

    resetSameHandleEndReuse() is called inside onTransportClose, which runs on network-level transport drops. If the pattern is onEnd to reuse (count = 1) to onTransportClose to counter reset to onEnd to reuse (count = 1) the counter never accumulates past 1, so require-replacement is never reached via the sameHandleEndReuseCount >= HOST_SESSION_SAME_HANDLE_END_REUSE_LIMIT path.

    A session that alternates clean stream-ends with transport closes (e.g. due to a flapping connection alongside a rapidly-cycling PTY process) would never be capped, defeating the protection this PR intends to provide. Consider whether the transport-close reset should be gated on the transport being non-recoverable (i.e. only reset on !recoverable).

Reviews (1): Last reviewed commit: "fix(terminal): add replacement policies ..." | Re-trigger Greptile

Comment on lines +290 to +297
const beginResubscribeInventoryWindow = (recoveryEpoch: number): number => {
if (resubscribeInventoryEpoch !== recoveryEpoch) {
resubscribeInventoryEpoch = recoveryEpoch
resubscribeInventoryWindows = 0
}
resubscribeInventoryWindows += 1
return resubscribeInventoryWindows
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY cap never takes effect

resubscribeInventoryWindows is reset to 0 every time resubscribeInventoryEpoch !== recoveryEpoch, which happens on every retry because each retry receives a new nextEpoch from recovery.schedule. As a result inventoryWindow is always 1 at the check site, so 1 < HOST_SESSION_INVENTORY_MAX_WINDOWS_PER_RECOVERY (2) is always true.

In practice this means: inventory failures always throw the recoverable error and keep retrying until the recovery system's own timeout exhausts — never because the window cap was reached. The constant value of 2 is indistinguishable from 100; only the value 1 would change behavior (preventing any inventory retry).

The "caps unavailable host inventory at two recovery windows" test passes because the recovery engine's own scheduling limit stops things at ~2 x 15 s, not because of this constant. The constant as written is dead code for its named purpose.

@AmethystLiang
AmethystLiang merged commit 036b1e7 into main Aug 1, 2026
43 checks passed
@AmethystLiang
AmethystLiang deleted the getting-reconnecting-to-remote-runtime branch August 1, 2026 20:46
enrick-dev pushed a commit to enrick-dev/orca that referenced this pull request Oct 15, 2026
…am (stablyai#12003)

Recovery logic strengthened with replacement-policy tiers (reuse/prefer-replacement/require-replacement) and bounded same-handle end cycles. Prevents infinite flapping by capping reuse attempts and inventory wait windows. Tracks ready evidence to reattach from prior snapshots when inventory becomes unavailable.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant