Skip to content

[Bug]: Offline paired desktop clients retain stale tabs in orca-data.json and re-inject them upon reconnect #21341

Description

@LesleyMurfin

[Bug]: Offline paired desktop clients retain stale tabs in orca-data.json and re-inject them upon reconnect

Target Repository: stablyai/orca
Subsystem: Multi-Client Session Synchronization / Workspace Persistence / Client Tab Reconciler
Target Files: src/renderer/src/lib/workspace-session-host-persistence.ts, src/renderer/src/lib/workspace-session-host-hydration.ts
Related Upstream Issues: #21236 (Offline paired client waking from sleep retains stale tabs and re-injects zombie sessions), #21235 (Cross-client tab bar and split tree desynchronization)
Related Upstream PRs: #21295 (Multi-client split tree layout synchronization), #21296 (Session persistence debounce and partition isolation), #21300 (Distributed tombstone propagation), #21288 (Execution host fencing for sleeping agent sessions)
Severity / Priority: P1 (Session state regression; closed/reaped tabs resurrect across multi-device developer fleets)
Affects: Orca v1.4.185 – v1.4.202 (Paired multi-client setups connecting macOS/Windows desktop clients to orca serve on Linux servers)


1. Overview & Operational Summary

Orca enables multi-client paired development where developers connect one or more desktop workstations (generic client laptops on macOS or Windows) to a centralized headless daemon (orca serve running on a Headless Linux Server at 10.x.x.x).

When a developer finishes working on their laptop, closes the laptop lid, or disconnects from the local network, the generic client enters an offline/sleep state. Meanwhile, automated agents, background tasks, or secondary paired sessions continue running on the headless server. Terminals complete, subagents are released, and obsolete workspace tabs are closed and torn down on the server.

However, the offline desktop client retains a frozen snapshot of the workspace session state in its local persistent store (orca-data.json / IndexedDB). This serialized snapshot records:

  • Open terminal tabs in tabsByWorktree.
  • Split tree layout references and active leaf pane pointers.
  • PTY handles and session keys that have since been terminated on the server.

When the developer reopens their laptop lid, the generic client reconnects to orca serve. During session hydration, workspace-session-host-persistence.ts and workspace-session-host-hydration.ts execute a local-first session restoration pass without an authoritative reconciliation handshake against the server's live session manifest.

Because the hydration logic treats local tab entries as durable user intent rather than a potentially stale cache:

  1. The waking client merges its stale local tabs into the active session store.
  2. The client immediately triggers a debounced session write (persistWorkspaceSessionByHost / patchWorkspaceSessionByHost), transmitting the merged list back to the server.
  3. The closed tabs are re-injected into the server's session state and broadcast to all other connected clients.
  4. The client's UI renders the resurrected tabs as ghost panes, mounting terminal components that attempt to attach to non-existent PTY handles.
  5. The transport throws terminal_not_found, leaving the developer with broken, unclosable split leaves and triggering infinite reconnect loops.

2. Distributed Lifecycle Context (6-Column Matrix)

Step / Lifecycle Phase Observed Defect Root Cause Mechanism Expected State Transition Target File & Symbol Status / Impact
Step 5: Reconcile / Prune Waking desktop client re-injects deleted tabs into headless server session Client merges cached tabsByWorktree from orca-data.json without verifying server tombstone ledger Stale local tabs omitted from server manifest are purged before session write workspace-session-host-persistence.ts (persistWorkspaceSessionByHost) P1 Defect: Zombie tab resurrection across paired developer devices
Step 2: Synchronize Reconnecting client performs additive union instead of authoritative catch-up sync fetchWorkspaceSessionWithRuntimeHostOwners adopts missing tabs from local partition Server session manifest acts as authoritative authority; client prunes dead entries workspace-session-host-hydration.ts (fetchWorkspaceSessionWithRuntimeHostOwners) P1 Defect: Naive union merge prevents tab deletions from sticking
Step 3: Reconnect / Attach Resurrected ghost tabs attempt PTY attachment to reaped remote processes Client UI mounts RemoteRuntimePtyTransport for re-injected tabs referencing dead PTY IDs Client reconciler validates PTY existence before mounting terminal panes src/renderer/src/components/terminal-pane/remote-runtime-pty-transport.ts P0 Blocker: Permanent UI freeze; terminal_not_found error storm
Step 2: Synchronize Split tree layout in orca-data.json retains dead pane leaf nodes Serialized layout tree is written back to server partition without pruning dead leaves Dead split leaves pruned or collapsed prior to persisting layout state workspace-session-host-persistence.ts (splitWorkspaceSessionForWrite) P1 Defect: Broken split layouts with orphaned empty pane containers
Step 5: Reconcile / Prune PR #21288 fences agent sessions but leaves tab rosters desynchronized PR #21288 checks executionHostId on resume, but does not reconcile background tab deletions Tab lifecycle synchronization aligned with execution host fencing workspace-session-host-persistence.ts / PR #21288 In Review: Gap remains for passive tab roster reconciliation
Step 4: Close & Teardown Client write race overwrites server-side tab close operations patchWorkspaceSessionByHost fires on client wake, racing against server event streams Client write lock enforced until initial server manifest synchronization settles workspace-session-host-persistence.ts (patchWorkspaceSessionByHost) P1 Defect: Server-side closures undone by delayed client wake patches

3. Detailed Root Cause Analysis

3.1. Additive Session Hydration in workspace-session-host-hydration.ts

When a client application boots or wakes, fetchWorkspaceSessionWithRuntimeHostOwners() loads partition slices:

// src/renderer/src/lib/workspace-session-host-hydration.ts:174-202
const slices: HostSessionSlices = {
  [LOCAL_EXECUTION_HOST_ID]: await api.get()
}
// Read remote partitions over RPC
const [, sshPartitions] = await Promise.all([
  Promise.all([...runtimeHostIds].map(async (hostId) => {
    const slice = await readPartition(hostId)
    if (slice) {
      slices[hostId] = slice
    }
  })),
  ...
])

The hydration algorithm then merges the slices:

// src/renderer/src/lib/workspace-session-host-hydration.ts:202-231
const merged = mergeWorkspaceSessionsWithHostShadow(slices)
let session = merged.session
for (const [hostId, slice] of sshPartitions) {
  const adoption = adoptStrandedHostPartitionSession(session, slice, ...)
  session = adoption.session
}

Notice the design assumption:

  • Any tab present in the local partition that is absent from the remote host partition is assumed to be "stranded" or "unsaved local user work".
  • adoptStrandedHostPartitionSession explicitly adopts these tabs and merges them into the active session state:

    "Why: into a populated row would re-add tabs the user had closed on every launch... a workspace the merged session has no tabs for is adopted rather than read as a deletion (SSH host session partitions are written but never read — tab lists round-trip as empty and remote sync permanently deletes them #12721)."

  • While this protected local unstamped drafts from being wiped on boot, in a paired multi-client architecture it turns the client's local orca-data.json into an undeletable cache: tabs closed on the server while the client was asleep are diagnosed as "stranded" and forcibly adopted.

3.2. Blind Full-Partition Replacement in workspace-session-host-persistence.ts

Immediately following hydration or upon UI state updates, the client store invokes:

// src/renderer/src/lib/workspace-session-host-persistence.ts:245-259
export async function persistWorkspaceSessionByHost(
  api: DurableSessionApi,
  payload: WorkspaceSessionState,
  state: HostPersistenceState
): Promise<void> {
  const slices = splitWorkspaceSessionForWrite(payload, state, 'replace')
  const writes: Promise<void>[] = [api.set(slices[LOCAL_EXECUTION_HOST_ID] ?? payload)]
  for (const [hostId, slice] of nonLocalHostSessionEntries(slices)) {
    writes.push(api.set(slice, hostId))
  }
  await Promise.all(writes)
  await api.flush()
}
  • When api.set(slice, hostId) is called, it performs a complete replacement of the remote host partition with the waking client's locally merged state.
  • Because payload.tabsByWorktree contains the re-adopted stale tabs, the server's partition is overwritten with the stale tab list.
  • Other active clients connected to orca serve receive a state change notification, forcing their tab bars to re-render the dead tabs.

4. Reproduction Steps & Minimal Test Script

Reproduction Environment

  • Server: Headless Linux Server (10.x.x.x) running orca serve.
  • Client A (Desktop): Active desktop workstation connected to Headless Linux Server.
  • Client B (Laptop): Generic paired client connected to Headless Linux Server, configured to sleep/wake.

Minimal Repro Steps

  1. Connect Client A and Client B to the shared workspace on Headless Linux Server.
  2. Open three terminal tabs in worktree wt-main:
    • Tab-1: Running bash
    • Tab-2: Running build watcher
    • Tab-3: Running log tail
  3. Confirm all three tabs appear on both Client A and Client B.
  4. Put Client B to sleep (close laptop lid / disconnect network interface).
  5. On Client A (or via headless automated CLI), explicitly close Tab-2 and Tab-3:
    • Server reaps backing PTYs for Tab-2 and Tab-3.
    • Client A's UI displays only Tab-1.
    • Server's active session state contains only [Tab-1].
  6. Wake Client B (open laptop lid / reconnect network interface).

Observed Result

  1. Client B wakes up and reads its local orca-data.json, which still has [Tab-1, Tab-2, Tab-3].
  2. Client B merges [Tab-1, Tab-2, Tab-3] and writes it back to Headless Linux Server via persistWorkspaceSessionByHost.
  3. Client A's screen suddenly flashes: Tab-2 and Tab-3 reappear in Client A's tab bar!
  4. Both Client A and Client B attempt to attach to Tab-2 and Tab-3.
  5. The server responds with terminal_not_found because the backing PTYs were already terminated in Step 5.
  6. Tab-2 and Tab-3 become permanently broken ghost panes displaying error toasts ("Remote terminal was closed / 远程终端已关闭").

Expected Result

  1. Upon reconnect, Client B executes a reconciliation handshake with Headless Linux Server.
  2. Client B compares its local tabsByWorktree against the server's authoritative active manifest.
  3. Observing that Tab-2 and Tab-3 are no longer present on the server, Client B purges them from its local orca-data.json.
  4. Client B renders only [Tab-1]. No ghost tabs are re-injected to the server or Client A.

5. Proposed Fix & Implementation Blueprint

5.1. Introduce Authoritative Reconnect Handshake in workspace-session-host-persistence.ts

Add an explicit reconciliation phase upon client reconnect that validates local partition tabs against the server's active roster before allowing write-back operations:

// src/renderer/src/lib/workspace-session-host-persistence.ts

export type RemoteSessionReconciliationManifest = {
  hostId: ExecutionHostId
  activeTabIds: readonly string[]
  tombstonedTabIds: readonly string[]
  manifestEpoch: number
}

/**
 * Reconciles local cached tabs against the authoritative server manifest.
 * Drops any local tabs that have been explicitly closed on the remote host.
 */
export function reconcileLocalTabsWithServerManifest(
  localTabs: readonly TerminalTab[],
  manifest: RemoteSessionReconciliationManifest
): readonly TerminalTab[] {
  const activeSet = new Set(manifest.activeTabIds)
  const tombstoneSet = new Set(manifest.tombstonedTabIds)

  return localTabs.filter((tab) => {
    // If the server explicitly tombstoned the tab, discard local copy
    if (tombstoneSet.has(tab.id)) {
      return false
    }
    // If the tab belongs to this remote host and is absent from active manifest, discard
    if (!activeSet.has(tab.id)) {
      return false
    }
    return true
  })
}

5.2. Block Stale Overwrites in persistWorkspaceSessionByHost

Ensure that persistWorkspaceSessionByHost cannot broadcast a full replacement until the reconnecting client has completed its reconciliation handshake:

export async function persistWorkspaceSessionByHost(
  api: DurableSessionApi,
  payload: WorkspaceSessionState,
  state: HostPersistenceState,
  options: { isReconciled?: boolean } = {}
): Promise<void> {
  // If the client has just reconnected and has not settled the authoritative handshake,
  // defer or fence the write to prevent re-injecting zombie state.
  if (state.hasPendingReconnectionHandshake && !options.isReconciled) {
    console.warn('[session] Deferring session persist until authoritative handshake completes')
    return
  }

  const slices = splitWorkspaceSessionForWrite(payload, state, 'replace')
  ...
}

5.3. Align with PR #21300 and PR #21296

Integrate with the tombstone ledger established in PR #21300:

  • Maintain a circular buffer of recently closed tabIds on the headless daemon.
  • When an offline client reconnects, send down the tombstone delta so the waking client immediately prunes matching entries from orca-data.json.

6. Verification & Expected Test Matrix (6-Column Matrix)

Test Case / Scenario Precondition / Setup Action / Trigger Expected Sync Behavior Local orca-data.json State Multi-Client Verdict
Clean Reconnect Sync Client B sleeps; Tab-2 closed on Headless Linux Server Client B wakes and reconnects Tab-2 removed from Client B store; no write-back of Tab-2 Tab-2 deleted from tabsByWorktree PASS: Only live tabs displayed across all clients
Concurrent Active Edits Client B offline; Client A opens Tab-4; Tab-1 stays open Client B wakes and reconnects Client B adopts Tab-4; retains Tab-1; does not resurrect dead tabs Store updated to [Tab-1, Tab-4] PASS: Seamless convergence without ghost panes
Offline Local Draft Client B creates local unstamped terminal draft while offline Client B reconnects to remote server Local draft preserved under LOCAL_EXECUTION_HOST_ID partition Local partition retains draft; remote partition unaffected PASS: Preserves offline local work without server pollution
Split Layout Pruning Split layout with 2 leaves; right leaf closed on server Client B wakes with 2-pane split in cache Right leaf pruned; split tree collapses to single full pane Layout tree updated to single root node PASS: No empty unclosable split containers
Write Race Protection Client B triggers save immediately upon socket open Reconnect handshake in flight persistWorkspaceSessionByHost paused until handshake completes Outdated save rejected/deferred PASS: Zero zombie tab re-injections
Rapid Sleep/Wake Cycles Client B rapidly cycles network connection 5 times Server processing active agent tasks Handshake epoch guards prevent out-of-order state overwrites Store converges to latest server epoch PASS: No split tree corruption or React #185 errors

7. Upstream Coordination & Issue Checklist

  • Strict PII sanitization applied: Generic host and client names used (Headless Linux Server, 10.x.x.x, generic client).
  • Uniform 6-column matrix applied across all lifecycle and verification tables.
  • Verified cross-references to upstream issues: #21236, #21235.
  • Verified interlocking relationship with PRs: #21295, #21296, #21300, #21288.
  • Concrete reproduction scenario with multi-client sleep/wake sequence provided.
  • Concrete architectural fix blueprint authored for workspace-session-host-persistence.ts and workspace-session-host-hydration.ts.

Activity

  1. AmethystLiang commented on Sep 18, 2026

    @AmethystLiang
    Contributor

    A fix is being actively worked on. Target roll out date is early next week

  2. added
    bugSomething isn't working
    orca-remote-serverRemote Orca Server / orca serve / paired remote runtime
    on Sep 19, 2026
  3. added theissue type on Sep 19, 2026
  4. BearMett commented on Sep 22, 2026

    @BearMett
    Contributor

    Reproduced this on a macOS desktop ↔ macOS desktop pairing (no orca serve), and traced the re-injection to a concrete host-side path. Sharing in case it complements the fix in progress.

    Setup: Host = desktop Orca on Mac A hosting a local worktree. Client = desktop Orca on Mac B, paired to A. Mac B was sleeping/waking every 13–36 min all day.

    Observed on the host (daemon.log + main.trace.ndjson)

    • Host user closes 12 terminal tabs → 12 terminal.close spans, all succeeded (PTYs killed). Only 1 runtime.session-tabs.close span, for a different tab.
    • 5–20 min later, exactly when Mac B's renderer wakes, the host daemon spawns 12 new shells at once, all startup-command-delivery … hasCommand:false, and the same 12 tab ids reappear in the host's tabsByWorktree (with a fresh createdAt). Repeated 9× in one day; zero occurrences today with Mac B awake.
    • Mac B's orca-data.json backup from that day holds those 12 tab ids (bare host ids) under workspaceSessionsByHostId["runtime:<A>"].tabsByWorktree[<W>].

    Path (client wake → host spawn)

    1. Mac B wakes → system:resumed → retryAllRemoteRuntimePtyRecoveriesNow() replays each parked pane's connect() (remote-runtime-pty-transport.ts).
    2. The pane's host PTY is gone, so resolvePersistedHostPane() gets terminal_not_found and the transport falls through to the legacy terminal.create call, passing the pane's original tabId/leafId as a hint and no command (remote-runtime-pty-transport.ts ~L2226–2248).
    3. Host orca-runtime-create-terminal.ts L36–44: canAdoptPaneIdentity only checks the hint's format (isValidHostTerminalTabId + isTerminalLeafId). It never asks whether that tab id was already closed. So it adopts the id and spawns a fresh shell under it.
    4. The host republishes the worktree's session-tab snapshot; the host's own renderer mirrors it back in as web-terminal-<id>.

    So on this topology the stale tabs are not pushed as a bulk session write; they come back one terminal.create at a time, each carrying the closed tab's identity.

    Suggestion: the fix needs both halves to be idempotent

    • Client hydration (what this issue already frames): a waking client must not treat its persisted tab rows as durable intent for panes the host has since retired.
    • Host: terminal.create with a hinted tabId should consult the retirement/tombstone state and refuse with tab_not_found instead of adopting the id. Old clients already treat tab_not_found/terminal_not_found as absence proof (isMissingHostSessionSurfaceError), so this is host-only and needs no wire change — it protects a host even when the paired client has not updated. Relatedly, the host's local close path only kills the PTY and prunes renderer state; it doesn't retire the runtime session row, which is what leaves the id adoptable.

    Is the in-progress fix touching the host adoption gate, or only the client hydration side? Happy to send a host-side PR for the tombstone check if it doesn't overlap.

  5. LesleyMurfin commented on Sep 23, 2026

    @LesleyMurfin
    ContributorAuthor

    Cross-reference note: An inadvertent duplicate issue #22038 was filed earlier and has been flagged with a duplicate notice consolidating here. Client-side PR #22334 (offline client hydration reconciliation & write-race protection) has been retargeted to address this canonical issue #21341 alongside host-side PR #22149.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingorca-remote-serverRemote Orca Server / orca serve / paired remote runtimeterminal

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions