Skip to content

fix(terminals): attribute mass terminal kills and confirm bulk orphan sweep (#9949) - #10013

Closed
bbingz wants to merge 2 commits into
stablyai:mainfrom
bbingz:fix/9949-terminal-kill-attribution
Closed

bbingz wants to merge 2 commits into
stablyai:mainfrom
bbingz:fix/9949-terminal-kill-attribution

Conversation

@bbingz

@bbingz bbingz commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Addresses #9949 — implements both asks from the issue: make every mass-terminal-kill path attributable in one log read, and put a confirmation gate on the one bulk kill that had none.

Background

On 2026-07-22T14:04:33Z, ~25 daemon PTY sessions across ~20 worktrees were killed in ~1.5 s by a sequential per-terminal close storm executed through the desktop app, while the app itself stayed alive (continuous renderer heartbeat, no daemon restart, no new daemon client). Post-incident forensics ruled out the updater (#8261), Manage-Sessions/status-bar killAll, mass worktree sleep (zero keepHistory checkpoints, no sleep records), and agent hibernation (feature flag off). The kill signature matches per-session pty:kill (immediate:true) plus a terminal.close → RuntimePtyController.kill (immediate:false) leg for runtime-tracked terminals.

The investigation dead-ended on attribution: the daemon's session-killed log carries no client/request identity, none of the renderer's mass-kill entry points leave a breadcrumb, and runtime RPC dispatches (orca-cli / paired devices — two were active that day) log nothing durable. This PR closes that gap so the next occurrence is diagnosable from the logs directly.

Changes

Daemon (daemon-server.ts): the session-killed log line now includes clientId and requestId. (keepHistory is not in the kill protocol payload; the protocol is untouched.)

Renderer — one best-effort terminal_mass_kill crash-breadcrumb per invocation (never per session) at every mass-kill entry point, landing in the persisted renderer.breadcrumb trace:

source site
kill-all-surfaces runKillAllTerminalSurfaces (injectable dependency, tested)
worktree-sleep runSleepWorktrees (covers UI multi-sleep and externally-relayed ui:sleepWorktree; single producer, no relay-site duplicate)
hibernation runAgentHibernationTick when candidates > 0
orphan-sweep bulk orphan kill on confirm

Runtime RPC (runtime-rpc.ts): dispatches of kill-capable methods (terminal.close/closeTab/stop/stopExact, session.tabs.close, agentTeams.tmuxCompat, worktree.sleep/rm) log one attribution line with the calling device id/name/scope on both transports — WebSocket (paired devices) and the local socket (orca-cli), the latter with a synthetic local-socket origin. Ids-only target summary, no payload dumps.

Resource Manager (ResourceUsageStatusSegment.tsx + new orphan-terminal-kill-gate.ts): the bulk "Kill N orphan terminals" button no longer kills on click. It stages the orphan snapshot and opens a confirmation dialog; kills run only from confirm, which re-filters against live tab bindings so a session adopted by a tab while the dialog sits open is never swept (the no-confirm path was the #8459 hazard). Per-row single-session kill is unchanged. New strings localized in es/ja/ko/zh following the file's existing conventions.

Out of scope

No kill-semantics or ordering changes; no daemon protocol changes; the underlying external trigger of the 2026-07-22 incident remains unidentified — this instrumentation is what identifies it next time.

Testing

  • 103 tests pass across the 6 touched/new test files, including: daemon attribution (red-green verified), WS + local-socket audit emission, negative test on a read-only allowlisted method, breadcrumb count assertions, orphan-gate stages-without-killing, rebind-before-confirm never kills, hibernation tick candidates>0/=0.
  • tsc node + web projects: no errors. oxlint: clean. max-lines ratchet: no new bypasses. Localization catalog + locale parity: pass (coverage check fails only on 3 pre-existing ai-vault strings present on the base commit).
  • Reviewed by a multi-agent pass (correctness + hygiene + Codex); all 18 confirmed findings fixed, including auditing the local-socket transport, the agentTeams.tmuxCompat kill path, and the confirm-time rebind race.

@bbingz

bbingz commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Updated per @AmethystLiang's recommendations on the issue (#9949 (comment) — 18:09Z comment):

  • "log a reason/correlation ID so a repeat tells us exactly which action initiated every kill" — covered: daemon session-killed now carries clientId + requestId; every renderer mass-kill entry emits a terminal_mass_kill breadcrumb; and as of 3f6e39f the kill-capable RPC dispatch audit is persisted through the durable crash-breadcrumb store (which also lands in the trace file), not just console — a packaged GUI launch discards stdout, which is exactly where the original incident ran.
  • "guard both global and individual PTY-kill IPC paths" — the global paths are guarded/attributed here (incl. the previously confirmation-less bulk orphan sweep). Per-call reason threading through the individual pty:kill IPC was deliberately left out to keep this diff surgical; happy to do it as a follow-up if you want it in one go.
  • "E2E tripwire asserting browser actions emit zero PTY-kill RPCs" — agreed and not in this PR; it needs an IPC-observation hook in the e2e harness that doesn't exist yet. Sketch: expose a test-only counter on pty:kill/terminal.close dispatches, drive N embedded-browser tab opens via the existing electron-headless project, assert counter stays 0. Can file a follow-up issue/PR for it.

Also agentTeams.tmuxCompat is now audited only for kill-pane/kill-window/kill-session subcommands so the high-volume compat polling (list-panes etc.) doesn't spam the audit.

@coderabbitai

coderabbitai Bot commented Jul 22, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 500bd745-009c-44da-a4f9-af42e7d55664

📥 Commits

Reviewing files that changed from the base of the PR and between b1a6dd069fdae441198b9b3172082c8476f09a64 and 907a08e.

📒 Files selected for processing (19)
  • src/main/daemon/daemon-server-kill-attribution.test.ts
  • src/main/daemon/daemon-server.ts
  • src/main/runtime/runtime-rpc.test.ts
  • src/main/runtime/runtime-rpc.ts
  • src/renderer/src/components/shared/kill-all-terminal-surfaces.test.ts
  • src/renderer/src/components/shared/kill-all-terminal-surfaces.ts
  • src/renderer/src/components/sidebar/sleep-worktree-flow.test.ts
  • src/renderer/src/components/sidebar/sleep-worktree-flow.ts
  • src/renderer/src/components/status-bar/ResourceUsageStatusSegment.tsx
  • src/renderer/src/components/status-bar/orphan-terminal-kill-gate.test.ts
  • src/renderer/src/components/status-bar/orphan-terminal-kill-gate.ts
  • src/renderer/src/hooks/useIpcEvents.ts
  • src/renderer/src/i18n/locales/en.json
  • src/renderer/src/i18n/locales/es.json
  • src/renderer/src/i18n/locales/ja.json
  • src/renderer/src/i18n/locales/ko.json
  • src/renderer/src/i18n/locales/zh.json
  • src/renderer/src/lib/agent-hibernation-coordinator.test.ts
  • src/renderer/src/lib/agent-hibernation-coordinator.ts
🚧 Files skipped from review as they are similar to previous changes (18)
  • src/renderer/src/components/sidebar/sleep-worktree-flow.ts
  • src/renderer/src/hooks/useIpcEvents.ts
  • src/renderer/src/components/status-bar/orphan-terminal-kill-gate.test.ts
  • src/renderer/src/lib/agent-hibernation-coordinator.ts
  • src/main/daemon/daemon-server-kill-attribution.test.ts
  • src/renderer/src/components/status-bar/orphan-terminal-kill-gate.ts
  • src/renderer/src/i18n/locales/es.json
  • src/renderer/src/lib/agent-hibernation-coordinator.test.ts
  • src/renderer/src/components/sidebar/sleep-worktree-flow.test.ts
  • src/renderer/src/components/shared/kill-all-terminal-surfaces.test.ts
  • src/renderer/src/i18n/locales/ja.json
  • src/renderer/src/i18n/locales/en.json
  • src/renderer/src/i18n/locales/ko.json
  • src/renderer/src/components/shared/kill-all-terminal-surfaces.ts
  • src/renderer/src/components/status-bar/ResourceUsageStatusSegment.tsx
  • src/main/runtime/runtime-rpc.ts
  • src/renderer/src/i18n/locales/zh.json
  • src/main/runtime/runtime-rpc.test.ts

📝 Walkthrough

Walkthrough

The changes add structured attribution to daemon and runtime kill-capable RPC events, including request, device, origin, and target details. Renderer cleanup paths now record terminal_mass_kill breadcrumbs. Orphan terminal cleanup is staged behind a confirmation dialog, rechecks bindings before killing, updates session state, and refreshes afterward. Tests cover backend attribution, breadcrumb counts and sources, rebinding behavior, cancellation, and localized confirmation UI text.

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers the changes and testing, but it omits required template sections like Screenshots, AI Review Report, Security Audit, and Notes. Add the missing sections and include a Screenshots entry, an AI review summary with cross-platform checks, a security audit, and any platform-specific notes.
Docstring Coverage ⚠️ Warning Docstring coverage is 22.22% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: attributing mass terminal kills and adding confirmation for bulk orphan sweeps.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Checkov (3.3.8)
src/renderer/src/i18n/locales/es.json

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

src/renderer/src/i18n/locales/en.json

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

src/renderer/src/i18n/locales/ja.json

Traceback (most recent call last):
File "/usr/local/bin/checkov", line 2, in
from checkov.main import Checkov
ModuleNotFoundError: No module named 'checkov'

  • 2 others

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: a3112bd4-563f-4338-b264-1f4557f9633d

📥 Commits

Reviewing files that changed from the base of the PR and between 01bcc57 and 70ff80abdc7eb30ef90284db14dbce7c9776e4bd.

📒 Files selected for processing (19)
  • src/main/daemon/daemon-server-kill-attribution.test.ts
  • src/main/daemon/daemon-server.ts
  • src/main/runtime/runtime-rpc.test.ts
  • src/main/runtime/runtime-rpc.ts
  • src/renderer/src/components/shared/kill-all-terminal-surfaces.test.ts
  • src/renderer/src/components/shared/kill-all-terminal-surfaces.ts
  • src/renderer/src/components/sidebar/sleep-worktree-flow.test.ts
  • src/renderer/src/components/sidebar/sleep-worktree-flow.ts
  • src/renderer/src/components/status-bar/ResourceUsageStatusSegment.tsx
  • src/renderer/src/components/status-bar/orphan-terminal-kill-gate.test.ts
  • src/renderer/src/components/status-bar/orphan-terminal-kill-gate.ts
  • src/renderer/src/hooks/useIpcEvents.ts
  • src/renderer/src/i18n/locales/en.json
  • src/renderer/src/i18n/locales/es.json
  • src/renderer/src/i18n/locales/ja.json
  • src/renderer/src/i18n/locales/ko.json
  • src/renderer/src/i18n/locales/zh.json
  • src/renderer/src/lib/agent-hibernation-coordinator.test.ts
  • src/renderer/src/lib/agent-hibernation-coordinator.ts

@bbingz

bbingz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current origin/main (72a0a4dbb8).

Conflict resolution (2nd commit only):

  • runtime-rpc.ts: kept main's DeviceEntry import and this PR's recordDurableCrashBreadcrumb import
  • runtime-rpc.test.ts: kept main's DEVICE_REGISTRY_FILENAME / E2EE_KEYPAIR_FILENAME imports and this PR's getCrashBreadcrumbSnapshot import

Focused tests green (109): kill-attribution, runtime-rpc, orphan-kill-gate, kill-all surfaces, sleep-worktree, hibernation coordinator.

Heads: 2b585c933a + b1a6dd069f (ahead=2 behind=0 vs main).

bbingz added 2 commits July 23, 2026 11:25
… sweep

An unattributable close storm mass-killed ~25 agent PTY sessions across
~20 worktrees while the app stayed alive (stablyai#9949). No mass-kill path
recorded who invoked it, so the initiator could not be identified.

- daemon: session-killed log now carries clientId + requestId
- renderer: one terminal_mass_kill breadcrumb per invocation at every
  mass-kill entry (kill-all surfaces, worktree sleep, hibernation tick,
  orphan sweep)
- runtime-rpc: kill-capable dispatches (both WS and local-socket
  transports) log an attribution line naming the calling device/scope
- resource manager: bulk "Kill N orphan terminals" now requires a
  confirmation dialog, and confirm re-filters against live tab bindings
  so a session adopted while the dialog is open is never swept
… kill subcommands

A packaged GUI launch discards stdout, exactly where the stablyai#9949 kills
happened, so the console-only audit line left no post-incident record.
Route the record through the durable crash-breadcrumb store (which also
lands in the persisted trace) with console.info kept as the dev echo.

Audit agentTeams.tmuxCompat only for kill-pane/kill-window/kill-session
subcommands; the method itself is high-volume (list-panes polling).
@bbingz

bbingz commented Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Restack note

Rebased onto latest main (force-with-lease). Both commits applied cleanly.

Scope (unique vs main):

Coordinates with (do not duplicate UX):

Prefer one confirm surface for orphan/unbound bulk actions: this PR’s confirm gate for orphan sweep; #8467’s review dialog for idle-shell classification. #8888 should soft-deny at the protocol layer without adding a third confirm modal.

@OrcaWin

OrcaWin commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Superseded by #10129. The deterministic mixed-generation harness reproduces the burst through sequential reasonless session.tabs.close requests from a stale paired desktop viewer; the orphan-sweep confirmation guard in this PR does not intercept that RPC path. #10129 refuses the ambiguous legacy runtime request while preserving explicit current-runtime and legacy-mobile closes, and intentionally omits these diagnostics. Please review #10129 rather than merging this PR.

@bbingz

bbingz commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Closing in favor of #10129 per @OrcaWin's review — agreed that the orphan-sweep confirmation guard here does not intercept the session.tabs.close RPC path that actually produces the burst, so #10129 is the correct fix.

Noting for the record that the kill-attribution diagnostics in this PR (daemon-side attribution audit + the kill-all confirm gate) are intentionally omitted from #10129. Happy to re-propose them as a separate, smaller PR if they are wanted once #10129 lands — otherwise this can stay closed.

@bbingz bbingz closed this Jul 24, 2026
@bbingz
bbingz deleted the fix/9949-terminal-kill-attribution branch August 1, 2026 00:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants