Skip to content

fix(pty-daemon): pause flooding PTYs on backpressure instead of destroying the shared socket - #5747

Merged
Kitenite merged 1 commit into
mainfrom
fix-ptyd-flood-flow-control
Jul 17, 2026
Merged

Kitenite merged 1 commit into
mainfrom
fix-ptyd-flood-flow-control

Conversation

@Kitenite

@Kitenite Kitenite commented Jul 17, 2026 •

Copy link
Copy Markdown
Member

Links

Summary

  • One full-rate flooding terminal (e.g. yes, a huge build log) used to trip the daemon's 8 MiB outbound cap and socket.destroy() the daemon↔host-service connection — severing every terminal in the org (host-service logs pty-daemon disconnected; closing N terminal WS socket(s); users see all terminals blip). Reproduced deterministically in the 2026-07-17 stress test.
  • Fix: per-session flow control. When a subscriber's socket buffers past 1 MiB, pause the producing PTYs (pty.pause()); resume on socket drain. The kernel PTY buffer fills and the flooding process blocks on its own write — it throttles itself at the source. The 8 MiB destroy stays as a last-resort for pathological control-message floods.

How It Works

  • Pty gains pause()/resume(): node-pty's public pause/resume (added upstream for VS Code's ptyHost flow control, same actuator) for spawned sessions, ReadStream.pause() for adopted-fd sessions.
  • Server.wireSession: after fanning out a chunk, any subscribed conn with writableLength > outboundPauseThreshold records the session in conn.pausedSessions and the PTY is paused. Paused PTYs emit no further data, so this fires once per congestion episode.
  • Resume paths: socket drain (guaranteed to fire — the buffer only passes the threshold after write() returned false), and conn drop (a destroyed socket never drains). A session held paused by another still-congested conn stays paused.
  • This is not a return of the removed ACK flow control (SUPER-939): no client involvement, no protocol change — the congestion signal is the daemon's own socket buffer. Renderer-slow congestion is still handled by host-service's existing bounded buffering; this covers the daemon→host-service hop.

Manual QA Checklist

  • Flood a terminal (yes) with other terminals open — no org-wide disconnect, other terminals stay live (covered by integration test with real shells + real unix socket; app-level re-run of the CDP flood stress is a follow-up)

Testing

  • New test/flow-control.test.ts (node --test, real /bin/sh + yes flood + real unix socket):
    • slow consumer → connection survives, output resumes after the consumer drains
    • consumer hard-disconnect while paused → next subscriber still gets output (conn-drop resume path)
    • Both proven to fail against the old behavior (mutation-tested: destroy-only → test 1 fails; no conn-drop resume → test 2 fails)
  • Full bun run test:integration (51/51) and bun test unit suite (57/57) pass
  • bun run typecheck (pty-daemon + host-service), bun run lint clean

Risks / Rollout

  • Risk: a paused PTY blocks the foreground process's writes — that's the intended semantics (same as VS Code), and pause windows are bounded by drain/disconnect. Sessions with no subscriber are never paused, so background jobs in unattached terminals are unaffected.
  • Rollback: revert; behavior returns to destroy-on-overflow.

Open in Stage

Summary by cubic

Prevent org-wide terminal disconnects by pausing flooding PTYs on backpressure instead of destroying the shared daemon socket. Adds per-session flow control to pty-daemon so heavy output (e.g., yes/build logs) doesn’t take down other terminals.

  • Bug Fixes
    • Implemented per-connection flow control: when a subscriber’s socket.writableLength exceeds a threshold (default 1 MiB via outboundPauseThreshold), pause the session’s PTY; resume on socket drain or connection drop. The 8 MiB outboundBufferCap remains a last-resort.
    • Extended Pty with pause()/resume() and wired both node-pty sessions and adopted-fd sessions.
    • Tracked pausedSessions per connection to avoid resuming while another subscriber is still congested.
    • Added test/flow-control.test.ts and included it in test:integration to cover slow consumer and disconnect-resume paths.

Written for commit cc9b321. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • New Features

    • Added PTY flow-control support to pause output when clients cannot keep up and resume it when capacity is available.
    • Added configurable outbound pause thresholds for server connections.
    • Improved handling of disconnected clients so shared sessions remain available and recover correctly.
  • Tests

    • Added integration coverage for slow, interrupted, and reconnecting consumers during high-volume PTY output.

@ghost

ghost commented Jul 17, 2026

Copy link
Copy Markdown

Ready to review this PR? Stage has broken it down into 4 individual chapters for you:

Title
1 Add pause and resume to Pty interface
2 Update test fakes for Pty interface changes
3 Implement per-session flow control in Server
4 Add integration tests for flow control
Open in Stage

Chapters generated by Stage for commit cc9b321 on Jul 17, 2026 9:20pm UTC.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

The PTY daemon adds guarded pause/resume operations, applies outbound congestion thresholds per connection, resumes paused sessions after draining or disconnects, and adds integration coverage for slow consumers and interrupted connections.

Changes

PTY flow control

Layer / File(s) Summary
PTY pause/resume API
packages/pty-daemon/src/Pty/Pty.ts
The Pty interface and both PTY adapters now support guarded pause() and resume() operations.
Server congestion coordination
packages/pty-daemon/src/Server/Server.ts
Configurable outbound thresholds and per-connection paused-session tracking pause congested PTYs and resume them after drain, session exit, or connection loss.
Flow-control validation and integration wiring
packages/pty-daemon/test/flow-control.test.ts, packages/pty-daemon/package.json, packages/pty-daemon/src/**/**.test.ts, packages/pty-daemon/test/byte-fidelity.test.ts
Integration tests cover slow consumers and dropped connections; test doubles implement the expanded PTY interface, and the new suite is added to the integration command.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Client as Client
  participant Socket as Client socket
  participant Server as Server
  participant PTY as Pty

  Client->>Socket: Pause reading
  PTY->>Server: Produce output
  Server->>Socket: Write output
  Socket-->>Server: Buffer exceeds threshold
  Server->>PTY: pause()
  Client->>Socket: Resume reading
  Socket-->>Server: drain
  Server->>PTY: resume()
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is clear, conventional, and accurately summarizes the main PTY backpressure fix.
Description check ✅ Passed The description covers what changed, why, testing, and rollout risks, with only minor template sections not fully filled.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-ptyd-flood-flow-control

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

🧹 Preview Cleanup Complete

The following preview resources have been cleaned up:

  • ✅ Neon database branch

Thank you for your contribution! 🎉

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/pty-daemon/src/Server/Server.ts`:
- Around line 541-554: Update resumePausedSessions to iterate over a snapshot of
conn.pausedSessions rather than the live Set, while preserving deletion,
ownership checks, and session resume behavior. Ensure synchronous pty.resume
callbacks that re-add IDs do not affect the current iteration.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 710fa3b4-4199-446a-89f9-7b19b20944e4

📥 Commits

Reviewing files that changed from the base of the PR and between 027c886 and cc9b321.

📒 Files selected for processing (8)
  • packages/pty-daemon/package.json
  • packages/pty-daemon/src/Pty/Pty.ts
  • packages/pty-daemon/src/Server/Server.ts
  • packages/pty-daemon/src/SessionStore/SessionStore.test.ts
  • packages/pty-daemon/src/SessionStore/snapshot.test.ts
  • packages/pty-daemon/src/handlers/handlers.test.ts
  • packages/pty-daemon/test/byte-fidelity.test.ts
  • packages/pty-daemon/test/flow-control.test.ts

Comment on lines +541 to +554
private resumePausedSessions(conn: ConnState): void {
for (const id of conn.pausedSessions) {
conn.pausedSessions.delete(id);
let heldElsewhere = false;
for (const other of this.conns) {
if (other !== conn && other.pausedSessions.has(id)) {
heldElsewhere = true;
break;
}
}
if (heldElsewhere) continue;
const session = this.store.get(id);
if (session && !session.exited) session.pty.resume();
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

Fix potential infinite loop during Set iteration.

Iterating over conn.pausedSessions while synchronously deleting and calling an external callback (pty.resume()) can cause an infinite loop. If session.pty.resume() causes the PTY to emit cached data synchronously, the onData handler in wireSession will run. If the connection becomes congested again, session.id is added back to conn.pausedSessions. Because JavaScript Set iterators visit elements appended during iteration (even if they were previously deleted), the re-added id will be visited again, locking the event loop indefinitely.

Clone the Set before iterating to safely resume sessions without modifying the collection being traversed.

🔒️ Proposed fix to isolate the iteration state
 	private resumePausedSessions(conn: ConnState): void {
-		for (const id of conn.pausedSessions) {
-			conn.pausedSessions.delete(id);
+		const toResume = Array.from(conn.pausedSessions);
+		conn.pausedSessions.clear();
+		for (const id of toResume) {
 			let heldElsewhere = false;
 			for (const other of this.conns) {
 				if (other !== conn && other.pausedSessions.has(id)) {
 					heldElsewhere = true;
 					break;
 				}
 			}
 			if (heldElsewhere) continue;
 			const session = this.store.get(id);
 			if (session && !session.exited) session.pty.resume();
 		}
 	}
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
private resumePausedSessions(conn: ConnState): void {
for (const id of conn.pausedSessions) {
conn.pausedSessions.delete(id);
let heldElsewhere = false;
for (const other of this.conns) {
if (other !== conn && other.pausedSessions.has(id)) {
heldElsewhere = true;
break;
}
}
if (heldElsewhere) continue;
const session = this.store.get(id);
if (session && !session.exited) session.pty.resume();
}
private resumePausedSessions(conn: ConnState): void {
const toResume = Array.from(conn.pausedSessions);
conn.pausedSessions.clear();
for (const id of toResume) {
let heldElsewhere = false;
for (const other of this.conns) {
if (other !== conn && other.pausedSessions.has(id)) {
heldElsewhere = true;
break;
}
}
if (heldElsewhere) continue;
const session = this.store.get(id);
if (session && !session.exited) session.pty.resume();
}
}
🧰 Tools
🪛 ast-grep (0.44.1)

[warning] Importing child_process exposes a command-execution surface; ensure any command/argument built from input is validated, and prefer execFile/spawn with an argument array over exec.
Context: import * as childProcess from "node:child_process";
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

(detect-child-process-typescript)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/pty-daemon/src/Server/Server.ts` around lines 541 - 554, Update
resumePausedSessions to iterate over a snapshot of conn.pausedSessions rather
than the live Set, while preserving deletion, ownership checks, and session
resume behavior. Ensure synchronous pty.resume callbacks that re-add IDs do not
affect the current iteration.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 8 files

You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/pty-daemon/src/Server/Server.ts">

<violation number="1" location="packages/pty-daemon/src/Server/Server.ts:481">
P2: Lowering `outboundBufferCap` below the default pause threshold still disconnects the shared socket before flow control runs. Derive or validate a pause threshold below the configured cap so custom buffer limits retain the new isolation behavior.</violation>

<violation number="2" location="packages/pty-daemon/src/Server/Server.ts:492">
P2: Replay bursts bypass the new per-session backpressure check, so subscribing a slow socket to enough buffered sessions can still hit `outboundBufferCap` and disconnect it. Account for replay writes and pause their source sessions when that socket crosses the threshold.</violation>
</file>
Architecture diagram
sequenceDiagram
    participant Flood as Flooding Process (e.g., yes)
    participant Pty as Session PTY
    participant Daemon as Daemon Server
    participant Sub as Subscriber Conn (slow)
    participant Socket as Sub TCP Socket

    Note over Flood,Socket: NEW: Per-session flow control at daemon→host-service hop

    Flood->>Pty: writes flood data
    Pty->>Daemon: onData(chunk)
    Daemon->>Daemon: for each subscriber conn
    Daemon->>Sub: send(output, chunk)
    Sub->>Socket: write() (buffers growing)
    Daemon->>Daemon: check writableLength > pauseThreshold (1 MiB)
    alt writableLength > threshold
        Daemon->>Daemon: record session in conn.pausedSessions
        Daemon->>Pty: NEW: pty.pause()
        Pty->>Flood: kernel PTY buffer fills → flood process blocks on write
        Note over Pty,Flood: Self-throttling at source – no more onData until resume
    end

    Note over Daemon,Sub: Resume path 1: socket drains

    Socket->>Sub: drain event (buffer fully flushed)
    Sub->>Daemon: 'drain' event
    Daemon->>Daemon: resumePausedSessions(conn)
    Daemon->>Daemon: for each session in conn.pausedSessions
    alt no other conn holds session paused
        Daemon->>Pty: NEW: pty.resume()
        Pty->>Flood: allows writes again
        Flood->>Pty: writes continue
    else another conn still congested
        Daemon->>Daemon: skip resume – session stays paused
    end

    Note over Daemon,Sub: Resume path 2: subscriber disconnects (socket destroyed)

    Sub->>Socket: close/destroy
    Daemon->>Daemon: dropConn(conn)
    Daemon->>Daemon: resumePausedSessions(conn) – same logic as above
    Daemon->>Pty: pty.resume() (if no other conn holds it)
    Pty->>Flood: allows writes again

    Note over Daemon,Socket: Destroy cap (8 MiB) unchanged – last-resort fallback
Loading

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

if (!c.subscriptions.has(session.id)) continue;
c.send(out, chunk);
if (!c.socket.destroyed && c.socket.writableLength > pauseThreshold) {
c.pausedSessions.add(session.id);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Replay bursts bypass the new per-session backpressure check, so subscribing a slow socket to enough buffered sessions can still hit outboundBufferCap and disconnect it. Account for replay writes and pause their source sessions when that socket crosses the threshold.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/pty-daemon/src/Server/Server.ts, line 492:

<comment>Replay bursts bypass the new per-session backpressure check, so subscribing a slow socket to enough buffered sessions can still hit `outboundBufferCap` and disconnect it. Account for replay writes and pause their source sessions when that socket crosses the threshold.</comment>

<file context>
@@ -460,13 +478,24 @@ export class Server {
 				if (!c.subscriptions.has(session.id)) continue;
 				c.send(out, chunk);
+				if (!c.socket.destroyed && c.socket.writableLength > pauseThreshold) {
+					c.pausedSessions.add(session.id);
+					congested = true;
+				}
</file context>

* subscribed to this session id receives the output / exit frames.
*/
private wireSession(session: Session): void {
const pauseThreshold =

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Lowering outboundBufferCap below the default pause threshold still disconnects the shared socket before flow control runs. Derive or validate a pause threshold below the configured cap so custom buffer limits retain the new isolation behavior.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/pty-daemon/src/Server/Server.ts, line 481:

<comment>Lowering `outboundBufferCap` below the default pause threshold still disconnects the shared socket before flow control runs. Derive or validate a pause threshold below the configured cap so custom buffer limits retain the new isolation behavior.</comment>

<file context>
@@ -460,13 +478,24 @@ export class Server {
 	 * subscribed to this session id receives the output / exit frames.
 	 */
 	private wireSession(session: Session): void {
+		const pauseThreshold =
+			this.opts.outboundPauseThreshold ??
+			DEFAULT_OUTBOUND_PAUSE_THRESHOLD_BYTES;
</file context>

@greptile-apps

greptile-apps Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces per-session flow control to the pty-daemon to prevent a flooding PTY (e.g., yes or a large build log) from tripping the 8 MiB outbound cap and destroying the shared daemon↔host-service socket — which was severing all terminals org-wide. The fix uses pty.pause()/pty.resume() (node-pty's existing pause/resume API) to throttle the producing PTY at the kernel level when any subscriber's socket buffer exceeds the new 1 MiB outboundPauseThreshold.

  • Pty interface gains pause()/resume() implemented in both NodePtyAdapter (delegates to this.term.pause/resume) and AdoptedPty (delegates to ReadStream.pause/resume).
  • Server.wireSession is updated to track which sessions each connection has caused to pause (conn.pausedSessions); pausing happens when writableLength > outboundPauseThreshold, and resuming happens on socket drain or connection drop.
  • test/flow-control.test.ts adds two integration tests using real shells and a real Unix socket — one for the slow-consumer path and one for the conn-drop-while-paused path.

Confidence Score: 4/5

The core flow-control mechanism is logically sound and well-tested for the happy path; the main concerns are an unvalidated configuration invariant and a timing-dependent test.

The drain-guarantee invariant is correct for all current defaults and tests, but the ServerOptions interface allows arbitrary values with no validation. The 800ms sleep in test 2 may not hold on resource-constrained runners, making the conn-drop resume path potentially unexercised.

packages/pty-daemon/src/Server/Server.ts (drain-guarantee invariant, Set iteration pattern) and packages/pty-daemon/test/flow-control.test.ts (timing-dependent pause-before-disconnect assertion).

Important Files Changed

Filename Overview
packages/pty-daemon/src/Server/Server.ts Core flow control logic: adds pausedSessions tracking per connection, pause on writableLength > threshold, and resume on socket drain and conn drop; the drain guarantee relies on an implicit invariant that threshold > socket highWaterMark.
packages/pty-daemon/src/Pty/Pty.ts Adds pause()/resume() to the Pty interface and implements them in both NodePtyAdapter and AdoptedPty with exited guards; straightforward delegation.
packages/pty-daemon/test/flow-control.test.ts Two integration tests covering the slow-consumer and conn-drop-while-paused paths; test 2 relies on timing assumptions (PTY must be paused within 800ms) that could flake on slow CI.
Prompt To Fix All With AI
Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
packages/pty-daemon/src/Server/Server.ts:541-555
**Set mutated while being iterated**

`conn.pausedSessions.delete(id)` removes the element that was just yielded by the outer `for...of` iterator over that same Set. This is technically safe in JavaScript — the spec guarantees the remaining elements are still visited — but it's an unusual pattern that relies on this subtlety. A harder-to-misread form would be to snapshot the set first and clear it up front:

```ts
const ids = [...conn.pausedSessions];
conn.pausedSessions.clear();
for (const id of ids) { … }
```

This also makes it clearer that the set is meant to be fully drained on every call.

### Issue 2 of 3
packages/pty-daemon/src/Server/Server.ts:338-341
**`drain` guarantee depends on an unvalidated invariant**

The comment correctly explains why `drain` is guaranteed: `writableLength > outboundPauseThreshold` implies `writableLength > highWaterMark` (default 16 KB for a Node.js `net.Socket`), which means `socket.write()` has already returned `false`, guaranteeing a future `drain`. This holds for the defaults (1 MiB pause threshold >> 16 KB HWM) and the test value (64 KB), but the `ServerOptions` interface accepts any value. If a caller passes `outboundPauseThreshold` below the socket's `highWaterMark` (e.g., 4 KB), the PTY could be paused with no `drain` guaranteed to fire, silently stalling the session until conn drop. An assertion or clamp at construction time would prevent this.

### Issue 3 of 3
packages/pty-daemon/test/flow-control.test.ts:89-92
**Fixed 800 ms sleep to guarantee PTY is paused**

The test pauses `c1`'s socket and sleeps for 800 ms, expecting the PTY to have been paused within that window. On a busy CI runner (or a system where `yes` is slow to start / the socket send buffer is large), 800 ms may not be enough time for the server's `writableLength` to exceed the 64 KB `PAUSE_THRESHOLD`, so `conn.pausedSessions` would be empty when `c1.socket.destroy()` fires and `resumePausedSessions` becomes a no-op — the PTY was never paused. `c2`'s `waitFor` would then succeed trivially (the PTY was always running), meaning the conn-drop resume path was never exercised and the test is a false positive.

Reviews (1): Last reviewed commit: "fix(pty-daemon): pause flooding PTYs on ..." | Re-trigger Greptile

Comment on lines +541 to 555
private resumePausedSessions(conn: ConnState): void {
for (const id of conn.pausedSessions) {
conn.pausedSessions.delete(id);
let heldElsewhere = false;
for (const other of this.conns) {
if (other !== conn && other.pausedSessions.has(id)) {
heldElsewhere = true;
break;
}
}
if (heldElsewhere) continue;
const session = this.store.get(id);
if (session && !session.exited) session.pty.resume();
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Set mutated while being iterated

conn.pausedSessions.delete(id) removes the element that was just yielded by the outer for...of iterator over that same Set. This is technically safe in JavaScript — the spec guarantees the remaining elements are still visited — but it's an unusual pattern that relies on this subtlety. A harder-to-misread form would be to snapshot the set first and clear it up front:

const ids = [...conn.pausedSessions];
conn.pausedSessions.clear();
for (const id of ids) { … }

This also makes it clearer that the set is meant to be fully drained on every call.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/pty-daemon/src/Server/Server.ts
Line: 541-555

Comment:
**Set mutated while being iterated**

`conn.pausedSessions.delete(id)` removes the element that was just yielded by the outer `for...of` iterator over that same Set. This is technically safe in JavaScript — the spec guarantees the remaining elements are still visited — but it's an unusual pattern that relies on this subtlety. A harder-to-misread form would be to snapshot the set first and clear it up front:

```ts
const ids = [...conn.pausedSessions];
conn.pausedSessions.clear();
for (const id of ids) { … }
```

This also makes it clearer that the set is meant to be fully drained on every call.

How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +338 to +341
// Flow control: the buffer only reaches the pause threshold after
// write() has returned false, so a 'drain' (buffer fully flushed) is
// guaranteed to follow and is our resume signal.
socket.on("drain", () => this.resumePausedSessions(conn));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 drain guarantee depends on an unvalidated invariant

The comment correctly explains why drain is guaranteed: writableLength > outboundPauseThreshold implies writableLength > highWaterMark (default 16 KB for a Node.js net.Socket), which means socket.write() has already returned false, guaranteeing a future drain. This holds for the defaults (1 MiB pause threshold >> 16 KB HWM) and the test value (64 KB), but the ServerOptions interface accepts any value. If a caller passes outboundPauseThreshold below the socket's highWaterMark (e.g., 4 KB), the PTY could be paused with no drain guaranteed to fire, silently stalling the session until conn drop. An assertion or clamp at construction time would prevent this.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/pty-daemon/src/Server/Server.ts
Line: 338-341

Comment:
**`drain` guarantee depends on an unvalidated invariant**

The comment correctly explains why `drain` is guaranteed: `writableLength > outboundPauseThreshold` implies `writableLength > highWaterMark` (default 16 KB for a Node.js `net.Socket`), which means `socket.write()` has already returned `false`, guaranteeing a future `drain`. This holds for the defaults (1 MiB pause threshold >> 16 KB HWM) and the test value (64 KB), but the `ServerOptions` interface accepts any value. If a caller passes `outboundPauseThreshold` below the socket's `highWaterMark` (e.g., 4 KB), the PTY could be paused with no `drain` guaranteed to fire, silently stalling the session until conn drop. An assertion or clamp at construction time would prevent this.

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines +89 to +92
// never emits 'drain').
c1.socket.pause();
await new Promise((r) => setTimeout(r, 800));
c1.socket.destroy();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Fixed 800 ms sleep to guarantee PTY is paused

The test pauses c1's socket and sleeps for 800 ms, expecting the PTY to have been paused within that window. On a busy CI runner (or a system where yes is slow to start / the socket send buffer is large), 800 ms may not be enough time for the server's writableLength to exceed the 64 KB PAUSE_THRESHOLD, so conn.pausedSessions would be empty when c1.socket.destroy() fires and resumePausedSessions becomes a no-op — the PTY was never paused. c2's waitFor would then succeed trivially (the PTY was always running), meaning the conn-drop resume path was never exercised and the test is a false positive.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/pty-daemon/test/flow-control.test.ts
Line: 89-92

Comment:
**Fixed 800 ms sleep to guarantee PTY is paused**

The test pauses `c1`'s socket and sleeps for 800 ms, expecting the PTY to have been paused within that window. On a busy CI runner (or a system where `yes` is slow to start / the socket send buffer is large), 800 ms may not be enough time for the server's `writableLength` to exceed the 64 KB `PAUSE_THRESHOLD`, so `conn.pausedSessions` would be empty when `c1.socket.destroy()` fires and `resumePausedSessions` becomes a no-op — the PTY was never paused. `c2`'s `waitFor` would then succeed trivially (the PTY was always running), meaning the conn-drop resume path was never exercised and the test is a false positive.

How can I resolve this? If you propose a fix, please make it concise.

@Kitenite

Copy link
Copy Markdown
Member Author

E2E verification: CDP flood stress re-run against this branch (dev app)

Same harness that reproduced the bug (5 real sessions on the Main workspace, WS clients attached, yes-with-SGR floods), plus a 40s sustained 4-flood phase. Per-socket close counters and host-service log checked for disconnects.

metric before fix after fix
pty-daemon disconnected log lines every flood, org-wide 0
terminal WS closes (all 5 sockets) all closed within ~2s of flood 0 across all phases
flood delivery 0 KB/s (transport dead) 50 MB/s (1 flooder) / ~44 MB/s sustained (4 flooders, ~11 MB/s each — fair split)
echo probe (quiet terminal) dead, 15/15 timeouts alive: p50 1.9ms baseline → 6–9ms under flood, p95 45ms sustained, 0 timeouts
daemon CPU 50–77% with zero delivery 44–57% while actually delivering

The flood is now bounded by consumer throughput instead of killing the shared socket, and the quiet terminal keeps working throughout.

One observation for the follow-up work (not this PR): with the transport surviving, host-service now visibly pays the per-chunk relay cost — ~93% CPU at 44–50 MB/s (ModeTracker parse + WS broadcast on the event loop). That's the measurement the loop-offload plan (plans/20260717-host-service-git-worker-pool.md, SUPER-1544) was blocked on.

@Kitenite
Kitenite merged commit 0d62c89 into main Jul 17, 2026
19 checks passed
Kitenite added a commit that referenced this pull request Jul 17, 2026
…#5751)

* perf(desktop): LRU-evict parked terminal runtimes past a cap

Parked xterm instances were kept alive forever (5,000-line buffer + WebGL
context each, no cap). Track recency on mount/detach and release the
least-recently-used parked runtimes past the cap: buffer serialized to
localStorage first, PTY untouched, rebuilt on next mount. Eviction is
deferred one tick so workspace switches re-adopt before the sweep counts.

* feat(desktop): user-configurable background-terminal cap

Settings > Terminal > "Background terminal memory": number input (2-64,
default 12, blur-commit like FontSettingSection). Stored on the local-db
settings row (migration 0042), seeded into the registry at boot, applied
live on change so lowering the cap reclaims memory immediately.

* docs(desktop): memory profile & worker offload review

CDP-verified findings across renderer/main/host-service/pty-daemon,
controlled A/Bs for the parked-terminal eviction, and the ranked
recommendation backlog (SUPER-1544..1550).

* docs(desktop): note daemon backpressure fix landed (#5747)

* test(desktop): parked-runtime LRU eviction unit tests

Covers recency ordering (not insertion order), attached-runtime
protection, deferred-sweep rescue on remount, live cap lowering,
invalid cap rejection, buffer persistence on evict, and runtime-less
entries. Each assertion verified to fail under a targeted impl
mutation (sort removed, attached included, sweep made synchronous,
setter sweep dropped, persist flag flipped).

* test(desktop): extract pure eviction policy; drop process-global module mocks

The first test cut used bun mock.module, which is process-wide and
replaced the real terminal-ws-transport for its own suite (15 downstream
failures in CI). Restructure along a real seam instead: the LRU
selection + cap clamp move to dependency-free terminal-runtime-eviction.ts,
unit-tested without mocks (mutation-verified: sort removal, attached
inclusion, bare-entry counting, clamp boundary all caught; one equivalent
mutant on the <=cap guard documented). Timer deferral and dispose wiring
remain covered by the CDP verification in the PR.

* style(host-service): fix biome formatting in ai-workspace-names

origin/main tip fails biome 2.4.2 on this block, which fails lint on
every PR's merge ref; carrying the fix here.

This branch was previously deployed

1 inactive deployment
preview — cc9b321f Deployed Jul 17, 2026 by Kitenite via Deploy Docs #6604
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant