Skip to content

fix(terminal): bound producer pause so a lost renderer ACK cannot wedge a PTY - #17216

Open
lertian wants to merge 2 commits into
stablyai:mainfrom
lertian:fix/pty-producer-pause-ceiling
Open

lertian wants to merge 2 commits into
stablyai:mainfrom
lertian:fix/pty-producer-pause-ceiling

Conversation

@lertian

@lertian lertian commented Aug 29, 2026 •

Copy link
Copy Markdown

ELI5

When a terminal pane produces output faster than the UI can draw it, Orca tells the OS to stop reading that terminal so the program inside blocks and stops flooding. It undoes that once the backlog drains. The problem: "the backlog drained" is only ever learned from the renderer sending an acknowledgement. If those acknowledgements stop arriving, nothing ever says "you can resume", and the program stays frozen mid-sentence. This adds a 10-second ceiling on how long one of those pauses can be held, so a stuck agent always gets un-stuck by itself.

What Changed

PtyProducerFlowController now bounds a single continuous paused span:

  • New PRODUCER_PAUSE_MAX_HOLD_MS = 10_000. When a PTY is paused, a timer is armed; if the span reaches the ceiling, the pause is released unconditionally and the bookkeeping dropped.
  • The ceiling measures from the first pause of a span. A re-assert (PRODUCER_PAUSE_REASSERT_INTERVAL_MS) deliberately does not re-arm it, or a sustained flood would extend the span forever and reintroduce the same unbounded hold.
  • Cleared on every exit path already in the class: drain below LOW, release(), releaseAll().
  • Timer is unref()d so a pending resume never holds the process open at exit.

No call sites changed. Sustained backpressure behavior is unchanged: pending still above HIGH re-pauses on the very next report, so this removes only the unbounded hold, not the throttling.

Why

Resume is entirely ACK-driven. updateProducerFlowControl is only ever invoked from a data/flush/ACK event, and it computes pendingChars from session.pendingData, which shrinks only when the renderer acknowledges. So the resume edge depends on a signal that can stop arriving (wedged renderer, dropped push channel, lifecycle reset mid-flight). Once it does, update either stops being called or keeps reporting a value above LOW, and the pause is held with no upper bound.

How much that hurts depends on the provider, and the two live providers disagree:

  • Daemon path — SessionProducerPause arms PRODUCER_PAUSE_FAILSAFE_MS = 5_000 and auto-resumes. Its comment states the invariant directly: "a lost resume must never wedge a shell." A held pause degrades to a 5s stutter.
  • Local path — LocalPtyProvider.pauseProducer is ptyProcesses.get(id)?.pause() with no failsafe. Nothing ever resumes it. The child stays blocked on write() indefinitely.

That asymmetry is the bug. The daemon already treats "a lost resume must never wedge a shell" as a hard invariant; the main-side controller is the layer that can hold a pause across any provider, so the same guarantee belongs here rather than being re-implemented per provider.

Why this matters more for agent panes than for a shell: a full-screen TUI runs the tty in raw mode, where a full buffer returns EAGAIN and the writer blocks. A cooked-mode shell silently discards instead, so it never wedges. On macOS, Node's stdout-to-TTY writes are synchronous, so a blocked write stalls the agent's whole event loop — the spinner freezes on one frame and input stops being read, while the process is alive and S+. That is why this presents as "the agent is frozen" rather than "output is slow".

Consistent with the existing releaseAll() comment on this class — "a local PTY left paused here would stay wedged forever" — which already identifies the wedge; this generalizes that safety net from window-destroyed to any unbounded hold.

10s is chosen to sit clearly above the daemon's 5s failsafe so the daemon's own recovery always wins on that path and this stays a genuine last resort, while remaining short enough that a user reads it as a hitch rather than a hang.

Linked Issue

Refs #12523 — the narrower match. Its diagnosis names this exact layer as root-cause stage 3: "main-process delivery window (rendererInFlightChars saturated at 1 MB, ackGatedFlushSkipCount > 100)". This PR bounds how long that saturated window can hold the producer paused. It does not claim to fix #12523's headline symptom: #12523 is bounded-but-large echo latency under a flood that still drains, whereas this is an unbounded wedge where the resume edge never arrives at all. #12523 also notes "keeping the in-flight delivery window small alone did not fix it" — agreed, and this PR does not propose that as a latency fix; it only removes the infinite hold.

Refs #12283 — Pi agent pane appears frozen at 'working' with no streamed output during long responses.

Refs #11178 — daemon busy-waits on EAGAIN writing to /dev/ptmx when the agent stops draining its PTY; same mechanism observed from the provider side.

Honest status of the #12283 link

I linked #12283 on 2026-08-29 because its reported symptom matched. On 2026-09-03 the reporter re-attributed that Windows first-tool freeze to a different mechanism — #18446, a DEC 2026 synchronized-output frame not upgraded to the interactive render path — and explicitly called it "distinct from the dense-SGR backlog in #12523". So I no longer expect this PR to resolve #12283. Its "eventually recovers" detail is in fact evidence against this path: a hold that clears on its own looks like the daemon's 5s failsafe already firing, which is the behaviour this ceiling is a backstop for, not the failure it prevents. I keep the reference because an unbounded pause on the local provider produces the same visible symptom, but this PR should not be read as #12283's fix.

Relationship to the other open fixes in this area

#12624 and #12525 are both open against the dense-SGR echo-latency symptom and change the renderer write/scheduler path. This PR is neither a duplicate nor a competitor: it lives in PtyProducerFlowController on the main side and changes only the lifetime of a producer pause, so it is composable with whichever of those lands. #15319 addresses the provider-side EAGAIN retry loop from #11178; a pause ceiling is orthogonal to write-retry pacing.

Visual Proof

N/A — no UI surface. The change is internal PTY flow-control timing; there is no visual or interaction change to capture, only the absence of a freeze.

Testing

pnpm test src/main/ipc/pty-producer-flow-control.test.ts — 17 passed (10 pre-existing, unmodified; 7 new). Verified the 10 original assertions still pass unchanged, so hysteresis, re-assert, per-PTY isolation, and throw-safety are not regressed.

New cases in a pause ceiling describe block:

  • resumes a PTY held past the ceiling with no further reports (the wedge itself)
  • ceiling is not extended by a re-assert within the same span
  • a still-flooded PTY re-pauses after the ceiling released it (throttling preserved)
  • no double resume when pending drains normally before the ceiling
  • no late resume after release() / releaseAll() already cancelled it
  • each PTY bounded on its own independent ceiling
  • bookkeeping stays consistent when the ceiling's resume throws

Typecheck clean on the changed file under --strict.

Platforms: logic is platform-agnostic (setTimeout + existing transport calls) and I did not touch any platform-guarded path. The provider gap it protects is most consequential on the local (non-daemon) provider on all three OSes. The diagnosis behind it was done on macOS against 1.4.192, whose panes are daemon-hosted; the patch itself has only been exercised by the unit tests above, on no platform end-to-end. I have not reproduced the wedge on Windows or Linux. The fix is derived from the ACK-driven resume path plus the provider asymmetry, both platform-independent.

SSH/remote: SshPtyProvider.pauseProducer routes to the relay's own pause path; this only bounds how long main holds the pause, adding an upper bound where there was none. No wire format, RPC param, or stream opcode is touched, so there is no mixed-version concern.

  • I manually tested these changes locally
  • Automated tests added/updated, or explained why not below

To be explicit about the unchecked box: I verified this at the unit level (17 tests, fake timers) and by typecheck, not by building the app and reproducing a live wedge. I could not get the wedge to occur on demand — my own panes are daemon-hosted, where the 5s failsafe masks it, which is exactly why the local-provider gap is easy to miss. A reviewer who can reach the local (non-daemon) provider path is better placed to confirm the end-to-end recovery.

AI Disclosure

This PR was produced end-to-end by an AI coding agent (an Anthropic Claude model, accessed through a gateway under the alias claude-ultimate; I can't verify which exact published version that maps to, so I won't name one). That covers the investigation, the root-cause analysis, the patch, the tests, and this PR description.

It has not had a line-by-line human review before submission. The human in the loop asked for the fix and approved sending it, but is relying on your review rather than vouching for the diff independently. Treat the reasoning in Why as a claim to check, not as a verified account.

What was actually verified, mechanically: the 17 unit tests pass, the 10 pre-existing tests still pass unmodified, and the changed file typechecks under --strict. What was not verified: any live wedge or its recovery in a running build (see Testing).

The root cause was established by reading this repository's source. The constants and the provider asymmetry were additionally cross-checked against the shipped 1.4.192 bundle to confirm the behavior is live in a release build and not just on main.

Review

Summary from the agent that wrote the patch, per the contributor guide. Unreviewed by a human — verify rather than trust.

Cross-platform: no platform branch touched; setTimeout/clearTimeout/unref only. unref?.() is optional-called so a non-Node timer shape cannot throw.

SSH/remote/local: all three providers reach this class through the same ProducerFlowControlTransport. The ceiling only adds an upper bound to a pause each provider already receives. No wire compatibility surface.

Agent/integration neutrality: no agent-specific logic. The class never learns which agent occupies a PTY, and nothing keys off agent identity.

Performance: one setTimeout per paused PTY — paused is already the rare path, and the timer is cleared on all three exit paths, so steady state adds no timers and no work in the data path. Map entries are removed with their timers, so no unbounded growth. Worst case for a genuinely sustained flood is one extra pause/resume pair per 10s per PTY, which is cheaper than the 5s daemon re-assert already in place.

Security: no new input, IPC surface, or privileged call; no data logged.

Backwards compatibility: additive. New exported constant, two new optional constructor options; existing construction sites compile and behave identically. Behavior changes only in the state that was previously a wedge.

Risk: the deliberate trade-off is that releasing a still-flooded PTY lets it produce again, so a real flood re-fills pending and re-pauses. That is the intended failure mode — a bounded stutter under flood is strictly better than an unbounded freeze, and it matches the reasoning the daemon failsafe already encodes.

Agent skill upstream boundary

  • Not applicable, or this change follows docs/reference/agent-skill-sharing-upstream-boundary.md and copies or mechanically translates no upstream skill-installer source, tests, fixtures, registry entries, path tables, comments, or documentation.

Notes

One adjacent thing I deliberately left out of scope:

  1. LocalPtyProvider.pauseProducer could grow its own failsafe mirroring SessionProducerPause, giving defense in depth at the provider layer. I put the bound in the controller instead because it covers every provider at once and keeps the invariant in the layer that owns the hold. Happy to add the provider-level one too if you'd prefer it there.

Author: not on X.

…ge a PTY

Resume is driven by renderer ACKs draining pending below LOW. If those ACKs
stop arriving, pending never drains and the pause is held indefinitely. The
daemon path survives this on its own 5s lost-resume failsafe, but
LocalPtyProvider.pauseProducer is a bare node-pty pause() with no failsafe, so
the child stays blocked on write until the window is destroyed — a live agent
TUI frozen mid-turn with only Cmd+Shift+R (releaseAll) to recover.

Cap one continuous paused span at 10s. A still-flooded pending re-pauses on the
next report, so sustained backpressure is unchanged; only the unbounded hold is
removed. The ceiling measures from the first pause of a span and is not extended
by a re-assert, or a sustained flood would never release.
@greptile-apps

greptile-apps Bot commented Aug 29, 2026 •

Copy link
Copy Markdown

Greptile Summary

The PR bounds each continuous producer pause so a lost renderer acknowledgement cannot leave a PTY indefinitely wedged.

  • Arms an unreferenced 10-second ceiling when producer backpressure pauses a PTY.
  • Cancels the ceiling when normal draining or teardown releases the pause.
  • Adds unit coverage for timeout recovery, reassertion, cleanup, per-PTY isolation, and transport errors.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
src/main/ipc/pty-producer-flow-control.ts Adds bounded pause lifetime and complete timer cleanup; the previously unsafe diagnostics callback has been removed.
src/main/ipc/pty-producer-flow-control.test.ts Covers ceiling expiry, non-extension by reassertion, re-pausing, cancellation, isolation, and throwing resume callbacks.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Pending output exceeds high watermark] --> B[Pause PTY producer]
  B --> C[Arm maximum-hold timer]
  C --> D{Pending drains below low watermark?}
  D -->|Yes| E[Cancel timer and resume producer]
  D -->|No, ceiling expires| F[Drop paused bookkeeping and resume producer]
  F --> G{Later report still exceeds high watermark?}
  G -->|Yes| B
  G -->|No| H[Continue normally]
Loading

Reviews (2): Last reviewed commit: "refactor: drop unused pause-ceiling diag..." | Re-trigger Greptile

@coderabbitai

coderabbitai Bot commented Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Team

Run ID: a68c5ad2-45c5-4e66-afc2-e6c37b60106b

📥 Commits

Reviewing files that changed from the base of the PR and between 37e9d55 and e51054f.

📒 Files selected for processing (2)
  • src/main/ipc/pty-producer-flow-control.test.ts
  • src/main/ipc/pty-producer-flow-control.ts
💤 Files with no reviewable changes (2)
  • src/main/ipc/pty-producer-flow-control.test.ts
  • src/main/ipc/pty-producer-flow-control.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

The producer flow controller adds a 10-second default ceiling for continuous PTY pauses. It arms the ceiling on the initial pause and does not extend it on re-assertion. It clears the ceiling when output drains or the PTY is released. When the ceiling expires, the controller resumes the producer and updates pause bookkeeping. Tests cover timer behavior, cancellation, independent PTYs, re-pausing, and resume exceptions.

Merge Risk: 🟡 Moderate · up to e5105

The new PTY pause-ceiling timer can still allow a throwing diagnostics callback to surface as an uncaught main-process error, potentially destabilizing the application during backpressure handling; this exception path should be handled or explicitly accepted before merge.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Linked Issues check ⚠️ Warning The PR implements a relevant mitigation for [#12283] by preventing indefinite PTY pauses, but it does not demonstrate that the Windows freeze is caused by this path or that the issue is resolved end t… Either provide Windows end-to-end evidence that this pause ceiling resolves [#12283], or link this PR to a narrower issue for the pause-ceiling mitigation and keep [#12283] as related context rather than the direct issue being addressed.
✅ Passed checks (4 passed)
Check name Status Explanation
Out of Scope Changes check ✅ Passed The code and tests remain focused on the producer pause ceiling, its lifecycle cancellation, re-assertion behavior, exception safety, and API cleanup. No unrelated changes are evident.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…
Title check ✅ Passed The title clearly and concisely describes the main change: bounding PTY producer pauses to prevent a lost renderer acknowledgement from wedging a PTY.
Description check ✅ Passed The description is detailed and covers the required sections, including the problem, implementation, rationale, linked issues, visual proof, testing, AI disclosure, review notes, scope boundaries, and…
Full details: Linked Issues check

Explanation

The PR implements a relevant mitigation for [#12283] by preventing indefinite PTY pauses, but it does not demonstrate that the Windows freeze is caused by this path or that the issue is resolved end to end.

  • Fix all pre-merge checks with AI

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1


ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5f2c5e76-6368-47aa-babb-9ae462de5e76

📥 Commits

Reviewing files that changed from the base of the PR and between a1f198b and 37e9d55.

📒 Files selected for processing (2)
  • src/main/ipc/pty-producer-flow-control.test.ts
  • src/main/ipc/pty-producer-flow-control.ts

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Review feedback (Greptile/CodeRabbit): the callback had no callers and a future
implementation could throw inside the ceiling timer and take down the main
process. Removing it shrinks the API surface and the risk.
@lertian

lertian commented Sep 8, 2026

Copy link
Copy Markdown
Author

CI has never run — needs a maintainer to approve workflows

PR test LoC and PR Checks are both sitting at action_required (since 2026-08-31, commit e51054f). This is a first-time outside contributor on a fork, so the workflows are gated behind manual approval and nothing has executed. Nothing can merge until someone with write access clicks Approve to run workflows — that approval is the only thing blocking this PR; the diff itself is unchanged since both bots last scored it (Greptile 5/5, CodeRabbit "no actionable comments").

For whoever opens it: 30-second version

PtyProducerFlowController pauses a PTY above a 256 KB pending watermark and resumes only when renderer ACKs drain pending below 32 KB. Every exit from a paused state depends on that ACK arriving. The daemon path survives a lost ACK via PRODUCER_PAUSE_FAILSAFE_MS = 5_000; LocalPtyProvider.pauseProducer is a bare node-pty pause() with no equivalent, so a lost ACK leaves the child blocked in write() until releaseAll() on window teardown — i.e. a live agent TUI frozen mid-turn, recoverable only with Cmd+Shift+R. This PR adds a 10 s ceiling on one continuous paused span. Deliberately above the daemon's 5 s, so on the daemon path the existing failsafe always wins and this stays a last resort.

Two properties reviewers usually ask about, both covered by tests:

  • A re-assert does not re-arm the ceiling, otherwise a sustained flood would extend the span forever and reintroduce the same unbounded hold.
  • A still-flooded PTY re-pauses on the next report, so real backpressure is untouched — this removes only the infinite hold.

Not a duplicate of the other open terminal PRs

I linked #12523 as the narrower issue and #12283 is now re-attributed to #18446 (synchronized-output frames), so I want to be explicit: this does not claim to fix either. #12624 / #12525 change the renderer write/scheduler path for dense-SGR echo latency; #15319 paces the provider-side EAGAIN retry. This one changes only pause lifetime in main, and is composable with whichever of those lands. Happy to relink or split if you'd rather track the unbounded-hold case as its own issue — say the word and I'll file it.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant