Coalesce high-frequency assistant streaming deltas

[colonelpanic8]

What Changed

• Coalesce high-frequency streaming assistant text before dispatching it to the orchestration engine.
• Flush the first delta immediately, then flush at most every 100 ms or once 512 buffered characters accumulate.
• Preserve pending text across setting transitions and flush it before message completion or lifecycle pauses.
• Add a regression test covering 250 one-character deltas and exact final-text preservation.

Why

Providers can emit assistant text one token or character at a time. Persisting and broadcasting every tiny delta creates thousands of durable events, repeatedly rebuilds the client message, and forces the renderer to reconcile and re-render Markdown at token frequency. In long-running sessions this can drive the Electron renderer past its practical memory limit even while the backend remains healthy.

The bounded coalescing keeps the streaming UI responsive while reducing persistence, replay, and renderer work by roughly two orders of magnitude for high-frequency streams. Completion and pause paths still flush all pending text, so no content is lost.

UI Changes

No visual changes. Assistant text continues to stream, with updates grouped into intervals of at most 100 ms under normal timestamped provider traffic.

Checklist

• Added focused coverage for high-frequency streaming deltas
• Ran pnpm --config.enableGlobalVirtualStore=false exec vp test run apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.test.ts
• Ran pnpm --config.enableGlobalVirtualStore=false exec vp check
• Ran pnpm --config.enableGlobalVirtualStore=false exec vp run typecheck

---

Note

High Risk
Changes the core provider-to-orchestration message pipeline, event ordering, and completion semantics; regressions could lose assistant text, duplicate completions, or stall threads on stuck retries.

Overview
Provider runtime ingestion now batches high-frequency assistant text when streaming is enabled: pending text flushes on a 100ms interval or when 512 characters accumulate, routed through dedicated worker inputs (assistant-delta, assistant-flush, assistant-finalize) instead of dispatching every token immediately.

Failed thread.message.assistant.delta / complete dispatches retain buffered text and retry with bounded backoff; turn completion, session exit, runtime errors, and approval pauses wait until deferred assistant work drains so lifecycle events do not overtake partial messages. Session thread.session.set for terminal boundaries can be deferred until assistant finalization finishes.

thread.message.assistant.complete gains optional appendText; the decider emits a final streaming delta for undispatched text, then an empty-text completion so projectors do not replace already-streamed content. Terminal fallback completion after exhausted retries uses the same path.

finalizeTrackedAssistantMessages is exported for partial turn finalization (release segment state only for messages that actually completed). Integration tests cover coalescing event counts, dispatch failures, timer flush, ordering vs plans/requests, and deduplication.

^{Reviewed by Cursor Bugbot for commit 8638e03. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Coalesce high-frequency assistant streaming deltas with retry and deferred finalization

• Streaming assistant text is now coalesced by time interval (STREAMING_ASSISTANT_FLUSH_INTERVAL_MILLIS) and size threshold (MAX_COALESCED_STREAMING_ASSISTANT_CHARS) in ProviderRuntimeIngestion.ts to reduce high-frequency tiny delta updates.
• Adds timer-driven flush scheduling, per-message deferred delta counters, and bounded retry logic so transient dispatch failures do not lose buffered text or stall turn/session boundaries.
• Boundary events (turn completed/aborted, session exited, runtime error) are deferred until all outstanding assistant deltas drain, then finalized with chunked dispatch and retries.
• ThreadMessageAssistantCompleteCommand gains an optional appendText field in orchestration.ts; decider.ts emits a preceding streaming delta event when this field is set, enabling best-effort fallback completion.
• Risk: significantly increased internal state per message (flush timestamps, deferred delta counts, finalization retry flags); clearing turn state now conditionally preserves per-message buffers when deferred deltas are outstanding, changing previous always-clear behavior.

^{Macroscope summarized 8638e03.}

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3fe7ef91-a42a-4214-8c87-5f18c363cdb0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces significant new runtime behavior for coalescing high-frequency streaming assistant deltas, including new state caches, retry/deferral mechanisms, and multiple new processing paths. The scope and complexity of these orchestration layer changes warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 770d450. Configure here.}

[juliusmarminge]

Closing in favor of #2829 (orchestration V2).

#2829 deletes the V1 orchestration layer this PR builds on — apps/server/src/orchestration/**, provider/Layers/*Adapter.ts and provider/Services/** are removed and replaced by apps/server/src/orchestration-v2/**, with the IPC surface renamed to ORCHESTRATION_V2_WS_METHODS. The files this PR touches either no longer exist or are rewritten, so it can't be rebased — it would need reimplementing against the V2 adapters.

This is not a judgement on the change itself. Several of these are real gaps we still want fixed; the base just moved out from under them.

Once #2829 merges, please rebase onto main, port the change to the V2 equivalent, and reopen (or open a fresh PR). Ping me and I'll prioritise the review.