fix(think,agents): opt-in proactive + reactive recovery for mid-turn context-window overflow by threepointone · Pull Request #1662 · cloudflare/agents

threepointone · 2026-06-03T00:18:34Z

Summary

Compaction (Session.compactAfter) is only checked between turns — on each appendMessage. But a single long, tool-heavy turn grows the prompt step by step inside one streamText loop and can exceed the model's context window mid-turn, before the next pre-turn check. The provider then rejects the request ("prompt is too long" / context_length_exceeded) and the turn dies terminally. Think shipped no provider-specific error matching, so it could neither detect nor recover from this.

This PR adds opt-in, provider-agnostic recovery, configured through a single contextOverflow property on Think. All layers default off (no behavior change unless enabled) and all reuse the session's existing compaction function.

Design principle (same as tokenCounter): the framework owns the mechanism; the app owns the provider-specific knowledge. Think never matches provider error strings in core — the app classifies the error, Think reacts.

What's added

1. `classifyChatError(error, ctx)` — the classification seam

Maps a raw error (or the in-stream error string) to a ChatErrorClassification: "context_overflow" | "rate_limit" | "transient" | "fatal" | "unknown". The classification is threaded to onChatError and observers via ChatErrorContext.classification. During recovery the context also carries { stage: "stream", requestId }, so a classifier can correlate the error with the in-flight turn (e.g. to cancelChat(requestId)). For the common case, a bundled, exported defaultContextOverflowClassifier covers Anthropic / OpenAI / Google / Bedrock / Mistral patterns:

import { Think, defaultContextOverflowClassifier } from "@cloudflare/think";

export class MyAgent extends Think<Env> {
  override contextOverflow = { reactive: true };
  override classifyChatError = defaultContextOverflowClassifier;
}

2. `contextOverflow.reactive` (+ `maxRetries`) — reactive backstop

When a turn fails with a context_overflow, Think discards the truncated partial assistant message, runs session.compact(), and re-runs the turn (bounded by maxRetries, default 1) from the compacted history. The partial is intentionally not persisted: the retry restarts the turn from scratch, so keeping the cut-off message would orphan it beside the recovered answer (and duplicate any tool work the retry re-issues). A no-op compaction or a spent budget surfaces terminally through onChatError with classification: "context_overflow" — never a silent end, never an infinite loop. Wired into the WebSocket, chat()/RPC, and programmatic (saveMessages / submitMessages) turn paths.

3. `contextOverflow.proactive` — proactive guard (the root-cause fix)

A { maxInputTokens, headroom?, maxCompactions? } pre-step guard. Before each step, Think reads the previous step's model-reported usage.inputTokens and, if it crosses maxInputTokens * (headroom ?? 0.9), compacts in place and feeds the recompacted history into the upcoming step — heading off the provider rejection before it happens. Keys off model-reported usage (every provider reports it), not provider error strings. Bounded per turn by its own maxCompactions (default 1), independent of the reactive maxRetries budget, so it cannot thrash and a no-op can't compact on every step.

override contextOverflow = {
  reactive: true,                          // backstop
  proactive: { maxInputTokens: 200_000 }   // heads most overflows off
};

4. Abort responsiveness

All three reactive driver loops now skip compaction + retry when the turn's abortSignal is already aborted, falling through to the existing terminal delivery. An aborted turn no longer runs an expensive compaction (an LLM summarization) or a wasted retry.

Observability

chat:context:compacted on the agents:chat channel — { reason: "proactive" | "reactive", shortened, requestId?, attempt? } — emitted once per compaction.

Why only the in-stream seam (investigation, now test-locked)

Recovery hooks the in-stream error-part seam (action.type === "error"), not the thrown-error catch. streamText re-enqueues even top-level request rejections as { type: "error" } fullStream parts, and toUIMessageStream passes them through without throwing — so provider context-overflow 400s always reach the in-stream seam on every path. This is no longer just asserted: a test drives a model whose doStream rejects (top-level throw) and confirms recovery still fires, so a future AI-SDK change would fail the suite rather than silently regress.

Edge cases / safety

No-op compaction (history can't be shortened) → terminal via onChatError, classified. Not a loop.
Budget exhausted → same unified terminal path.
Proactive splice (recompacted head + in-flight tail re-baseline) is best-effort, try/catch-guarded; on failure it proceeds unchanged and the reactive backstop catches the eventual overflow. The multi-fire path (guard fires twice in one turn) is covered by a test asserting tool-call/result pairing stays intact across the splice.
WebSocket regression fixed: the overflow early-return in _streamResult previously left doneSent = false, firing a spurious terminal done frame mid-recovery — now suppressed (regression-guarded).
DX guard: a one-time warning fires if contextOverflow.reactive is enabled but classifyChatError was never overridden.
Effectiveness depends on the app's compaction config — a no-op degrades gracefully (terminal, not a loop).

Verified live against Workers AI

The new example was run end-to-end against @cf/meta/llama-3.1-8b-instruct (~7968-token window):

An oversized prompt is rejected with 5021: ...exceeded this model context window limit (7968), and defaultContextOverflowClassifier classifies it as context_overflow (both Error and string forms).
An overflowing turn recovers: compaction summarizes stale history and the turn completes (status: "completed", one compaction event).

Test plan

npm run check (sherif + export checks + oxfmt + oxlint + typecheck) — all 91 projects clean
@cloudflare/think full suite — 544 tests pass
New / strengthened coverage (assistant-agent-loop.test.ts): discard-partial regression; reactive success; opt-in-off terminal; no-op-compaction terminal; budget-bounded overflow; WebSocket / chat() / programmatic paths; abort-during-recovery on all three paths; combined proactive+reactive in one turn; top-level doStream rejection recovery; multi-fire proactive splice integrity (tool pairing + recompacted head); chat:context:compacted payload contract; defaultContextOverflowClassifier matches + non-matches
CI: npx nx affected -t test

Docs & example

docs/think/index.md (config + Context-window overflow recovery section, proactive.maxCompactions, example link), docs/think/lifecycle-hooks.md (classifyChatError + requestId in context), docs/observability.md (chat:context:compacted)
packages/think/README.md (the two independent caps), root README.md (example listed)
New examples/context-overflow-recovery — a focused, full-stack Think + Workers AI demo (live-verified). Inflates history (not the triggering message) so compaction can reclaim room — the realistic, reliable shape. examples/assistant also wires the opt-in.

Reminder: docs/* changes are not auto-synced — port to cloudflare/cloudflare-docs (src/content/docs/agents/) when this merges.

Changeset

@cloudflare/think (patch) + agents (patch) — additive, opt-in; the agents bump is the new observability event type.

Notes for review

New public surface: contextOverflow / ContextOverflowConfig (incl. proactive.maxCompactions), classifyChatError / ChatErrorClassification, ChatErrorContext.classification, defaultContextOverflowClassifier.
Riskiest bit is the proactive guard's head/tail reconciliation (try/catch-guarded, reactive backstop underneath, multi-fire splice test).

changeset-bot · 2026-06-03T00:18:39Z

🦋 Changeset detected

Latest commit: 87457d8

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages

Name	Type
@cloudflare/think	Patch
agents	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

pkg-pr-new · 2026-06-03T00:25:34Z

Open in StackBlitz

agents

npm i https://pkg.pr.new/agents@1662

@cloudflare/ai-chat

npm i https://pkg.pr.new/@cloudflare/ai-chat@1662

@cloudflare/codemode

npm i https://pkg.pr.new/@cloudflare/codemode@1662

hono-agents

npm i https://pkg.pr.new/hono-agents@1662

@cloudflare/shell

npm i https://pkg.pr.new/@cloudflare/shell@1662

@cloudflare/think

npm i https://pkg.pr.new/@cloudflare/think@1662

@cloudflare/voice

npm i https://pkg.pr.new/@cloudflare/voice@1662

@cloudflare/worker-bundler

npm i https://pkg.pr.new/@cloudflare/worker-bundler@1662

commit: 87457d8

devin-ai-integration

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 7 additional findings.

Compaction (`Session.compactAfter`) is only checked between turns, on each `appendMessage`. A single long, tool-heavy turn grows the prompt step by step inside one `streamText` loop and can exceed the model's context window mid-turn, before the next pre-turn check. The provider then rejects the request ("prompt is too long" / context_length_exceeded) and the turn dies terminally. Think shipped no provider-specific error matching, so it could neither detect nor recover from this. This adds opt-in, provider-agnostic recovery, configured through a single `contextOverflow` property on `Think`. All layers default off — no behavior change unless enabled — and all reuse the session's existing compaction function. - classifyChatError(error, ctx): app maps a raw error (or the in-stream error string) to a ChatErrorClassification ("context_overflow" | "rate_limit" | "transient" | "fatal" | "unknown"). Same framework-owns-the-mechanism / app-owns-the-provider-knowledge split as `tokenCounter`. The classification is threaded to onChatError and observers via ChatErrorContext.classification. The bundled, exported defaultContextOverflowClassifier covers the common providers (Anthropic, OpenAI, Google, Bedrock, ...) for apps that do not need custom classification. - contextOverflow.reactive + .maxRetries: when a turn fails with a context_overflow, persist the partial assistant message, run session.compact(), and re-run the turn (bounded) from the compacted history instead of dying. A no-op compaction or a spent budget surfaces terminally through onChatError with classification "context_overflow" — never a silent end, never an infinite loop. Wired into the WebSocket, chat()/RPC, and programmatic (saveMessages / submitMessages) turn paths. - contextOverflow.proactive: a { maxInputTokens, headroom? } pre-step guard. When the previous step's model-reported usage.inputTokens crosses maxInputTokens * (headroom ?? 0.9), compact in place and feed the recompacted history into the upcoming step, heading off the provider rejection before it happens. Keys off model-reported usage (every provider reports it), not provider error strings. Bounded per step loop by maxRetries (floored at 1). Adds a chat:context:compacted observability event (agents), emitted once per compaction on both the proactive and reactive paths. Notes: - Provider context-overflow errors always surface as in-stream error parts (verified against the AI SDK: streamText re-enqueues even top-level rejections as { type: "error" } fullStream parts, and toUIMessageStream passes them through without throwing), so the in-stream seam catches them on every path; the thrown-error catch path needs no separate wiring. - A one-time warning fires if contextOverflow.reactive is enabled but classifyChatError was never overridden (recovery would otherwise be a silent no-op). - Recovery effectiveness depends on the app's compaction config — a no-op compaction cannot rescue an over-budget turn (handled gracefully: terminal, not a loop). Tests cover reactive success, opt-in-off terminal, no-op-compaction terminal (via onChatError, classified), budget-bounded persistent overflow, the WebSocket and programmatic paths, the proactive guard (capped per run), and defaultContextOverflowClassifier. Docs (think/index, lifecycle-hooks, observability, sessions), the @cloudflare/think README, and the assistant example are updated. Co-authored-by: Cursor <cursoragent@cursor.com>

Three follow-ups from review: - Proactive guard baseline: `_turnModelMessageBaseline` was captured from the assembled messages BEFORE `beforeTurn` ran, so a `beforeTurn` that overrides `messages` with a different length left the baseline stale — the guard's `event.messages.slice(baseline)` tail splice could then duplicate head messages or drop steps. Capture the baseline (and the active tool set) from the FINAL messages/tools, after beforeTurn, so the splice stays correct. - Overflow retry semantics: the reactive backstop re-ran `_runInferenceLoop` with `continuation: attempt > 0`, conflating an overflow retry with an auto-continuation. An app whose `beforeTurn` gates one-time setup on `!ctx.continuation` would skip it on the retry. An overflow retry re-runs the SAME user turn from compacted history, so it now passes `continuation: false` on every attempt (all three turn paths). The flag only ever fed `TurnContext.continuation`, so this has no other effect. - classifyChatError docs/JSDoc overstated the contract: it is consulted ONLY for context-overflow recovery (when `contextOverflow.reactive` is enabled), and `ChatErrorContext.classification` is populated ONLY as `"context_overflow"` on the terminal `onChatError` of an unrecoverable overflow. Other categories are reserved (returning them is a no-op) and are NOT forwarded to onChatError. Corrected the JSDoc, the Think README, and the lifecycle-hooks doc (hook summary, classifyChatError section, and the ChatErrorContext.classification row) to match. New test asserts both attempts of a recovered turn see `beforeTurn` `continuation: false`. Full check + 502 think tests pass. Co-authored-by: Cursor <cursoragent@cursor.com>

Review follow-up. The proactive context guard reads `usage.inputTokens` and falls back to `usage.totalTokens` when a provider omits the former. Document that fallback (a safe over-approximation — compacts slightly early rather than missing the threshold; no-op when neither is reported) on the `ContextOverflowConfig.proactive` JSDoc and in the Think docs, so the `maxInputTokens` name vs. total-token comparison does not surprise users. No behavior change. (The other two review notes were already addressed: classifyChatError docs were corrected in 5c4deb2; the RPC overflow early-return clears `_streamingAssistant` via the method's existing `finally`.) Co-authored-by: Cursor <cursoragent@cursor.com>

…fectiveness Review follow-up. The overflow mock previously overflowed on call 1 and succeeded on call 2 regardless of the prompt, so the recovery tests verified the retry plumbing but not that compaction actually shortened what was sent — a retry that resent the full (uncompacted) history would still have passed. The mock now overflows while a seeded marker ("earlier question", part of the messages compaction collapses) is still present in the prompt, and only succeeds once compaction has removed it. The agent records, per model call, whether that marker was still in the prompt, and the reactive / programmatic / WebSocket recovery tests now assert `promptIncludedSeedMarker === [true, false]` — proving the retry's prompt is genuinely shorter and that the refreshed in-memory message cache (via `session.compact()`'s awaited `compact` change event → `_syncMessages()`) reaches the retry. This also guards against a regression where the cache is not refreshed (the retry would resend the marker and overflow forever). `alwaysOverflow` still forces an overflow on every call to model the "single remaining message exceeds the window" terminal case. No production code change. Full check + 509 think tests pass. Co-authored-by: Cursor <cursoragent@cursor.com>

…le + docs Review hardening of the mid-turn context-overflow recovery feature: fixes a correctness bug, decouples two coupled knobs, improves abort behavior, expands test coverage across all turn paths, and adds a runnable example verified live against Workers AI. think.ts - Discard the truncated partial assistant message on a reactive retry instead of persisting it. Persisting orphaned a cut-off bubble beside the recovered answer (and risked duplicating re-issued tool work). Regression test added. - Decouple the proactive cap from the reactive budget: maxRetries now bounds ONLY reactive compact-and-retries; add proactive.maxCompactions (default 1) for the per-turn proactive cap. The two are independent. - Skip compaction + retry when the turn is aborted: add `!abortSignal?.aborted` guards to all three reactive driver loops (WebSocket, chat()/RPC, programmatic saveMessages). An aborted turn no longer runs an expensive compaction or retry. - Thread requestId into classifyChatError's context at the overflow stream seam ({ stage: "stream", requestId }) so a classifier can correlate/cancel the in-flight turn. - Tighten defaultContextOverflowClassifier: anchor the vaguest fragment ("reduce the length" -> "reduce the length of") while keeping recall for the other provider patterns. tests (packages/think/src/tests) - New mock models + harness for: top-level doStream rejection (proves the AI SDK re-enqueues it as an in-stream error part the recovery seam catches), combined proactive+reactive recovery in one turn, and abort-during-recovery. - Abort-during-recovery tested on all three turn paths. - Strengthened the multi-fire proactive test to assert spliced-prompt structure (tool-call/result pairing intact + recompacted head present), not just a clean finish. - Assert the chat:context:compacted payload contract (reason/shortened/ requestId/attempt). - Negative + positive cases for the classifier regex change. example (examples/context-overflow-recovery) - New full-stack Think + Workers AI demo of recovery. Verified LIVE against @cf/meta/llama-3.1-8b-instruct (~7968-token window): the model rejects an oversized prompt with a classifiable error, and an overflowing turn recovers (compaction summarizes stale history, the turn completes). - The "Add background document" action inflates stored HISTORY (not the triggering message) so compaction can reclaim room — the realistic, reliable shape. Sending the bulk as the triggering message is the path where keep-last compaction can't shrink below it. docs - docs/think/index.md: proactive.maxCompactions, decoupling note, link to the example. - docs/think/lifecycle-hooks.md: document the requestId now in classifyChatError's context. - packages/think/README.md: clarify the two independent caps. - README.md: list the new example. - .changeset updated to reflect the partial-discard and decoupling.

devin-ai-integration Bot reviewed Jun 3, 2026

View reviewed changes

threepointone force-pushed the fix/think-context-overflow-recovery branch from 70c6d28 to 4bccb6e Compare June 3, 2026 08:21

threepointone and others added 5 commits June 4, 2026 14:38

threepointone force-pushed the fix/think-context-overflow-recovery branch from 4658560 to 87457d8 Compare June 4, 2026 16:16

threepointone changed the title ~~fix(think,agents): opt-in recovery for mid-turn context-window overflow~~ fix(think,agents): opt-in proactive + reactive recovery for mid-turn context-window overflow Jun 4, 2026

threepointone merged commit df6c0d6 into main Jun 5, 2026
4 of 5 checks passed

threepointone deleted the fix/think-context-overflow-recovery branch June 5, 2026 06:19

github-actions Bot mentioned this pull request Jun 5, 2026

Version Packages #1674

Merged

threepointone mentioned this pull request Jun 5, 2026

fix(chat-recovery): exempt human-in-the-loop turns from recovery budgets; close think hibernation gap #1684

Merged

Navigation Menu

Repository navigation

fix(think,agents): opt-in proactive + reactive recovery for mid-turn context-window overflow - #1662

fix(think,agents): opt-in proactive + reactive recovery for mid-turn context-window overflow#1662
threepointone merged 5 commits into
mainfrom
fix/think-context-overflow-recovery

threepointone commented Jun 3, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Jun 3, 2026 •

edited

Loading

Uh oh!

pkg-pr-new Bot commented Jun 3, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

threepointone commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's added

1. classifyChatError(error, ctx) — the classification seam

2. contextOverflow.reactive (+ maxRetries) — reactive backstop

3. contextOverflow.proactive — proactive guard (the root-cause fix)

4. Abort responsiveness

Observability

Why only the in-stream seam (investigation, now test-locked)

Edge cases / safety

Verified live against Workers AI

Test plan

Docs & example

Changeset

Notes for review

Uh oh!

changeset-bot Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

pkg-pr-new Bot commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

✅ Devin Review: No Issues Found

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

threepointone commented Jun 3, 2026 •

edited

Loading

1. `classifyChatError(error, ctx)` — the classification seam

2. `contextOverflow.reactive` (+ `maxRetries`) — reactive backstop

3. `contextOverflow.proactive` — proactive guard (the root-cause fix)

changeset-bot Bot commented Jun 3, 2026 •

edited

Loading

pkg-pr-new Bot commented Jun 3, 2026 •

edited

Loading