fix(think): keep the prompt-cache prefix stable under truncation (#2200)

.changeset/think-stable-truncation-prefix.md

@@ -0,0 +1,5 @@
+---
+"@cloudflare/think": patch
+---
+
+Keep the prompt-cache prefix stable across turns when older messages are truncated (#2200). The read-time truncation cutoff moved forward every turn, so each turn rewrote a message near the end of the cached prefix. Think now cuts at a multiple of 8 messages, keeping at least the 4 most recent messages whole and rewriting the prefix about once every 4 turns instead of every turn. Set the new `truncationStep` class field to change the step, or to `1` to restore the previous cut-every-turn behavior for models with a small context window. Media eviction and compaction were measured and already rewrite the prefix once per message or compaction.

docs/agents/sessions.md

@@ -244,6 +244,22 @@ Sessions stamps each message with a token estimate when the row is written. `com
 
 To trim a transcript before handing it to a model, `truncateOlderMessages` is exported from [`agents/chat`](./chat-agents.md), not from `agents/sessions`. If your tools define `toModelOutput`, a truncated tool output may no longer match the tool's output schema. In that case, pass `toolOutputs: false` to `truncateOlderMessages` and call `truncateOlderToolResults` on the result of `convertToModelMessages`, which truncates what the model reads after `toModelOutput` has run. Provider-executed tool outputs are never truncated.
 
+With the default `keepRecent`, the truncation cutoff moves forward by one message whenever a message is added, so every turn rewrites a message near the end of the prompt and invalidates the provider's prompt cache from that point. To keep the prefix stable, move the cutoff in steps and pass the same `keepRecent` to both functions:
+
+```ts
+const step = 8;
+const keepRecent = history.length <= 4 ? 4 : 4 + ((history.length - 4) % step);
+const truncated = truncateOlderMessages(history, {
+  keepRecent,
+  toolOutputs: false
+});
+const modelMessages = truncateOlderToolResults(
+  await convertToModelMessages(truncated, { tools }),
+  truncated,
+  { keepRecent }
+);
+```
+
 ## Large messages
 
 A message is stored as one JSON string. When that string exceeds the row budget it is cut into slices: slice 0 lives in the message row, the rest become numbered continuation rows in `cf_agents_session_message_chunks`. A read concatenates them back, so what you append is exactly what you read.

docs/think/index.md

@@ -405,6 +405,7 @@ path.
 | `getActions()`             | `{}`                             | Server actions (idempotency, approvals, authorization) compiled into tools — see [Actions](./actions.md)                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | `configureChannels()`      | `{}`                             | Per-channel policy and surfaces beyond the implicit `web` channel — see [Channels](./channels.md)                                                                                                                                                                                                                                                                                                                                                                                                                           |
 | `messengerConcurrency`     | Burst, 600 ms                    | Chat SDK concurrency strategy for every messenger on the agent. Set it as a class field — see [Messengers](./messengers.md#change-the-concurrency-strategy)                                                                                                                                                                                                                                                                                                                                                                 |
+| `truncationStep`           | `8`                              | Move the read-time truncation cutoff once every this many messages. `1` cuts every turn — see [Prompt caching](#prompt-caching)                                                                                                                                                                                                                                                                                                                                                                                             |
 | `maxSteps`                 | `10`                             | Max tool-call rounds per turn                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
 | `sendReasoning`            | `true`                           | Send reasoning chunks to chat clients                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
 | `configureSession()`       | identity                         | Configure the default session handle: compaction and search — see [Sessions](https://github.com/cloudflare/agents/blob/main/docs/agents/sessions.md)                                                                                                                                                                                                                                                                                                                                                                        |
@@ -958,6 +959,32 @@ Passes are bounded (`maxRowsPerPass`, 64 by default) and run in the background a
 
 Startup hydration reads a recent window bounded by `hydrationByteBudget` (32 MiB by default). The budget charges each row its stored bytes plus the attachment bytes it re-inflates, so it bounds isolate memory rather than the on-disk footprint.
 
+### Prompt caching
+
+Providers cache on a byte-identical prompt prefix, so a request only reads the cache up to the first byte that differs from an earlier request. Think keeps that prefix stable: the frozen system prompt is persisted, and each turn's first model request extends the previous request unless one of the context-reduction mechanisms rewrote history. Each mechanism rewrites the prefix on a bounded schedule:
+
+| Mechanism            | What it rewrites                                                                   | How often the prefix changes                |
+| -------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------- |
+| Read-time truncation | Tool outputs over 500 characters and text over 10,000 characters in older messages | Once every 8 messages (about every 4 turns) |
+| Media eviction       | An aged file part becomes a marker                                                 | Once per evicted message                    |
+| Compaction           | A span of older turns becomes one summary                                          | Once per compaction                         |
+
+Read-time truncation keeps at least the 4 most recent messages at full fidelity and cuts the rest at a multiple of 8 messages, so between cuts up to 11 recent messages stay whole. A cutoff that moved every turn would rewrite a message near the end of the prefix on every turn. In a 16-turn run with a 4,000-character tool output per turn, and cached input billed at a tenth of fresh input, that cost about twice as much as sending the untruncated history. Cutting every 8 messages brings it close to the untruncated cost while still bounding the context.
+
+The extra full-fidelity messages cost context. For a model with a small context window, set `truncationStep` to cut more often, or to `1` to cut every turn and keep only the 4 most recent messages whole:
+
+```ts
+import { Think } from "@cloudflare/think";
+
+export class SmallModelAgent extends Think<Env> {
+  truncationStep = 1;
+}
+```
+
+A compaction threshold that the compacted history still exceeds compacts on every append, which rewrites the summary every turn. Set `compactAfter()` well above the size of a summary plus the recent messages it keeps.
+
+`hydrationByteBudget` does not affect caching while the transcript fits in it. Its 32 MiB default is larger than any model's context window, so this holds unless large attachments fill the budget. Once the transcript exceeds the budget, the hydrated window starts later as messages are added, which rewrites the start of the prompt. Media eviction keeps a transcript with large attachments under the budget.
+
 ## Package Exports
 
 | Export                                  | Description                                                   |

packages/think/src/tests/agents/index.ts

@@ -51,3 +51,4 @@ export {
   ThinkMessengerRouteTestAgent
 } from "./messengers";
 export { ThinkMcpToolMaterializationAgent } from "./mcp-tool-materialization";
+export { ThinkPromptCacheTestAgent } from "./prompt-cache";

packages/think/src/tests/agents/prompt-cache.ts

@@ -0,0 +1,181 @@
+import type { LanguageModel, ToolSet, UIMessage } from "ai";
+import { tool } from "ai";
+import { z } from "zod";
+import { Think } from "../../think";
+import type { MediaEvictionConfig } from "../../think";
+
+/**
+ * Prompt-cache prefix measurement (#2200). Providers cache on a byte-identical
+ * prompt prefix, so for each turn this records how much of the first model
+ * request is a prefix of the request sent before it.
+ */
+export type PromptCacheScenario = {
+  turns: number;
+  /** Size of the `lookup` tool output called once per turn. 0 skips the tool. */
+  toolOutputChars?: number;
+  /** Size of each user message. */
+  userTextChars?: number;
+  /** Attach a data-URL image of this size to the first user message. */
+  firstTurnMediaChars?: number;
+  /** Run a Think media eviction pass after each turn. */
+  mediaEviction?: MediaEvictionConfig;
+  /** Compact every older message into one summary past this token estimate. */
+  compactAfterTokens?: number;
+  /** Override `truncationStep`. */
+  truncationStep?: number;
+};
+
+export type PromptCacheTurn = {
+  turn: number;
+  messages: number;
+  requestChars: number;
+  /** Chars of this turn's first request shared with the previous request. */
+  sharedPrefixChars: number;
+  /** Index of the first model message that differs, or null if none did. */
+  firstChangedMessage: number | null;
+  compactionCalls?: number[];
+};
+
+const v3Usage = {
+  inputTokens: { total: 10, noCache: 10, cacheRead: 0, cacheWrite: 0 },
+  outputTokens: { total: 5, text: 5, reasoning: 0 }
+};
+
+export class ThinkPromptCacheTestAgent extends Think {
+  override mediaEviction: MediaEvictionConfig | boolean = false;
+  private _requests: unknown[][] = [];
+  private _toolOutputChars = 0;
+  private _compactionCalls: number[] = [];
+
+  override getModel(): LanguageModel {
+    const requests = this._requests;
+    const useTool = () => this._toolOutputChars > 0;
+    return {
+      specificationVersion: "v3",
+      provider: "test",
+      modelId: "prompt-cache-mock",
+      supportedUrls: {},
+      doGenerate() {
+        throw new Error("doGenerate not implemented in mock");
+      },
+      doStream(options: { prompt: unknown[] }) {
+        requests.push(options.prompt);
+        const call = requests.length;
+        const last = options.prompt.at(-1) as { role?: string } | undefined;
+        const callTool = useTool() && last?.role === "user";
+        const stream = new ReadableStream({
+          start(controller) {
+            controller.enqueue({ type: "stream-start", warnings: [] });
+            if (callTool) {
+              controller.enqueue({
+                type: "tool-call",
+                toolCallId: `lookup-${call}`,
+                toolName: "lookup",
+                input: "{}"
+              });
+            } else {
+              controller.enqueue({ type: "text-start", id: `t${call}` });
+              controller.enqueue({
+                type: "text-delta",
+                id: `t${call}`,
+                delta: "Here is the answer."
+              });
+              controller.enqueue({ type: "text-end", id: `t${call}` });
+            }
+            controller.enqueue({
+              type: "finish",
+              finishReason: {
+                unified: callTool ? "tool-calls" : "stop",
+                raw: undefined
+              },
+              usage: v3Usage
+            });
+            controller.close();
+          }
+        });
+        return Promise.resolve({ stream });
+      }
+    } as LanguageModel;
+  }
+
+  override getTools(): ToolSet {
+    return {
+      lookup: tool({
+        description: "Look something up",
+        inputSchema: z.object({}),
+        execute: async () => "r".repeat(this._toolOutputChars)
+      })
+    };
+  }
+
+  async measurePromptCacheForTest(
+    scenario: PromptCacheScenario
+  ): Promise<PromptCacheTurn[]> {
+    this._toolOutputChars = scenario.toolOutputChars ?? 0;
+    if (scenario.mediaEviction) this.mediaEviction = scenario.mediaEviction;
+    if (scenario.truncationStep !== undefined) {
+      this.truncationStep = scenario.truncationStep;
+    }
+    if (scenario.compactAfterTokens !== undefined) {
+      this.session
+        .onCompaction(async (messages) => {
+          this._compactionCalls.push(messages.length);
+          const older = messages.slice(0, -4);
+          if (older.length < 2) return null;
+          return {
+            summary: `summary of ${older.length} messages`,
+            fromMessageId: older[0].id,
+            toMessageId: older[older.length - 1].id
+          };
+        })
+        .compactAfter(scenario.compactAfterTokens);
+    }
+
+    const report: PromptCacheTurn[] = [];
+    for (let turn = 0; turn < scenario.turns; turn++) {
+      const before = this._requests.length;
+      const text = `question ${turn} ${"q".repeat(scenario.userTextChars ?? 0)}`;
+      const parts: UIMessage["parts"] = [{ type: "text", text }];
+      if (turn === 0 && scenario.firstTurnMediaChars) {
+        parts.push({
+          type: "file",
+          mediaType: "image/png",
+          url: `data:image/png;base64,${"A".repeat(scenario.firstTurnMediaChars)}`
+        });
+      }
+      await this.saveMessages([{ id: `u${turn}`, role: "user", parts }]);
+      if (scenario.mediaEviction) await this._evictAgedMediaBestEffort();
+
+      const current = this._requests[before];
+      const previous = this._requests[before - 1];
+      const currentJson = current.map((message) => JSON.stringify(message));
+      const serialized = JSON.stringify(current);
+      let sharedPrefixChars = 0;
+      let firstChangedMessage: number | null = null;
+      if (previous) {
+        const previousSerialized = JSON.stringify(previous);
+        while (
+          sharedPrefixChars < serialized.length &&
+          serialized[sharedPrefixChars] ===
+            previousSerialized[sharedPrefixChars]
+        ) {
+          sharedPrefixChars++;
+        }
+        const previousJson = previous.map((message) => JSON.stringify(message));
+        const index = previousJson.findIndex(
+          (message, i) => message !== currentJson[i]
+        );
+        firstChangedMessage = index === -1 ? null : index;
+      }
+      report.push({
+        turn,
+        messages: current.length,
+        requestChars: serialized.length,
+        sharedPrefixChars,
+        firstChangedMessage,
+        compactionCalls: [...this._compactionCalls]
+      });
+    }
+    return report;
+  }
+}

packages/think/src/tests/prompt-cache.test.ts

@@ -0,0 +1,93 @@
+import { describe, expect, it } from "vitest";
+import { env } from "cloudflare:workers";
+import { getAgentByName } from "agents";
+import type {
+  PromptCacheScenario,
+  PromptCacheTurn
+} from "./agents/prompt-cache";
+
+async function measure(
+  name: string,
+  scenario: PromptCacheScenario
+): Promise<PromptCacheTurn[]> {
+  const agent = await getAgentByName(env.ThinkPromptCacheTestAgent, name);
+  return agent.measurePromptCacheForTest(scenario);
+}
+
+/** Turns whose first request did not extend the request sent before it. */
+function prefixBreaks(report: PromptCacheTurn[]): number[] {
+  return report
+    .filter((turn) => turn.firstChangedMessage !== null)
+    .map((turn) => turn.turn);
+}
+
+// Providers cache on a byte-identical prompt prefix. Each turn's first model
+// request should extend the previous request unless a context-reduction
+// mechanism rewrote history, and those rewrites should be one-shot rather
+// than happen every turn.
+describe("prompt-cache prefix stability (#2200)", () => {
+  it("extends the previous request every turn with no reduction", async () => {
+    const report = await measure("prefix-baseline", { turns: 8 });
+    expect(prefixBreaks(report)).toEqual([]);
+  });
+
+  it("rewrites truncated tool outputs once per step, not every turn", async () => {
+    const report = await measure("prefix-tool-output", {
+      turns: 16,
+      toolOutputChars: 4000
+    });
+    expect(prefixBreaks(report)).toEqual([6, 10, 14]);
+  });
+
+  it("cuts every turn when truncationStep is 1", async () => {
+    const report = await measure("prefix-step-one", {
+      turns: 8,
+      toolOutputChars: 4000,
+      truncationStep: 1
+    });
+    expect(prefixBreaks(report)).toEqual([3, 4, 5, 6, 7]);
+    expect(report.at(-1)!.requestChars).toBeLessThan(
+      (
+        await measure("prefix-step-default", {
+          turns: 8,
+          toolOutputChars: 4000
+        })
+      ).at(-1)!.requestChars
+    );
+  });
+
+  it("rewrites truncated long text once per step, not every turn", async () => {
+    const report = await measure("prefix-long-text", {
+      turns: 16,
+      userTextChars: 12_000
+    });
+    expect(prefixBreaks(report)).toEqual([6, 10, 14]);
+  });
+
+  it("rewrites the prefix once when media is evicted", async () => {
+    const report = await measure("prefix-media", {
+      turns: 8,
+      firstTurnMediaChars: 50_000,
+      mediaEviction: { keepRecentMessages: 2, minPartBytes: 10_000 }
+    });
+    expect(prefixBreaks(report)).toHaveLength(1);
+    expect(report.at(-1)!.requestChars).toBeLessThan(10_000);
+  });
+
+  it("rewrites the prefix once per compaction", async () => {
+    const report = await measure("prefix-compaction", {
+      turns: 16,
+      userTextChars: 400,
+      compactAfterTokens: 1000
+    });
+    const compactedAt = report
+      .filter(
+        (turn, i) =>
+          (turn.compactionCalls?.length ?? 0) >
+          (report[i - 1]?.compactionCalls?.length ?? 0)
+      )
+      .map((turn) => turn.turn);
+    expect(compactedAt.length).toBeGreaterThan(0);
+    expect(prefixBreaks(report)).toEqual(compactedAt);
+  });
+});