Skip to content

CLI mutates historical tool results via cch= billing hash substitution, permanently breaking prompt cache #40652

Description

@eumemic

Summary

Certain Claude Code sessions permanently lose prompt cache hits mid-conversation. Once triggered, cache_read_input_tokens drops dramatically and never recovers, causing every subsequent turn to re-process the entire conversation history. For long sessions this wastes 30-50K+ tokens per turn.

Root Cause Theory

The CLI performs a find-and-replace of cch=XXXXX billing hash values across all message content (including stored historical tool results) before each API call. Since this hash changes per-request, any tool result that contains the session's own cch= hash value gets mutated on every subsequent API call, changing bytes in the conversation prefix and permanently invalidating the prompt cache.

Evidence

1. Confirmed cache breakage pattern

Multiple sessions observed with this pattern:

  • Healthy: cache_read_input_tokens grows steadily, input_tokens stays at 1-3
  • Broken: cache_read drops to ~15K (system prompt only), cache_creation jumps to 30K+ every turn
  • Once broken, never self-heals — every subsequent turn re-creates the cache

2. Diff of consecutive API requests from broken session

A prior investigation set up a local proxy (via ANTHROPIC_BASE_URL) to capture raw API request bodies. Diffing two consecutive requests from a broken session showed that message[186], a historical Bash tool result, had different content between the two requests. The diff was in an x-anthropic-billing-header value embedded in the tool result:

Request 1: cch=14f72
Request 2: cch=59b51

The tool result contained proxy log output that incidentally captured billing headers. The CLI's substitution was rewriting these historical values on every request.

3. Live reproduction (this session)

This investigation session (e9212a5a) ran grep on an infected session's JSONL to count cch= patterns:

  14 cch=80528
   6 cch=59b51
   6 cch=14f72

This grep output landed in a Bash tool result. The session's cache broke immediately on the next turn:

Turn cache_read cache_create Notes
48 42,668 1,854 Healthy
49 15,559 29,408 BROKEN — cache_read dropped 27K
50+ 15,559 30K-35K Permanently broken

4. Substitution is session-specific

We attempted to infect a separate session by putting cch=59b51 and cch=14f72 (the hashes from the broken session) into its tool results. Its cache did not break. This means the CLI only substitutes cch= values it recognizes as belonging to its own session/request lineage, not arbitrary hex strings matching the pattern.

5. Synthetic values don't trigger it

Putting cch=a1b2c or cch=a1b2c3d4e5 into tool results via file reads also did not break caching. Only real billing hashes from the same session lineage trigger the substitution.

What We Don't Know

  1. Exact substitution logic — Is it tied to the OAuth token? The session ID? A per-process value? We couldn't inspect the CLI source to find the regex/replacement logic.
  2. Where in the CLI this happens — The substitution occurs somewhere between reading the session JSONL and sending the API request. We haven't located the code path.
  3. Whether this is intentional — The substitution may be a security measure to avoid leaking billing attribution tokens in logged content, but the side effect of mutating historical messages is catastrophic for caching.

Reproduction Steps

  1. Start a Claude Code session, verify cache is healthy (cache_read growing, input_tokens = 1-3)
  2. Run a command that captures the session's own raw API traffic (e.g., proxy via ANTHROPIC_BASE_URL that logs request headers)
  3. The proxy output will contain cch=XXXXX in the billing header
  4. This output lands in a tool result in the session JSONL
  5. On the next turn, the CLI rewrites the cch= value in that historical tool result → prefix changes → cache invalidated
  6. Every subsequent turn: the value gets rewritten again → permanent cache miss

Key insight: The session doesn't need to intentionally capture billing headers. Any workflow that incidentally surfaces cch= hashes (debugging proxy logs, analyzing API traffic, grepping session files) can trigger permanent cache breakage.

Impact

  • Token burn: A broken 50K-context session wastes ~50K tokens per turn instead of ~3. Over 50 turns, that's 2.5M wasted tokens.
  • Contagious: Investigating a broken session (reading its JSONL, grepping for patterns) can infect the investigating session — the real hashes propagate through tool results.
  • Silent: No error, no warning. The session just silently burns tokens at 10,000x the normal rate.

Suggested Fix

The cch= substitution should not modify content inside tool_result blocks in the conversation history. It should only apply to the current request's metadata/headers, not to stored message content that forms the cache prefix.

Alternatively, the substitution should be scoped to specific fields (e.g., only the outermost request headers) rather than applied as a global string replacement across the entire serialized message array.

Activity

  1. github-actions commented on Mar 29, 2026

    @github-actions

    Found 2 possible duplicate issues:

    1. [BUG] Conversation history invalidated on subsequent turns #40524
    2. [BUG] Prompt cache regression in --print --resume since v2.1.69(?): cache_read never grows, ~20x cost increase #34629

    This issue will be automatically closed as a duplicate in 3 days.

    • If your issue is a duplicate, please close it and 👍 the existing issue instead
    • To prevent auto-closure, add a comment or 👎 this comment

    🤖 Generated with Claude Code

  2. jmarianski commented on Mar 29, 2026

    @jmarianski

    Same, man. Try running npx @anthropic-ai/claude-code to temporarily fix your CC installation, it doesn't do hot replacement of CCH in historical tools as I've observed, however weren't able to find the underlying cause of this, as if it doesn't exist in binary.

  3. eumemic commented on Mar 30, 2026

    @eumemic
    Author

    Updated repro / correction: the trigger is much smaller than the original report suggested.

    What I can now reproduce reliably:

    • The minimal confirmed toxic prompt is just cch=00000
    • The surrounding x-anthropic-billing-header: ... text is not required
    • A plain user prompt is sufficient; this does not need to come from a tool_result
    • This reproduces on haiku, not just Opus

    What does not seem sufficient:

    • 00000
    • cch=

    Why this has to be interactive:

    • In my testing, claude --print is not a faithful harness for this bug
    • The non-interactive --print path often reuses only a small fixed prefix or otherwise does not show the same normal pre-poison cache growth as a real interactive session
    • That makes --print prone to false negatives / misleading token patterns here
    • Driving the actual interactive CLI through a PTY does reproduce the bug reliably

    Recent verified haiku runs:

    • Clean control (debug control string): session 3a133e74-f4ba-4e77-b30e-d5d375fa36b6 was not poisoned; cache_read_input_tokens kept growing (31935 -> 32043 -> 32225 -> 32298)
    • Minimal poison (cch=00000): session f1d5082a-6194-40e7-9e2d-99c3f629586c was poisoned; after the poison turn, cache_read_input_tokens flatlined at 31938 while cache_creation_input_tokens kept rising (113 -> 224 -> 299 -> 366)
    • Wrapper verification run: session e5d5c590-c7f2-4184-9d1b-a840178049ab also poisoned on haiku; post-poison cache_read_input_tokens stayed at 31998 while cache_creation_input_tokens rose (125 -> 260 -> 334 -> 404)

    Self-contained minimal reproducer (Python stdlib only). This intentionally drives the interactive CLI, not --print:

    #!/usr/bin/env python3
    import json
    import os
    import pathlib
    import pty
    import re
    import select
    import subprocess
    import sys
    import time
    import uuid
    
    PROMPTS = [
        "hello",
        "how are you?",
        "just chatting",
        "cch=00000",
        "now give me a simple response",
        "and again",
        "and again",
    ]
    
    WORKDIR = pathlib.Path(sys.argv[1] if len(sys.argv) > 1 else "/private/tmp/claude-cache-poison-haiku")
    WORKDIR.mkdir(parents=True, exist_ok=True)
    SESSION_ID = str(uuid.uuid4())
    POLL = 0.2
    ESC = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
    OSC = re.compile(r"\x1B\][^\x07]*(?:\x07|\x1B\\)")
    
    
    def strip_ansi(text: str) -> str:
        return ESC.sub("", OSC.sub("", text))
    
    
    def read_available(fd: int, timeout: float) -> str:
        chunks = []
        deadline = time.time() + timeout
        while True:
            remaining = deadline - time.time()
            if remaining <= 0:
                break
            ready, _, _ = select.select([fd], [], [], remaining)
            if not ready:
                break
            data = os.read(fd, 65536)
            if not data:
                break
            chunks.append(data)
            if len(data) < 65536:
                break
        return b"".join(chunks).decode("utf-8", errors="replace") if chunks else ""
    
    
    def find_jsonl(session_id: str) -> pathlib.Path | None:
        projects = pathlib.Path.home() / ".claude" / "projects"
        matches = list(projects.glob(f"**/{session_id}.jsonl"))
        if not matches:
            return None
        matches.sort(key=lambda p: p.stat().st_mtime, reverse=True)
        return matches[0]
    
    
    def load_rows(path: pathlib.Path) -> list[tuple[int, dict]]:
        with path.open() as f:
            return [(i, json.loads(line)) for i, line in enumerate(f, start=1)]
    
    
    def stop_hook_count(rows) -> int:
        return sum(
            1
            for _, row in rows
            if row.get("type") == "system" and row.get("subtype") == "stop_hook_summary"
        )
    
    
    def last_assistant_usage_after(rows, line_no: int):
        last = None
        for current_line, row in rows:
            if current_line <= line_no or row.get("type") != "assistant":
                continue
            usage = row.get("message", {}).get("usage")
            if usage:
                last = {
                    "line_no": current_line,
                    "timestamp": row.get("timestamp"),
                    "cache_read_input_tokens": usage.get("cache_read_input_tokens", 0),
                    "cache_creation_input_tokens": usage.get("cache_creation_input_tokens", 0),
                    "input_tokens": usage.get("input_tokens", 0),
                    "output_tokens": usage.get("output_tokens", 0),
                }
        return last
    
    
    def wait_until_ready(fd: int, timeout: float = 60.0) -> None:
        deadline = time.time() + timeout
        buf = ""
        while time.time() < deadline:
            chunk = read_available(fd, POLL)
            if chunk:
                buf += strip_ansi(chunk)
                if "1. Yes, I trust this folder" in buf:
                    os.write(fd, b"\r")
                    buf = ""
                if "❯" in buf:
                    time.sleep(0.5)
                    return
        raise RuntimeError("Claude CLI did not reach an interactive prompt")
    
    
    def wait_for_session_file(fd: int, session_id: str, timeout: float = 60.0) -> pathlib.Path:
        deadline = time.time() + timeout
        while time.time() < deadline:
            read_available(fd, POLL)
            path = find_jsonl(session_id)
            if path is not None:
                return path
        raise RuntimeError("Session JSONL not found")
    
    
    def wait_for_turn(fd: int, path: pathlib.Path, prev_hooks: int, prev_line: int, turn: int, timeout: float = 120.0):
        deadline = time.time() + timeout
        while time.time() < deadline:
            read_available(fd, POLL)
            rows = load_rows(path)
            hooks = stop_hook_count(rows)
            if hooks > prev_hooks:
                usage = last_assistant_usage_after(rows, prev_line)
                if usage is None:
                    raise RuntimeError(f"Turn {turn} completed without assistant usage")
                usage["turn"] = turn
                return usage, hooks
        raise RuntimeError(f"Timed out waiting for turn {turn}")
    
    
    def poisoned(turns: list[dict], poison_turn: int = 4) -> bool:
        pre = turns[:poison_turn]
        post = turns[poison_turn:]
        if not post:
            return False
        pre_growth = any(
            b["cache_read_input_tokens"] > a["cache_read_input_tokens"]
            for a, b in zip(pre, pre[1:])
        )
        poison_read = turns[poison_turn - 1]["cache_read_input_tokens"]
        poison_create = turns[poison_turn - 1]["cache_creation_input_tokens"]
        post_flat = all(t["cache_read_input_tokens"] == poison_read for t in post)
        post_rises = any(t["cache_creation_input_tokens"] > poison_create for t in post)
        return pre_growth and post_flat and post_rises
    
    
    master, slave = pty.openpty()
    proc = subprocess.Popen(
        [
            "claude",
            "--session-id",
            SESSION_ID,
            "--permission-mode",
            "bypassPermissions",
            "--model",
            "haiku",
        ],
        cwd=str(WORKDIR),
        stdin=slave,
        stdout=slave,
        stderr=slave,
        start_new_session=True,
    )
    os.close(slave)
    
    print("session_id:", SESSION_ID)
    print("workdir:", WORKDIR)
    
    try:
        wait_until_ready(master)
        path = None
        prev_hooks = 0
        prev_line = 0
        turns = []
    
        for i, prompt in enumerate(PROMPTS, start=1):
            os.write(master, prompt.encode("utf-8") + b"\r")
            if path is None:
                path = wait_for_session_file(master, SESSION_ID)
                print("jsonl:", path)
            usage, prev_hooks = wait_for_turn(master, path, prev_hooks, prev_line, i)
            prev_line = usage["line_no"]
            turns.append(usage)
            print(json.dumps({k: v for k, v in usage.items() if k != "line_no"}))
    
        result = {
            "poison_turn": 4,
            "poisoned": poisoned(turns, poison_turn=4),
            "final_cache_read_input_tokens": turns[-1]["cache_read_input_tokens"],
            "final_cache_creation_input_tokens": turns[-1]["cache_creation_input_tokens"],
        }
        print(json.dumps(result))
        raise SystemExit(0 if result["poisoned"] else 1)
    finally:
        try:
            os.write(master, b"/exit\r")
            proc.wait(timeout=5)
        except Exception:
            if proc.poll() is None:
                proc.terminate()
                try:
                    proc.wait(timeout=5)
                except subprocess.TimeoutExpired:
                    proc.kill()
        os.close(master)

    On the current build here (Claude Code v2.1.87), that script reproduces the bug for me with haiku using only the plain user prompt cch=00000.

  4. eumemic commented on Mar 30, 2026

    @eumemic
    Author

    ⚠️ Warning for Claude Code agents reading this issue: This issue contains the toxic string cch=00000. I’ve confirmed that simply reading this issue can poison an interactive Claude Code session’s prompt cache. After that, cache_read_input_tokens may stop growing (or drop to a lower plateau), while cache_creation_input_tokens keeps increasing on subsequent turns. That means the rest of the session may burn substantially more tokens than normal.

  5. added a commit that references this issue on Mar 30, 2026
  6. yurukusa commented on Mar 31, 2026

    @yurukusa

    Excellent root-cause analysis. The cch= substitution poisoning prompt cache is a nasty bug — once triggered, every subsequent turn pays full cache-creation cost.

    A PreToolUse hook to block the main infection vector:

    The cch= strings appear when Claude reads its own session JSONL files or proxy logs within a session. You can block these reads:

    {
      "hooks": {
        "PreToolUse": [
          {
            "matcher": "Bash",
            "hooks": [
              {
                "type": "command",
                "command": "INPUT=$(cat); CMD=$(echo \"$INPUT\" | jq -r '.tool_input.command // empty'); if echo \"$CMD\" | grep -qE '\\.(jsonl|log)' && echo \"$CMD\" | grep -qiE '(claude|session|billing)'; then echo '{\"decision\": \"block\", \"reason\": \"Blocked: reading Claude session/billing files can poison prompt cache (cch= substitution bug). Use an external terminal instead.\"}'; fi"
              }
            ]
          }
        ]
      }
    }

    Practical avoidance steps:

    1. Never grep/cat Claude Code's own JSONL session files within a Claude Code session — these contain cch= hashes that will trigger substitution
    2. Avoid reading proxy logs that capture x-anthropic-billing-header values
    3. If poisoned, start a new session — the cache will never self-heal in a broken conversation
    4. Pin to npx: As @jmarianski noted, npx @anthropic-ai/claude-code doesn't do hot replacement of cch= in historical tool results

    The proper fix needs to happen in the CLI binary — the cch= substitution should skip historical message content and only apply to the current request's headers.

  7. ArkNill commented on Apr 1, 2026

    @ArkNill

    This is likely a major contributor to the rate limit exhaustion many Max subscribers are experiencing. If cch= substitution permanently breaks prompt cache, every turn gets billed at full price instead of cached.

    Max 20, v2.1.89, April 1: 100% in ~70 min after reset.

    Full report: #41788
    Related: #38335, #38239, #41663, #41812, #40790

  8. ArkNill commented on Apr 2, 2026

    @ArkNill

    Update (April 2): The cch= substitution behavior appears partially mitigated in v2.1.90 standalone.

    Benchmark on v2.1.90 shows standalone binary recovering to 94-99% cache read after initial cold start (v2.1.89 never recovered, sustained 4-17%). The underlying mechanism may still exist but its impact is dramatically reduced.

    npm installation remains unaffected by design. Full comparison: BENCHMARK.md

  9. ArkNill commented on Apr 3, 2026

    @ArkNill

    April 3 update: The cache regression (Bugs 1-2) is fixed in v2.1.91. However, systematic proxy testing revealed additional unfixed mechanisms — a 200K tool result budget cap, a client-side false rate limiter (151 synthetic entries found), and silent microcompact clearing (327 events). Anthropic responded on X (Lydia Hallie) acknowledging peak-hour tightening but stating "none were over-charging you" — our measured data shows mechanisms their statement does not cover. Full analysis: claude-code-cache-analysis

  10. weilhalt commented on Apr 3, 2026

    @weilhalt

    The "none were over-charging you" statement is hard to reconcile with what we're all measuring independently.

    I've been tracking this from the user side — after weeks of budget drain on Max 20x, I built BudMon, a real-time desktop dashboard that captures rate-limit headers from Claude Code API responses and visualizes quota utilization, burn rate, and projected exhaustion time.

    What BudMon consistently showed before v2.1.91:

    • Budget burn rates of 15-25% per hour during light work (read → edit → test cycles, no agents)
    • 100% exhaustion in under 2 hours on sessions that previously lasted a full workday

    I'll run fresh measurements on v2.1.91 to see if the cache fix changes the burn rate profile. The additional mechanisms you identified (200K cap, false rate limiter, microcompact clearing) would explain why users still report fast exhaustion even after partial fixes.

    Related: #42052 (my original report with reproduction data)

  11. jmarianski commented on Apr 3, 2026

    @jmarianski

    April 3 update: The cache regression (Bugs 1-2) is fixed in v2.1.91.

    Were you able to confirm it? I could not unfortunately :( Perhaps my testing methodology is broken.

  12. weilhalt commented on Apr 6, 2026

    @weilhalt

    The refined repro by @eumemic is very valuable — cch=00000 as minimal toxic prompt makes this highly actionable.

    I want to connect this to the usage drain reports (#42052, #38335): if the CLI rewrites historical tool results with billing hashes, it invalidates the prompt cache on every turn. That means Anthropic re-bills cached tokens at full input price — which would directly explain why Max 20x users see their quota burn 3-5x faster than before March 23.

    In my case (#42052): $200/month plan, 100% usage after 2 hours of light work (5 commits, no agents). If prompt cache is silently broken, the math checks out.

    This might be the root cause behind the entire wave of usage complaints. Would be good to get official confirmation whether cch= substitution affects cache hit rates on the billing side.

  13. added a commit that references this issue on Apr 22, 2026
  14. junaidtitan commented on May 4, 2026

    @junaidtitan

    Cache hash mutation permanently breaking prompt cache is a sneaky source of token waste — 30-50K extra tokens per turn adds up fast. Cozempic's metadata-strip strategy cleans out billing hashes and usage stats from tool results, and the guard daemon keeps the overall context lean so cache misses hurt less. pip install cozempic https://github.com/Ruya-AI/cozempic — happy to hear how it goes.

  15. github-actions commented on Jun 1, 2026

    @github-actions

    Closing for now — inactive for too long. Please open a new issue if this is still relevant.

  16. github-actions commented on Aug 4, 2026

    @github-actions

    This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.

  17. locked as resolved and limited conversation to collaborators on Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:corearea:costbugSomething isn't workinghas reproHas detailed reproduction stepsstaleIssue is inactive

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions