Skip to content

fix(runtime): history compaction accepts truncated/unstructured LLM summaries without validation #3029

Closed

Description

@me2seeks

Problem

A history compaction wrote a checkpoint whose LLM summary was a 138-token free-form fragment that (a) ignored the mandated structured format, (b) ended mid-sentence, and (c) omitted all load-bearing context (which files were changed, current worktree state, test status). The continuing model then confabulated implementation details (it described a TypeScript CLI change as non-existent Rust files), which looked like foreign content leaking into the session.

Incident evidence (session fbdb3fd3, checkpoint hcheckpoint-981ceab88738d4819971747aeb292203):

  • Compaction folded 742 RuntimeEvents / ~235,503 estimated tokens into a 138-token summary (~1700:1).
  • The summary text: 确认服务端语义后,决定:…现在看 desktop 的 retry 循环结尾: — free-form, ends with a colon mid-sentence, and follows none of the SUMMARIZATION_SYSTEM_PROMPT sections (## Goal / ## Progress / ## Key Decisions / ## Next Steps / ## Critical Context).
  • The run ledger shows 3 failedOpen attempts with failOpenReason: provider_error immediately before the write that succeeded — the accepted response was produced right after repeated provider failures and is consistent with a degraded/truncated provider completion.

Root cause

history-compact-summarizer.ts + ai-sdk-compaction.ts accept any non-empty string as a checkpoint summary:

  1. The only rejection gate is empty_summary (!summary?.trim()). There is no structural validation (required sections), no minimum-size sanity check relative to the folded span, and no truncated-output detection beyond finishReason === 'length' — which does not catch providers that return partial text with a stop finish reason (observed with the session's own model serving as summarizer).
  2. The summarizer reuses the session model (resolveModel()). A weak instruction-follower can silently produce a free-form note instead of the structured checkpoint, and the pipeline records it as a valid checkpoint with decision: 'replaced', replacing 235k tokens of history.

No cross-session data is involved in this incident: buildHistoryCompactCheckpoint rejects multi-session event sets, and the checkpoint's coverage references this session's own event projection. The "foreign" content was reference material deliberately fetched into the session (a codex-rs source study requested by the user) that the thin summary failed to contextualize, leaving the continuation model to hallucinate around it.

Proposal

  1. Structural validation of summaries: require the mandated section headers (## Goal, ## Progress, ## Next Steps at minimum) and reject/flag summaries that end mid-sentence (e.g. trailing colon/open code fence). Failure → HistoryCompactSummarizerError('malformed_summary') → fail-open (keep history, retry next turn) instead of persisting the fragment.
  2. Size sanity bound: reject summaries smaller than a floor relative to the folded span (or an absolute floor such as ~200 tokens when folding >10k tokens), so a 235k→138 collapse cannot be accepted silently.
  3. Truncation robustness: treat missing/unknown finish reasons on suspiciously short output as output_length (retryable), and record the summarizer's finishReason/usage in compaction diagnostics.
  4. Deterministic facts appendix: attach model-independent anchors to the checkpoint — files touched (from tool-call args), task ledger snapshot, git worktree/branch — so the continuation has ground truth even when the prose summary is thin.
  5. Consider a dedicated (stronger or cheaper-but-reliable) summarizer model rather than always reusing the session model, or at least a retry with elevated instructions when validation fails.

Acceptance

  • A degraded provider response (short, section-less, mid-sentence) can never replace folded history; the turn fails open and retries.
  • Compaction diagnostics record summarizer finishReason, output tokens, and validation outcome.
  • Regression test: a summarizer stub returning the incident's 138-token fragment fails validation and the checkpoint is not written.

Generated-by: Maka

Activity

  1. me2seeks commented on Aug 15, 2026

    @me2seeks
    ContributorAuthor

    Reproduction confirmed

    Reproduced on three levels:

    E1 — deterministic (pipeline gate). Driving planMidTurnCapacityCompaction with a stub summarizer returning the incident's exact fragment yields decision: 'compacted' and a written checkpoint. Boundary sweep:

    REJECTED  empty string / whitespace only        (existing empty_summary gate)
    ACCEPTED  single word "done"
    ACCEPTED  incident fragment (mid-sentence colon)
    ACCEPTED  unclosed code fence
    ACCEPTED  section-less prose
    

    E2 — real model (DeepSeek deepseek-chat, real incident events as input). Two consecutive calls both returned ~160 tokens of raw tool-call markup (<||DSML||tool_calls>…) instead of the structured summary — zero required sections present — and the current gate accepts it. So a real weak-model response silently replaces folded history.

    Related defect found during reproduction (Bug B): replayPlanItemsToModelMessages in history-compact-summarizer.ts emits each parallel tool_call as its own assistant message. Strict OpenAI-compatible providers reject this: DeepSeek returns 400 "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'". The primary replay path's materializer explicitly merges a step's tool calls into one provider message (see comments in model-history.ts); the summarizer path diverges. This matches the incident run ledger: 3 provider_error fail-opens immediately before the degraded write succeeded. Filed separately as #3030.

    Generated-by: Maka

  2. added a commit that references this issue on Aug 15, 2026
  3. added a commit that references this issue on Aug 19, 2026
    170b313
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions