From 091e342af6573dcb65801bced36475002967f672 Mon Sep 17 00:00:00 2001 From: ranxianglei Date: Sun, 20 Sep 2026 23:51:53 +0800 Subject: [PATCH] fix: verify source-message availability before committing decompression (#446) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Block/range decompression could report 'Restored N message(s)' and terminally deactivate the summary block even when the indexed originals were absent from host history (removed by native compaction or external deletion): the commit path treated a membership transition as proof of restored content. Add a fail-closed pre-commit gate in createDecompressTool.execute (after the toFile branch, before any state mutation) backed by two pure functions in decompress-logic.ts: - collectRequiredDecompressMessageIds: closed-form set of message IDs whose visibility this decompression would restore — exclusively covered by the target blocks, minus one-tier shielded IDs (transitive consumed closure that sync reactivates; full mode requires everything). - checkDecompressSourceAvailability: splits that set against the fetched history IDs into available/missing. Missing sources => warn log + explicit error naming targets, counts, and missing refs; nothing is mutated, summary coverage preserved. Transient host fetch failures still reject out of prepareDecompressSession before any mutation, so the two failure classes stay structurally distinct. Because every committed ID is verified present, restored counts/stats now reflect real recovered content. Tests: 13 unit + 9 E2E regressions (all-missing / ref-display / retry-after- abort / partial / complete / nested one-tier / nested full abort+success / transient-fetch rejection). Bug-detection proof: with the gate removed the all-missing E2E fails with the issue's exact phantom output 'Decompressed block b5. Restored 1 message(s) (~100 tokens)'. Closes #446 --- .../DESIGN.md | 76 ++++ .../REQ.md | 48 ++ .../WORKLOG.md | 50 ++ lib/compress/decompress-logic.ts | 122 +++++ lib/compress/decompress.ts | 37 ++ tests/decompress-logic.test.ts | 248 ++++++++++ tests/decompress-source-availability.test.ts | 426 ++++++++++++++++++ 7 files changed, 1007 insertions(+) create mode 100644 devlog/2026-09-20_decompress-source-availability/DESIGN.md create mode 100644 devlog/2026-09-20_decompress-source-availability/REQ.md create mode 100644 devlog/2026-09-20_decompress-source-availability/WORKLOG.md create mode 100644 tests/decompress-source-availability.test.ts diff --git a/devlog/2026-09-20_decompress-source-availability/DESIGN.md b/devlog/2026-09-20_decompress-source-availability/DESIGN.md new file mode 100644 index 00000000..e449cb1b --- /dev/null +++ b/devlog/2026-09-20_decompress-source-availability/DESIGN.md @@ -0,0 +1,76 @@ +# DESIGN: source-availability gate for decompress + +## Layer analysis (triage) + +- **Symptom**: `Restored 1 message(s)` with empty preview; stats decremented for content that never returns to context. +- **Root cause**: the decompression commit path treats _membership transition_ (`activeBlockIds` → empty) as proof of _content restoration_. It never intersects the restored set with the IDs actually present in fetched host history. `computeRestoredMessages` does not even receive rawMessages. +- The issue's suggested direction (verify before commit, explicit missing reporting, fail closed) hits the root cause. Adopted. + +## Key architectural facts that shape the design + +1. A block's summary lives in its compress tool-call message in host history (`summary` param); originals are hidden from the window by `prune.filterCompressedRanges` while `entry.activeBlockIds.length > 0`. +2. Decompression = deactivate target blocks + `syncCompressionBlocks`. One-tier: consumed (non-user-deactivated) blocks are reactivated by sync, so their raw messages stay hidden and only the previous tier's compress-call messages become visible. Full: consumed blocks get `deactivatedByUserDeep` and stay dead, so ALL covered raw messages must return. +3. User-deactivated blocks are terminal (sync never reactivates them) — a blind deactivation with missing originals is irreversible. +4. `byMessageId` membership is bidirectional with `block.effectiveMessageIds` (built in `compress/state.ts`), and prune filters exactly on `entry.activeBlockIds`. So "which messages become visible" is a pure function of membership + sync reactivation semantics. + +## Design + +**Fail-closed pre-commit gate**, inserted between target resolution and the deactivation loop in `createDecompressTool.execute` (after the toFile branch, which makes no state changes). + +New pure functions in `lib/compress/decompress-logic.ts`: + +```ts +export interface DecompressAvailabilityResult { + requiredMessageIds: string[] // IDs whose visibility this decompression would restore + availableMessageIds: string[] // subset present in fetched history + missingMessageIds: string[] // subset absent — non-empty ⇒ abort +} + +export function collectRequiredDecompressMessageIds( + messagesState: PruneMessagesState, + targets: CompressionTarget[], + options: { full?: boolean }, +): string[] + +export function checkDecompressSourceAvailability( + messagesState: PruneMessagesState, + targets: CompressionTarget[], + options: { full?: boolean }, + availableMessageIds: Set, +): DecompressAvailabilityResult +``` + +### Required set R (closed form, no state mutation) + +For each `byMessageId` entry: include iff + +1. `entry.activeBlockIds.length > 0`, AND +2. every ID in `entry.activeBlockIds` belongs to the target set T (exclusively covered by what we are about to kill — otherwise another live block still hides it, and its summary remains), AND +3. one-tier only: the ID is NOT in the union of `effectiveMessageIds` of the transitive consumed closure of T whose members would be reactivated by sync (not user-deactivated / not deep-deactivated). Those messages stay hidden under reactivated lower tiers, so they are not required sources. + +Full mode skips condition 3 (everything under T must be present). + +Why closed form instead of mutate→diff→rollback: zero crash window, no rollback code path, no partial-mutation risk. Cost: it mirrors sync's reactivation rule (non-user-deactivated consumed blocks revive). Soundness argument: over-inclusion in the shield set is safe (a shielded message either stays hidden under a revived block or under another live consumer — in both cases it is still covered, so excluding it from R can never let a false "restored" claim through). Over-requirement is possible only if a target-exclusive entry is also listed in a consumed block's `effectiveMessageIds` without that block reviving — which cannot happen: any consumed block of a target either revives (one-tier, non-user-flagged) or was already dead via user flags (then its messages were already exposed, i.e. their entries do not list the target exclusively… conservatively excluded anyway). + +Note on empty `byMessageId`: when membership is unverified/empty, R = ∅ and the gate passes — consistent with existing behavior, because prune then filters nothing and `computeRestoredMessages` also reports nothing. The tool's restoration claim is exactly the membership-transition claim; verifying precisely those IDs against history closes the gap completely. + +### Gate behavior + +- `missingMessageIds.length > 0` → warn log + return an explicit error naming the target blocks, counts, and missing refs (via `state.messageIds.byRawId`, falling back to raw IDs). **No state mutation, no stat change.** Block stays compressed; summary remains available. Message also notes this is distinct from a transient fetch failure (that would have thrown in `prepareDecompressSession` before any mutation). +- Empty → proceed exactly as today. Because every ID in R is verified present, `computeRestoredMessages`' count/tokens are honest recovered-content numbers (acceptance #4 falls out for free). + +### What we deliberately do NOT do + +- No partial-commit mode (acceptance #3 default): restoring some originals while losing summary coverage semantics mid-block is worse than refusing; escape hatches (force flag / policy key) are future work, not requested here. +- No retry logic: fetch failure already surfaces as an exception before mutation; the two failure classes are structurally distinguished (acceptance #6 satisfied by construction + documented in the error text). +- toFile path untouched (#445 owns those field bugs). + +## Known limitation (from dual-agent code review) + +In one-tier nested decompress, what becomes visible after commit also includes the **reactivated consumed block's compress tool-call message** (its summary carrier), whose visibility is keyed on block liveness in `hideConsumedCompressCalls` — not on `byMessageId` membership. Compress calls are hard-protected, so carriers never have `byMessageId` entries and cannot enter R. If host history lost an intermediate tier's carrier while the topmost block is still resolvable, the gate passes, the commit proceeds, and the topmost summary is terminally discarded while nothing usable returns to context. Rare trigger (multi-tier nesting + selective deletion of exactly the intermediate carrier); pre-fix behavior was strictly worse (phantom claim + stats corruption on top). Fixing it properly means extending R with reactivating closure members' `compressMessageId`s — deliberately deferred: it changes R semantics beyond #446's stated scope (indexed originals) and needs its own review. Recorded here so the follow-up is findable; per project rules no separate issue is filed from this working session. + +## Testing strategy + +- Unit (extend `tests/decompress-logic.test.ts`): R computation for single-target, multi-target, one-tier nested (raws shielded, prev-tier compress call required), full nested (all required), unrelated overlapping active block (not required), empty membership (∅). +- Integration (new `tests/decompress-source-availability.test.ts`, reusing the `inactive-block-decompress.test.ts` harness with a controllable client history): all-missing aborts with exact message + zero state change; partial aborts listing only missing IDs; complete succeeds with correct restored count/tokens; nested one-tier succeeds with raws missing but prev-tier call present; nested full aborts; preview non-empty on success. +- Bug-detection proof: the integration all-missing test must fail against pre-fix code (it asserts the abort + no deactivation; pre-fix returns "Restored 1 message(s)" and deactivates). Verify by temporarily reverting the gate insertion. diff --git a/devlog/2026-09-20_decompress-source-availability/REQ.md b/devlog/2026-09-20_decompress-source-availability/REQ.md new file mode 100644 index 00000000..5e1a4031 --- /dev/null +++ b/devlog/2026-09-20_decompress-source-availability/REQ.md @@ -0,0 +1,48 @@ +# REQ: decompress must verify source-message availability before committing + +- **Issue**: https://github.com/ranxianglei/opencode-acp/issues/446 +- **Branch**: `2026-09-20_decompress-source-availability` +- **Date**: 2026-09-20 +- **Type**: Bug fix + +## Problem + +When an indexed original message is absent from the history returned by the host +(native compaction, external deletion, V2 checkpoint divergence), block decompression +still reports `Restored N message(s)`, deactivates the summary block, and decrements +`totalPruneTokens` — restoring visibility metadata without verifying that any content +actually returns to context. + +Verified trace on master (`f6c70a7e` era, v1.18.x): + +1. `lib/compress/decompress.ts:340` — `snapshotActiveMessages` snapshots indexed IDs, not fetched originals. +2. `lib/compress/decompress.ts:343-347` — deactivates targets + `syncCompressionBlocks`. +3. `lib/messages/sync.ts:136-146` — membership update never intersects with available history IDs. +4. `lib/compress/decompress-logic.ts:162-177` — `computeRestoredMessages` counts membership transitions only; does not accept rawMessages. +5. `lib/compress/decompress.ts:358-361,382-384` — stats adjusted + success message emitted. +6. Preview (`decompress-logic.ts:191-231`) reads real history; empty when originals missing; `decompress.ts:395-399` silently omits it. No warning. + +Repro fixture (issue #446): active b5 covers `msg-gone`; +`byMessageId["msg-gone"] = { tokenCount: 100, allBlockIds: [5], activeBlockIds: [5] }`; +history contains a different message. Observed pre-fix: +`restoredMessageCount: 1, restoredTokens: 100, actualOriginalMessagesFound: 0, restoredContentPreview: "", blockDeactivated: true`. + +## Why it matters + +- **False success**: model believes "restored" content is visible → hallucination risk about details that are not in context. +- **Stats corruption**: `totalPruneTokens` decremented by phantom tokens → `/acp context` lies. +- **Irreversible coverage loss**: user-deactivated blocks are terminal (sync keeps them dead); if the summary carrier (compress tool call) was also removed from history, the content is unrecoverable after a blind deactivation. + +## Acceptance criteria (from issue) + +1. Verify required source messages are present before committing a decompression that claims to restore originals, respecting one-tier vs full semantics. +2. Return explicit complete/partial/missing-source results with missing IDs/counts; do not label metadata-only restoration as recovered text. +3. Preserve usable summary coverage on unavailable originals (fail closed), unless an explicit policy deliberately permits partial restoration. +4. Count restored originals / adjust statistics based on verified recovered content, not only membership transitions. +5. Integration regressions: all originals missing, partial availability, complete availability, nested one-tier/full cases. +6. Distinguish transient host-fetch failures (error before any mutation) from missing/deleted originals (gate aborts, no state change). + +## Scope + +- In scope: normal non-toFile block/range decompression path (`createDecompressTool.execute`). +- Out of scope: toFile export field bugs (#445), V2 host-history projection (#425), retry logic for fetch failures (fetch failure already errors before any mutation — documented, no new mechanism). diff --git a/devlog/2026-09-20_decompress-source-availability/WORKLOG.md b/devlog/2026-09-20_decompress-source-availability/WORKLOG.md new file mode 100644 index 00000000..ed65f571 --- /dev/null +++ b/devlog/2026-09-20_decompress-source-availability/WORKLOG.md @@ -0,0 +1,50 @@ +# WORKLOG — decompress-source-availability (#446) + +Branch: `2026-09-20_decompress-source-availability` · Base: master `026dbe73` · Date: 2026-09-20 + +## Summary + +Fix for #446: block decompression reported `Restored N message(s)` and terminally +deactivated the summary block even when the indexed original messages were absent +from the host history. Root cause: the commit path treated a membership transition +(`activeBlockIds` going empty) as proof that content was restored, without ever +intersecting the required source IDs with the fetched history. + +## Changes + +| File | Change | +|------|--------| +| `lib/compress/decompress-logic.ts` | New `DecompressAvailabilityResult`, `collectRequiredDecompressMessageIds()`, `checkDecompressSourceAvailability()` | +| `lib/compress/decompress.ts` | Pre-commit availability gate in block/range decompress path (after toFile branch, before any state mutation); explicit missing-source error; warn log with refs | +| `tests/decompress-logic.test.ts` | 13 unit tests for the two new functions (tier semantics, closure, multi-target, empty membership) | +| `tests/decompress-source-availability.test.ts` | 9 E2E regressions through `createDecompressTool` with controllable host history: all-missing / ref-display / retry-after-abort / partial / complete / nested one-tier / nested full abort / nested full success / transient-fetch-failure rejection | +| `devlog/2026-09-20_decompress-source-availability/` | REQ.md + DESIGN.md + this file | + +## Verification log + +- [x] `npm run typecheck` — clean +- [x] `npm run test` — 1312/1314 pass (final state incl. post-review additions; one intermediate run showed a transient soft-block crash-count flake of `fail 3`, re-run stable at 2). The 2 failures are pre-existing environment issues, reproduced identically on clean master (verified by stashing this change): + - `tests/soft-block.test.ts` — crashes at import: `EACCES mkdir '/tmp/opencode-dcp-dangerous-*'` (read-only `/tmp` in sandbox) + - `E2E: toFile on inactive block writes block summary` — tool path validation rejects `/tmp` when `$TMPDIR` points elsewhere +- [x] `npm run build` — success (dist/index.js 488.57 KB) +- [x] Bug-detection proof (§5.7.3): gate temporarily removed → all-missing E2E test fails with `Decompressed block b5. Restored 1 message(s) (~100 tokens)` (the exact phantom-restoration behavior from the issue) → gate restored → 7/7 pass +- [x] Dual-agent review (§5.3 code + §5.6 tests) — two independent agents, both verdicts **approve-with-nits** (details below) + +## Dual-agent review findings & disposition + +| # | Finding | Severity | Disposition | +|---|---------|----------|-------------| +| C1 | One-tier nested commit does not verify the reactivated tier's summary-carrier compress message (carriers are hard-protected, never in `byMessageId`, so cannot enter R); if host history lost exactly the intermediate carrier, the topmost summary is terminally discarded with nothing usable restored | minor | **Documented as known limitation in DESIGN.md**; extending R with `compressMessageId`s changes R semantics beyond #446's stated scope and needs its own review | +| C2 | toFile path bypasses the gate by design but its success message under-counts silently when some indexed sources are absent | minor | Deferred — #445 owns the toFile field bugs; noted for that work | +| C3 | Untested branches: `deactivatedByUserDeep` exclusion from shielding; stale closure id; range-mode multi-target E2E | nit | deep-flag: covered indirectly by stricter `assert.equal(..., undefined)` on the abort path; stale-id + range-mode: unit-level coverage exists (closure tolerance, multi-target union), range E2E skipped as mode-agnostic (gate runs identically after target resolution) | +| C4 | BFS uses `queue.shift()` — O(k²) worst case | nit | No action: k = blocks under one target (small), runs once per tool call, never per LLM turn | +| T1 | No E2E pin of transient host-fetch failure (acceptance #6) | minor | **Fixed**: new test with rejecting client asserts `assert.rejects` out of the tool call + zero mutation | +| T2 | Ref-resolution display branch (`byRawId ?? raw`) untested — all fixtures had empty maps | minor | **Fixed**: new test seeds `byRawId`/`byRef` mapping, asserts `Missing: m00052.` (mappings verified to survive `assignMessageRefs`, lib/message-ids.ts:174-180) | +| T3 | Nested success tests didn't assert stats (acceptance #4) | nit | **Fixed**: one-tier success asserts `totalPruneTokens === 970` (3×10 restored), full success asserts `=== 950` (5×10 restored) | +| T4 | Weak deep-flag assertion (`!t1.deactivatedByUserDeep` passes for undefined and false) | nit | **Fixed**: `assert.equal(t1.deactivatedByUserDeep, undefined)` | +| T5 | `(+N more)` truncation untested (>10 missing IDs) | nit | Skipped: low value, string-format detail | + +## Notes + +- `npm run format` (Prettier) reformats ~350 unrelated repo files in this checkout (formatting drift vs. the committed tree). All unrelated rewrites were reverted; only the 4 PR-relevant source/test files carry changes. CI does not gate on `format:check`, so the drift is out of scope here. +- No version bump (release-branch-only rule). No state-format changes; purely additive API surface in `decompress-logic.ts`. diff --git a/lib/compress/decompress-logic.ts b/lib/compress/decompress-logic.ts index 84d2bdd9..93c2fd22 100644 --- a/lib/compress/decompress-logic.ts +++ b/lib/compress/decompress-logic.ts @@ -176,6 +176,128 @@ export function computeRestoredMessages( return { restoredMessageCount, restoredTokens } } +export interface DecompressAvailabilityResult { + /** Message IDs whose visibility this decompression would restore (membership-wise). */ + requiredMessageIds: string[] + /** Subset of requiredMessageIds present in the fetched host history. */ + availableMessageIds: string[] + /** Subset of requiredMessageIds absent from the fetched host history. + * Non-empty ⇒ committing would claim restored content that does not exist. */ + missingMessageIds: string[] +} + +/** + * Compute the message IDs whose visibility changes if `targets` are deactivated. + * + * A message becomes visible iff every block currently covering it belongs to the + * deactivation set AND (one-tier only) it is not re-covered by a consumed block + * that syncCompressionBlocks will reactivate. One-tier decompression leaves + * non-user-deactivated consumed blocks alive, so their covered messages stay + * hidden and are not required sources; full decompression deep-deactivates them, + * so everything under the targets must be present in history. + * + * The result is conservative by design: over-requiring an ID can only cause a + * safe-direction abort, never a false "restored" claim. + */ +export function collectRequiredDecompressMessageIds( + messagesState: PruneMessagesState, + targets: CompressionTarget[], + options: { full?: boolean } = {}, +): string[] { + const targetBlockIds = new Set() + for (const target of targets) { + for (const block of target.blocks) { + targetBlockIds.add(block.blockId) + } + } + if (targetBlockIds.size === 0) { + return [] + } + + // Transitive consumed closure of the target blocks. + const consumedClosure = new Set() + const queue: number[] = [] + for (const target of targets) { + for (const block of target.blocks) { + for (const id of block.consumedBlockIds ?? []) { + if (!consumedClosure.has(id)) { + queue.push(id) + } + } + } + } + while (queue.length > 0) { + const id = queue.shift()! + if (consumedClosure.has(id)) continue + consumedClosure.add(id) + const consumed = messagesState.blocksById.get(id) + if (consumed) { + for (const nested of consumed.consumedBlockIds ?? []) { + if (!consumedClosure.has(nested)) { + queue.push(nested) + } + } + } + } + + // One-tier: consumed blocks that sync will reactivate keep their messages hidden. + let shieldedMessageIds: Set | null = null + if (!options.full) { + shieldedMessageIds = new Set() + for (const id of consumedClosure) { + const block = messagesState.blocksById.get(id) + if (!block || block.deactivatedByUser || block.deactivatedByUserDeep) continue + for (const messageId of block.effectiveMessageIds ?? []) { + shieldedMessageIds.add(messageId) + } + } + } + + const required: string[] = [] + for (const [messageId, entry] of messagesState.byMessageId) { + if (entry.activeBlockIds.length === 0) continue + let exclusivelyCoveredByTargets = true + for (const blockId of entry.activeBlockIds) { + if (!targetBlockIds.has(blockId)) { + exclusivelyCoveredByTargets = false + break + } + } + if (!exclusivelyCoveredByTargets) continue + if (shieldedMessageIds?.has(messageId)) continue + required.push(messageId) + } + required.sort() + return required +} + +/** + * [Issue #446] Verify that every source message a decompression would restore is + * actually present in the fetched host history. Membership transitions alone prove + * nothing about content: if the host removed the originals (native compaction, + * external deletion), deactivating the target still reports "Restored N" while + * nothing returns to context, stats are decremented by phantom tokens, and the + * terminal user-deactivation discards the summary coverage. + */ +export function checkDecompressSourceAvailability( + messagesState: PruneMessagesState, + targets: CompressionTarget[], + options: { full?: boolean }, + presentInHistory: Set, +): DecompressAvailabilityResult { + const requiredMessageIds = collectRequiredDecompressMessageIds(messagesState, targets, options) + const available: string[] = [] + const missing: string[] = [] + for (const messageId of requiredMessageIds) { + if (presentInHistory.has(messageId)) { + available.push(messageId) + } else { + missing.push(messageId) + } + } + return { requiredMessageIds, availableMessageIds: available, missingMessageIds: missing } +} + export function computeReactivatedBlockIds( messagesState: PruneMessagesState, activeBlockIdsBefore: Set, diff --git a/lib/compress/decompress.ts b/lib/compress/decompress.ts index 39945575..14cbe22e 100644 --- a/lib/compress/decompress.ts +++ b/lib/compress/decompress.ts @@ -26,6 +26,7 @@ import { computeRestoredMessages, computeReactivatedBlockIds, buildRestoredContentPreview, + checkDecompressSourceAvailability, } from "./decompress-logic" import { formatTokenCount } from "../ui/utils" @@ -337,6 +338,42 @@ export function createDecompressTool(factoryCtx: ToolFactoryContext): ReturnType return `Block(s) ${displayIds} content (${blockMessages.length} messages, ${fileContent.length} chars) written to ${targetPath}. Block(s) stay compressed — context unchanged. Use read tool to access specific parts.` } + // [Issue #446] Fail closed when host history no longer contains the + // originals this decompression would restore. A transient fetch failure + // already threw in prepareDecompressSession; reaching here with IDs + // missing means they were deleted/compacted away — deactivating anyway + // would report phantom restorations and discard summary coverage + // irreversibly. No state is mutated before this check. + const availability = checkDecompressSourceAvailability( + messagesState, + targets, + { full: args.full === true }, + new Set(rawMessages.map((m) => m.info.id)), + ) + if (availability.missingMessageIds.length > 0) { + const displayIds = targets.map((t) => `b${t.displayId}`).join(", ") + const missingRefs = availability.missingMessageIds.map( + (id) => ctx.state.messageIds.byRawId.get(id) ?? id, + ) + const shownRefs = missingRefs.slice(0, 10).join(", ") + const extraCount = missingRefs.length - Math.min(missingRefs.length, 10) + ctx.logger.warn("Decompress aborted: source messages missing from host history", { + targetDisplayIds: displayIds, + full: args.full === true, + requiredCount: availability.requiredMessageIds.length, + missingCount: availability.missingMessageIds.length, + missingRefs, + }) + return ( + `Error: Cannot decompress ${displayIds}: ${availability.missingMessageIds.length} of ` + + `${availability.requiredMessageIds.length} source message(s) are no longer present in session ` + + `history (removed by native compaction or external deletion). Missing: ${shownRefs}` + + (extraCount > 0 ? ` (+${extraCount} more)` : "") + + `. Block(s) stay compressed — the summary remains available and nothing was changed. ` + + `This is a missing-source condition, not a transient fetch failure.` + ) + } + const activeMessagesBefore = snapshotActiveMessages(messagesState) const activeBlockIdsBefore = new Set(messagesState.activeBlockIds) diff --git a/tests/decompress-logic.test.ts b/tests/decompress-logic.test.ts index da706784..dbe430ed 100644 --- a/tests/decompress-logic.test.ts +++ b/tests/decompress-logic.test.ts @@ -11,6 +11,8 @@ import { computeRestoredMessages, computeReactivatedBlockIds, buildRestoredContentPreview, + collectRequiredDecompressMessageIds, + checkDecompressSourceAvailability, } from "../lib/compress/decompress-logic" import type { CompressionBlock, PruneMessagesState, WithParts } from "../lib/state/types" import type { CompressionTarget } from "../lib/commands/compression-targets" @@ -617,3 +619,249 @@ test("findActiveAncestorBlockId returns active ancestor for consumed inactive bl const target = makeTarget({ blocks: [consumedBlock] }) assert.equal(findActiveAncestorBlockId(ms, target), 10) }) + +// --- collectRequiredDecompressMessageIds ([Issue #446]) --- + +function nestedMessagesState(): PruneMessagesState { + const t1 = makeBlock({ + blockId: 2, + active: false, + effectiveMessageIds: ["msg-a", "msg-b"], + }) + const t2 = makeBlock({ + blockId: 5, + active: true, + consumedBlockIds: [2], + effectiveMessageIds: ["msg-a", "msg-b", "msg-c", "msg-d", "msg-e"], + }) + return makeMessagesState({ + blocksById: new Map([ + [2, t1], + [5, t2], + ]), + activeBlockIds: new Set([5]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [2, 5], activeBlockIds: [5] }], + ["msg-b", { tokenCount: 10, allBlockIds: [2, 5], activeBlockIds: [5] }], + ["msg-c", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ["msg-d", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ["msg-e", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ]), + }) +} + +test("collectRequiredDecompressMessageIds returns all covered messages for a single active block", () => { + const block = makeBlock({ blockId: 5, effectiveMessageIds: ["msg-a", "msg-b"] }) + const ms = makeMessagesState({ + blocksById: new Map([[5, block]]), + activeBlockIds: new Set([5]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ["msg-b", { tokenCount: 20, allBlockIds: [5], activeBlockIds: [5] }], + ]), + }) + const target = makeTarget({ displayId: 5, blocks: [block] }) + assert.deepEqual(collectRequiredDecompressMessageIds(ms, [target]), ["msg-a", "msg-b"]) +}) + +test("collectRequiredDecompressMessageIds excludes already-visible messages", () => { + const block = makeBlock({ blockId: 5, effectiveMessageIds: ["msg-a", "msg-c"] }) + const ms = makeMessagesState({ + blocksById: new Map([[5, block]]), + activeBlockIds: new Set([5]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + // e.g. preserved first user message — not hidden by any active block + ["msg-c", { tokenCount: 20, allBlockIds: [5], activeBlockIds: [] }], + ]), + }) + const target = makeTarget({ displayId: 5, blocks: [block] }) + assert.deepEqual(collectRequiredDecompressMessageIds(ms, [target]), ["msg-a"]) +}) + +test("collectRequiredDecompressMessageIds excludes messages also covered by non-target active blocks", () => { + const targetBlock = makeBlock({ blockId: 5, effectiveMessageIds: ["msg-a", "msg-b"] }) + const otherBlock = makeBlock({ blockId: 7, effectiveMessageIds: ["msg-b"] }) + const ms = makeMessagesState({ + blocksById: new Map([ + [5, targetBlock], + [7, otherBlock], + ]), + activeBlockIds: new Set([5, 7]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ["msg-b", { tokenCount: 20, allBlockIds: [5, 7], activeBlockIds: [5, 7] }], + ]), + }) + const target = makeTarget({ displayId: 5, blocks: [targetBlock] }) + assert.deepEqual(collectRequiredDecompressMessageIds(ms, [target]), ["msg-a"]) +}) + +test("collectRequiredDecompressMessageIds one-tier shields messages re-covered by reactivated consumed block", () => { + const ms = nestedMessagesState() + const t2 = ms.blocksById.get(5)! + const target = makeTarget({ displayId: 5, blocks: [t2] }) + // msg-a/msg-b stay hidden under reactivated b2 → not required sources + assert.deepEqual(collectRequiredDecompressMessageIds(ms, [target]), ["msg-c", "msg-d", "msg-e"]) +}) + +test("collectRequiredDecompressMessageIds full:true requires everything under the target", () => { + const ms = nestedMessagesState() + const t2 = ms.blocksById.get(5)! + const target = makeTarget({ displayId: 5, blocks: [t2] }) + assert.deepEqual(collectRequiredDecompressMessageIds(ms, [target], { full: true }), [ + "msg-a", + "msg-b", + "msg-c", + "msg-d", + "msg-e", + ]) +}) + +test("collectRequiredDecompressMessageIds one-tier shields across transitive consumed closure", () => { + const inner = makeBlock({ + blockId: 1, + active: false, + effectiveMessageIds: ["msg-a"], + }) + const mid = makeBlock({ + blockId: 2, + active: false, + consumedBlockIds: [1], + effectiveMessageIds: ["msg-a", "msg-b"], + }) + const top = makeBlock({ + blockId: 5, + active: true, + consumedBlockIds: [2], + effectiveMessageIds: ["msg-a", "msg-b", "msg-c"], + }) + const ms = makeMessagesState({ + blocksById: new Map([ + [1, inner], + [2, mid], + [5, top], + ]), + activeBlockIds: new Set([5]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [1, 2, 5], activeBlockIds: [5] }], + ["msg-b", { tokenCount: 10, allBlockIds: [2, 5], activeBlockIds: [5] }], + ["msg-c", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ]), + }) + const target = makeTarget({ displayId: 5, blocks: [top] }) + assert.deepEqual(collectRequiredDecompressMessageIds(ms, [target]), ["msg-c"]) +}) + +test("collectRequiredDecompressMessageIds does not shield under user-deactivated consumed blocks", () => { + const ms = nestedMessagesState() + // b2 was previously user-decompressed → terminal, sync will NOT reactivate it + ms.blocksById.get(2)!.deactivatedByUser = true + const t2 = ms.blocksById.get(5)! + const target = makeTarget({ displayId: 5, blocks: [t2] }) + assert.deepEqual(collectRequiredDecompressMessageIds(ms, [target]), [ + "msg-a", + "msg-b", + "msg-c", + "msg-d", + "msg-e", + ]) +}) + +test("collectRequiredDecompressMessageIds unions required IDs across multiple targets", () => { + const blockA = makeBlock({ blockId: 5, effectiveMessageIds: ["msg-a"] }) + const blockB = makeBlock({ blockId: 6, effectiveMessageIds: ["msg-b"] }) + const ms = makeMessagesState({ + blocksById: new Map([ + [5, blockA], + [6, blockB], + ]), + activeBlockIds: new Set([5, 6]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ["msg-b", { tokenCount: 20, allBlockIds: [6], activeBlockIds: [6] }], + ]), + }) + const targets = [ + makeTarget({ displayId: 5, blocks: [blockA] }), + makeTarget({ displayId: 6, blocks: [blockB] }), + ] + assert.deepEqual(collectRequiredDecompressMessageIds(ms, targets), ["msg-a", "msg-b"]) +}) + +test("collectRequiredDecompressMessageIds returns empty when membership is empty", () => { + const block = makeBlock({ blockId: 5, effectiveMessageIds: ["msg-a"] }) + const ms = makeMessagesState({ + blocksById: new Map([[5, block]]), + activeBlockIds: new Set([5]), + byMessageId: new Map(), + }) + const target = makeTarget({ displayId: 5, blocks: [block] }) + assert.deepEqual(collectRequiredDecompressMessageIds(ms, [target]), []) +}) + +// --- checkDecompressSourceAvailability ([Issue #446]) --- + +test("checkDecompressSourceAvailability reports complete availability", () => { + const block = makeBlock({ blockId: 5, effectiveMessageIds: ["msg-a", "msg-b"] }) + const ms = makeMessagesState({ + blocksById: new Map([[5, block]]), + activeBlockIds: new Set([5]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ["msg-b", { tokenCount: 20, allBlockIds: [5], activeBlockIds: [5] }], + ]), + }) + const target = makeTarget({ displayId: 5, blocks: [block] }) + const result = checkDecompressSourceAvailability( + ms, + [target], + {}, + new Set(["msg-a", "msg-b"]), + ) + assert.deepEqual(result.requiredMessageIds, ["msg-a", "msg-b"]) + assert.deepEqual(result.availableMessageIds, ["msg-a", "msg-b"]) + assert.deepEqual(result.missingMessageIds, []) +}) + +test("checkDecompressSourceAvailability splits partial availability", () => { + const block = makeBlock({ blockId: 5, effectiveMessageIds: ["msg-a", "msg-b"] }) + const ms = makeMessagesState({ + blocksById: new Map([[5, block]]), + activeBlockIds: new Set([5]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ["msg-b", { tokenCount: 20, allBlockIds: [5], activeBlockIds: [5] }], + ]), + }) + const target = makeTarget({ displayId: 5, blocks: [block] }) + const result = checkDecompressSourceAvailability(ms, [target], {}, new Set(["msg-a"])) + assert.deepEqual(result.availableMessageIds, ["msg-a"]) + assert.deepEqual(result.missingMessageIds, ["msg-b"]) +}) + +test("checkDecompressSourceAvailability reports all missing when history has none of them", () => { + const block = makeBlock({ blockId: 5, effectiveMessageIds: ["msg-a", "msg-b"] }) + const ms = makeMessagesState({ + blocksById: new Map([[5, block]]), + activeBlockIds: new Set([5]), + byMessageId: new Map([ + ["msg-a", { tokenCount: 10, allBlockIds: [5], activeBlockIds: [5] }], + ["msg-b", { tokenCount: 20, allBlockIds: [5], activeBlockIds: [5] }], + ]), + }) + const target = makeTarget({ displayId: 5, blocks: [block] }) + const result = checkDecompressSourceAvailability(ms, [target], {}, new Set(["msg-other"])) + assert.deepEqual(result.availableMessageIds, []) + assert.deepEqual(result.missingMessageIds, ["msg-a", "msg-b"]) +}) + +test("checkDecompressSourceAvailability honors one-tier vs full semantics on nested targets", () => { + const ms = nestedMessagesState() + const t2 = ms.blocksById.get(5)! + const target = makeTarget({ displayId: 5, blocks: [t2] }) + const oneTier = checkDecompressSourceAvailability(ms, [target], {}, new Set()) + assert.deepEqual(oneTier.missingMessageIds, ["msg-c", "msg-d", "msg-e"]) + const full = checkDecompressSourceAvailability(ms, [target], { full: true }, new Set()) + assert.deepEqual(full.missingMessageIds, ["msg-a", "msg-b", "msg-c", "msg-d", "msg-e"]) +}) diff --git a/tests/decompress-source-availability.test.ts b/tests/decompress-source-availability.test.ts new file mode 100644 index 00000000..5f4bf531 --- /dev/null +++ b/tests/decompress-source-availability.test.ts @@ -0,0 +1,426 @@ +import assert from "node:assert/strict" +import test from "node:test" +import { createDecompressTool } from "../lib/compress/decompress" +import type { ToolFactoryContext } from "../lib/compress/types" +import { singletonRegistry } from "./registry-stub" +import type { + CompressionBlock, + PrunedMessageEntry, + SessionState, + WithParts, +} from "../lib/state/types" + +// [Issue #446] E2E regressions for the decompress source-availability gate. +// The host history (client.session.messages response) is controllable per test, +// so these exercise the real invariant: a decompression may only commit when the +// originals it claims to restore are actually present in history. + +const SID = "session-source-availability-e2e" + +function makeMsg(id: string, role: "user" | "assistant", text: string): WithParts { + return { + info: { + id, + sessionID: SID, + role, + time: { created: 1000 }, + }, + parts: [{ type: "text", text }], + } as unknown as WithParts +} + +function makeBlock(overrides: Partial = {}): CompressionBlock { + return { + blockId: 5, + runId: 5, + active: true, + deactivatedByUser: false, + compressedTokens: 1000, + summaryTokens: 200, + durationMs: 0, + mode: "range", + topic: "test", + batchTopic: "test", + startId: "m00001", + endId: "m00003", + anchorMessageId: "anchor-1", + compressMessageId: "comp-1", + compressCallId: undefined, + includedBlockIds: [], + consumedBlockIds: [], + parentBlockIds: [], + directMessageIds: [], + directToolIds: [], + effectiveMessageIds: ["msg-a"], + effectiveToolIds: [], + createdAt: 1000, + deactivatedAt: undefined, + deactivatedByBlockId: undefined, + summary: "Compressed conversation about topic X.", + survivedCount: 0, + generation: "young", + ...overrides, + } +} + +function makeEntry( + tokenCount: number, + allBlockIds: number[], + activeBlockIds: number[], +): PrunedMessageEntry { + return { tokenCount, allBlockIds, activeBlockIds } +} + +function makeState( + blocks: CompressionBlock[], + activeIds: number[], + byMessageId: Map, +): SessionState { + const blocksById = new Map() + for (const b of blocks) blocksById.set(b.blockId, b) + return { + sessionId: SID, + isSubAgent: false, + compressPermission: "allow", + qualityGateRetryPending: false, + prune: { + tools: new Map(), + messages: { + byMessageId, + blocksById, + activeBlockIds: new Set(activeIds), + activeByAnchorMessageId: new Map(), + nextBlockId: blocks.length + 1, + nextRunId: blocks.length + 1, + markedForCleanup: new Set(), + structureVersion: 1, + lastSyncedStructureVersion: undefined, + membershipsVerified: true, + }, + }, + nudges: { + contextLimitAnchors: new Set(), + turnNudgeAnchors: new Set(), + iterationNudgeAnchors: new Set(), + lastPerMessageNudgeTurn: 0, + lastPerMessageNudgeTokens: undefined, + }, + stats: { pruneTokenCounter: 0, totalPruneTokens: 1000 }, + compressionTiming: {} as any, + toolParameters: new Map(), + subAgentResultCache: new Map(), + toolIdList: [], + messageIds: { byRawId: new Map(), byRef: new Map(), nextRef: 1 }, + lastCompaction: 0, + currentTurn: 0, + modelContextLimit: 100000, + systemPromptTokens: undefined, + } +} + +interface RunOptions { + /** Simulate a transient host-history fetch failure (rejecting client). */ + failFetch?: boolean +} + +function makeToolContext(state: SessionState, history: WithParts[], options: RunOptions = {}): ToolFactoryContext { + const noop = () => {} + return { + client: { + session: { + messages: async () => { + if (options.failFetch) { + throw new Error("host history fetch failed (transient)") + } + return { data: history } + }, + }, + } as any, + registry: singletonRegistry(state), + logger: { enabled: false, info: noop, warn: noop, error: noop, debug: noop } as any, + config: {} as any, + prompts: { reload: () => {} } as any, + } +} + +async function runDecompress( + state: SessionState, + args: Record, + history: WithParts[], + options: RunOptions = {}, +): Promise { + const ctx = makeToolContext(state, history, options) + const tool = createDecompressTool(ctx) + return await (tool.execute as any)(args, { + ask: async () => {}, + metadata: () => {}, + sessionID: SID, + }) +} + +// --- All sources missing (issue #446 repro) --- + +test("E2E: aborts when every source message is missing from host history", async () => { + const block = makeBlock({ + blockId: 5, + effectiveMessageIds: ["msg-gone"], + directMessageIds: ["msg-gone"], + }) + const state = makeState([block], [5], new Map([["msg-gone", makeEntry(100, [5], [5])]])) + + // History contains a different message; msg-gone was removed from host history. + const result = await runDecompress(state, { blockId: "b5" }, [ + makeMsg("msg-other", "user", "x"), + ]) + + assert.ok(result.startsWith("Error: Cannot decompress b5"), `unexpected result: ${result}`) + assert.match(result, /1 of 1 source message\(s\) are no longer present/) + assert.match(result, /Missing: msg-gone\./) + assert.match(result, /not a transient fetch failure/) + + // No state mutated: block stays active, membership intact, stats untouched. + assert.equal(block.active, true) + assert.equal(block.deactivatedByUser, false) + assert.deepEqual(state.prune.messages.byMessageId.get("msg-gone")!.activeBlockIds, [5]) + assert.equal(state.stats.totalPruneTokens, 1000) +}) + +test("E2E: missing-source error lists model-facing refs when ref mappings exist", async () => { + const block = makeBlock({ + blockId: 5, + effectiveMessageIds: ["msg-gone"], + directMessageIds: ["msg-gone"], + }) + const state = makeState([block], [5], new Map([["msg-gone", makeEntry(100, [5], [5])]])) + // Pre-existing mappings survive assignMessageRefs (lib/message-ids.ts:174-180), + // so the error must surface the model-facing ref rather than the raw ID. + state.messageIds.byRawId.set("msg-gone", "m00052") + state.messageIds.byRef.set("m00052", "msg-gone") + + const result = await runDecompress(state, { blockId: "b5" }, [makeMsg("msg-other", "user", "x")]) + + assert.ok(result.startsWith("Error: Cannot decompress b5"), `unexpected result: ${result}`) + assert.match(result, /Missing: m00052\./) + assert.equal(block.active, true) +}) + +test("E2E: aborted decompression does not consume the block — retry with available history succeeds", async () => { + const block = makeBlock({ + blockId: 5, + effectiveMessageIds: ["msg-a"], + directMessageIds: ["msg-a"], + }) + const state = makeState([block], [5], new Map([["msg-a", makeEntry(100, [5], [5])]])) + + const failed = await runDecompress(state, { blockId: "b5" }, [ + makeMsg("msg-other", "user", "x"), + ]) + assert.ok(failed.startsWith("Error"), `expected abort: ${failed}`) + assert.equal(block.active, true) + + const retried = await runDecompress(state, { blockId: "b5" }, [makeMsg("msg-a", "user", "a")]) + assert.ok(!retried.includes("Error"), `retry should succeed: ${retried}`) + assert.match(retried, /Restored 1 message\(s\)/) + assert.equal(block.active, false) + assert.equal(block.deactivatedByUser, true) +}) + +// --- Partial availability --- + +test("E2E: partial availability aborts and lists only the missing IDs", async () => { + const block = makeBlock({ + blockId: 5, + effectiveMessageIds: ["msg-a", "msg-b"], + directMessageIds: ["msg-a", "msg-b"], + }) + const state = makeState( + [block], + [5], + new Map([ + ["msg-a", makeEntry(100, [5], [5])], + ["msg-b", makeEntry(50, [5], [5])], + ]), + ) + + const result = await runDecompress(state, { blockId: "b5" }, [ + makeMsg("msg-a", "user", "a"), + makeMsg("msg-other", "user", "x"), + ]) + + assert.ok(result.startsWith("Error: Cannot decompress b5"), `unexpected result: ${result}`) + assert.match(result, /1 of 2 source message\(s\) are no longer present/) + const missingList = result.match(/Missing: ([^.]+)\./)?.[1] ?? "" + assert.equal(missingList, "msg-b") + + // Nothing committed despite one source being available. + assert.equal(block.active, true) + assert.deepEqual(state.prune.messages.byMessageId.get("msg-a")!.activeBlockIds, [5]) + assert.deepEqual(state.prune.messages.byMessageId.get("msg-b")!.activeBlockIds, [5]) + assert.equal(state.stats.totalPruneTokens, 1000) +}) + +// --- Complete availability (happy path still works) --- + +test("E2E: complete availability succeeds with correct restored count, stats, and preview", async () => { + const block = makeBlock({ + blockId: 5, + effectiveMessageIds: ["msg-a", "msg-b"], + directMessageIds: ["msg-a", "msg-b"], + }) + const state = makeState( + [block], + [5], + new Map([ + ["msg-a", makeEntry(100, [5], [5])], + ["msg-b", makeEntry(50, [5], [5])], + ]), + ) + + const result = await runDecompress(state, { blockId: "b5" }, [ + makeMsg("msg-a", "user", "alpha content"), + makeMsg("msg-b", "user", "beta content"), + ]) + + assert.ok(!result.includes("Error"), `should not error: ${result}`) + assert.match(result, /Decompressed block b5\. Restored 2 message\(s\)/) + assert.match(result, /RESTORED CONTENT/) + assert.match(result, /alpha content/) + assert.match(result, /beta content/) + + assert.equal(block.active, false) + assert.equal(block.deactivatedByUser, true) + assert.deepEqual(state.prune.messages.byMessageId.get("msg-a")!.activeBlockIds, []) + assert.deepEqual(state.prune.messages.byMessageId.get("msg-b")!.activeBlockIds, []) + assert.equal(state.stats.totalPruneTokens, 850) +}) + +// --- Nested blocks: one-tier semantics --- + +function nestedFixture(): { state: SessionState; t1: CompressionBlock; t2: CompressionBlock } { + const t1 = makeBlock({ + blockId: 2, + runId: 2, + active: false, + createdAt: 900, + effectiveMessageIds: ["msg-a", "msg-b"], + directMessageIds: ["msg-a", "msg-b"], + summary: "T1 summary of inner messages.", + }) + const t2 = makeBlock({ + blockId: 5, + runId: 5, + active: true, + consumedBlockIds: [2], + effectiveMessageIds: ["msg-a", "msg-b", "msg-c", "msg-d", "msg-e"], + directMessageIds: ["msg-c", "msg-d", "msg-e"], + }) + const state = makeState( + [t1, t2], + [5], + new Map([ + ["msg-a", makeEntry(10, [2, 5], [5])], + ["msg-b", makeEntry(10, [2, 5], [5])], + ["msg-c", makeEntry(10, [5], [5])], + ["msg-d", makeEntry(10, [5], [5])], + ["msg-e", makeEntry(10, [5], [5])], + ]), + ) + return { state, t1, t2 } +} + +test("E2E: one-tier nested decompress succeeds when only direct raws remain (consumed tier shielded)", async () => { + const { state, t1, t2 } = nestedFixture() + + // Host history lost the consumed tier's raws (msg-a/msg-b) but kept b5's + // direct raws. One-tier semantics reactivates b2, so msg-a/msg-b stay hidden + // under it and must NOT be required sources. + const result = await runDecompress(state, { blockId: "b5" }, [ + makeMsg("msg-c", "user", "c"), + makeMsg("msg-d", "user", "d"), + makeMsg("msg-e", "user", "e"), + ]) + + assert.ok(!result.includes("Error"), `should not error: ${result}`) + assert.match(result, /Restored 3 message\(s\)/) + assert.match(result, /Also restored nested block\(s\): b2\./) + + assert.equal(t2.active, false) + assert.equal(t2.deactivatedByUser, true) + assert.equal(t1.active, true, "consumed tier must be reactivated by sync") + assert.equal(t1.deactivatedByUser, false) + // Shielded messages remain covered by the reactivated tier. + assert.deepEqual(state.prune.messages.byMessageId.get("msg-a")!.activeBlockIds, [2]) + assert.deepEqual(state.prune.messages.byMessageId.get("msg-c")!.activeBlockIds, []) + // Stats reflect only the 3 verified restored raws (3 x 10 tokens). + assert.equal(state.stats.totalPruneTokens, 970) +}) + +test("E2E: full nested decompress aborts when consumed-tier raws are missing", async () => { + const { state, t1, t2 } = nestedFixture() + + // full:true deep-deactivates b2, so its raws become required sources. + const result = await runDecompress(state, { blockId: "b5", full: true }, [ + makeMsg("msg-c", "user", "c"), + makeMsg("msg-d", "user", "d"), + makeMsg("msg-e", "user", "e"), + ]) + + assert.ok(result.startsWith("Error: Cannot decompress b5"), `unexpected result: ${result}`) + assert.match(result, /2 of 5 source message\(s\) are no longer present/) + const missingList = result.match(/Missing: ([^.]+)\./)?.[1] ?? "" + assert.equal(missingList, "msg-a, msg-b") + + // Nothing committed: both tiers keep their liveness, no deep-deactivation. + assert.equal(t2.active, true) + assert.equal(t2.deactivatedByUser, false) + assert.equal(t1.active, false) + assert.equal(t1.deactivatedByUserDeep, undefined, "no deep-deactivation may have been committed") + assert.equal(state.stats.totalPruneTokens, 1000) +}) + +test("E2E: full nested decompress succeeds when all raws are present", async () => { + const { state, t1, t2 } = nestedFixture() + + const result = await runDecompress(state, { blockId: "b5", full: true }, [ + makeMsg("msg-a", "user", "a"), + makeMsg("msg-b", "user", "b"), + makeMsg("msg-c", "user", "c"), + makeMsg("msg-d", "user", "d"), + makeMsg("msg-e", "user", "e"), + ]) + + assert.ok(!result.includes("Error"), `should not error: ${result}`) + assert.match(result, /Restored 5 message\(s\)/) + + assert.equal(t2.active, false) + assert.equal(t2.deactivatedByUser, true) + assert.equal(t1.active, false) + assert.equal(t1.deactivatedByUserDeep, true, "full mode must deep-deactivate consumed tiers") + for (const id of ["msg-a", "msg-b", "msg-c", "msg-d", "msg-e"]) { + assert.deepEqual(state.prune.messages.byMessageId.get(id)!.activeBlockIds, []) + } + // Stats reflect all 5 verified restored raws (5 x 10 tokens). + assert.equal(state.stats.totalPruneTokens, 950) +}) + +// --- Transient fetch failure vs missing sources (acceptance criterion 6) --- + +test("E2E: transient host-fetch failure rejects before commit — distinct from missing-source abort", async () => { + const block = makeBlock({ + blockId: 5, + effectiveMessageIds: ["msg-a"], + directMessageIds: ["msg-a"], + }) + const state = makeState([block], [5], new Map([["msg-a", makeEntry(100, [5], [5])]])) + + // A throwing host fetch is a transient failure: it must reject out of the tool + // call via prepareDecompressSession — never be misclassified as a missing-source + // abort string, and never mutate state. + await assert.rejects( + runDecompress(state, { blockId: "b5" }, [makeMsg("msg-a", "user", "a")], { failFetch: true }), + /host history fetch failed/, + ) + + assert.equal(block.active, true) + assert.equal(block.deactivatedByUser, false) +})