Repository navigation
observe: find a safe reduction for retained grep output #189
Description
Pinky baseline (since v0.5.1 release boundary)
Pinky history.db records 497 observe --tool grep invocations: 1,004,003 estimated input tokens, 90,107 output tokens, 913,896 saved (about 91% savings). 425 calls emitted 100–499 tokens; 15 emitted 500–1,999; 57 emitted under 100.
Evidence gap
The database stores command labels and token counts, not output bodies. Pinkys hook audit log has no observe payloads, and the tee directory contains curl captures only. The existing OMP grep tests explicitly preserve headers and five lines for real patterns. Without representative, redacted output, reducing that context risks suppressing useful matches.
Next step
Collect or construct representative, privacy-safe grep outputs for the high-output cases; classify which retained parts are repeated headers/context versus distinct matches. Then add a fixture-backed regression and only change the compaction rule if useful context can be retained with measurable savings. Do not persist raw user outputs as telemetry.
Acceptance
- A representative fixture explains a material share of the retained tokens.
- A focused filter change preserves file identity and distinct matches.
- Token savings are measured, and real CLI/hook wiring is exercised.
- Malformed/prose output continues to pass through unchanged.
Activity
thehoff commentedon Oct 17, 2026
Historical output-shape check (2026-10-17)
A metadata-only pass over 23 recent Pinky session files found 509 grep result blocks and about 54,555 whitespace-token units in 15,852 non-empty lines. 237 blocks contained exact repeated lines, accounting for about 12,087 repeated whitespace-token units (22.16%). For blocks of at least 500 units, repeats were about 5,460/25,678 (21.26%). These are whitespace estimates, not model-token counts.
The repeated lines were not a safe deduplication target: roughly 4,898 repeated-token units were match-shaped lines, 326 were headings, and 6,863 were other/unclassified lines. Identical match text can occur at distinct file/line locations, and the unclassified portion is not understood well enough to suppress. No Observe retention behavior was changed.
The opt-in diagnostics now collect bounded duplicate-line counts and static compaction reasons without storing output text, arguments, patterns, paths, or URLs. Brain and Pinky have the final build and opt-in enabled; wait for real usage before drawing conclusions from their synthetic verification records.
thehoff commentedon Oct 17, 2026
Added a fixture-backed safety regression at
tests/fixtures/observe/duplicate_matches.txt. It drives the real CLI and verifies that repeated match text at distinct line locations remains identifiable by file header, retained line number, or the omitted-line list; context is collapsed only with the existing recovery hint. No retention behavior or deduplication rule changed.Also added
contextcrawler observe-reportso we can inspect the active and rotated diagnostics metadata while waiting for real calls. The report is described on #190.