Repository navigation
feat(git): filter large git show blob dumps with recovery and Latin-1 decoding - #3064
breisnerlopez wants to merge 3 commits into
Conversation
…tors Remove RTK-invented text that carries no signal an agent needs: the "[hint] Run `ruff check --fix` / `ruff format` / `pip install --upgrade` / `black .` / `dotnet format`" advisory lines (the fixable/formatted counts are already shown) and the decorative box-drawing separator in the go build failure header. Rubocop keeps the correctable count and drops the "run `rubocop -A`" how-to.
… decoding `git show <rev>:<path>` dumps raw file content that RTK previously passed through unfiltered (0% savings). Large text blobs are now capped to an 8 KiB byte budget with a `tail -n +N` recovery pointer (the full blob is tee'd), mirroring the commit-diff path. Small blobs, tree listings, binary, and blobs too large for the recovery file to store intact are passed through unchanged. Blobs are captured as raw bytes and decoded encoding-aware: ISO-8859 / Latin-1 files (e.g. Oracle PL/SQL packages) are transcoded losslessly instead of being corrupted into U+FFFD by lossy UTF-8 decoding, which also unblocks their compression. Ambiguous single-byte encodings (CP1252 range) and binary content are passed through raw rather than guessed.
|
|
…rten hints The tee recovery filename embeds a command-derived slug (often a file path that duplicates the command the LLM already issued), costing ~12 tokens in every truncation hint. Long slugs (>24 chars) now collapse to a short readable prefix plus a 6-hex SHA-256 tag. This also fixes a latent collision in the old 40-char truncation: sibling paths sharing a long common prefix (e.g. several `git show` blobs from the same directory in the same second) truncated to an identical filename and overwrote each other's recovery file. Hashing the full slug makes distinct commands produce distinct filenames (~1-in-16M collision).
|
Gentle nudge for maintainer review when someone has a moment — no rush. Status: targets The load-bearing change is the encoding fix: On the single-focus rule in CONTRIBUTING: the PR also carries a |
|
Closing in favor of smaller, focused PRs (per the single-focus PR guideline in CONTRIBUTING) rebased onto current
The subjective advisory-hints refactor that was bundled here has been dropped. Splitting this way should make review much easier. Thanks! |
2 participants
Summary
feat(git):git show <rev>:<path>blob dumps (previously passed through at 0% savings) are now filtered — large text blobs are capped to an 8 KiB preview with atail -n +Nrecovery pointer (the full blob is tee'd); small blobs, tree listings, binary, and blobs too large to recover intact pass through unchanged..pck) are transcoded losslessly instead of being corrupted into U+FFFD by lossy UTF-8, which also unblocks their compression. Ambiguous single-byte encodings (CP1252) and binary pass through raw. Measured on a real 77 KB Latin-1 blob: 0% → 89% savings with theÉ→�corruption removed. (Same class of fix asrtk curl http://....tar.gz | tar -xzffails due rtk truncation, eg "(153 more lines, 94292 bytes total)" #1087.)fix(tee): recovery-file slugs (an embedded path duplicating the command) are hashed when long, saving ~12 tokens per truncation hint and fixing a latent collision where sibling paths in the same second overwrote each other's recovery file.refactor(output): dropped RTK-invented advisory hints ([hint] Run ruff/pip/black/dotnet/rubocop…) and the go-build decorative separator — the counts they restate are already shown, so no signal is lost.Test plan
cargo fmt --all && cargo clippy --all-targets && cargo test— clippy 0 warnings; 2449 tests pass. (The one failing test,git_checkout_dirty_tree_error_keeps_file_list, is pre-existing and locale-dependent — it also fails on a cleandevelop, unrelated to this PR.)rtk git show HEAD:src/main.rs→ 120 KB → 8 KB,tail -n +Nrecovery reconstructs byte-for-bytertk git show HEAD:<latin1.pck>→ 89% savings, no�, recovery byte-identical toiconv -f ISO-8859-1rtk git show HEAD:<cp1252.pck>→ byte-exact passthrough (no wrong transcode)rtk git show HEAD:<dir>(tree) andHEAD:<binary>→ passthrough unchangedNew unit + fixture tests cover decode (UTF-8 / Latin-1 / CP1252 / binary), byte-cap windowing, recovery-offset math, slug hashing/collision, and edge cases.