Skip to content

Part of #4366: fix(read): use window size as tracking baseline for --head-lines/--tail-lines - #4367

Closed
TechWizard9999 wants to merge 20 commits into
rtk-ai:developfrom
TechWizard9999:fix/4366-gain-head-tail-tracking
Closed

TechWizard9999 wants to merge 20 commits into
rtk-ai:developfrom
TechWizard9999:fix/4366-gain-head-tail-tracking

Conversation

@TechWizard9999

@TechWizard9999 TechWizard9999 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Part of #4366: Fix tracking baseline for head -N / tail -N rewrites in rtk read.

Changes

  1. Fast path (--head-lines N): Changed tracking baseline from cat {file} with full file size to head -N {file} with window.len() (the actual N-line window size).

  2. Byte-window path (--tail-lines or --head-lines with -n): Fixed tracking baseline from cat {file} with full file content to actual command (head -N/tail -N) with window.len().

  3. Fallback path: Updated tracking to use actual command (head -N/tail -N/cat) instead of always cat.

  4. Added regression test: test_head_tail_tracking_baseline_is_window_not_full_file uses in-memory tracker to verify the fix.

  5. Removed dead code: Removed regular_file_len function and its test (no longer used).

Fixed Regressions

Command Before (baseline) After (baseline)
head -5 big.txt cat big.txt (6673 tokens) head -5 big.txt (81 tokens)
tail -5 big.txt cat big.txt (6673 tokens) tail -5 big.txt (83 tokens)
`grep ... head -3` grep ... (7846 tokens)

Testing

  • All existing tests pass
  • New regression test test_head_tail_tracking_baseline_is_window_not_full_file added
  • Uses Tracker::new_in_memory() to avoid polluting developer's DB
  • Uses test_isolation::with_root for proper database isolation

Fixes #4366

- Add fs::create_dir_all for .claude dir in patch_settings_json_command and run_default_mode (fixes rtk-ai#2519, rtk-ai#4046 Windows regression)
- Hoist OpenCode plugin installation before Unchanged early return in run_claude_md_mode_with
- Fixes rtk-ai#4046 regressions.
The previous fix created the .claude directory unconditionally, but dry-run
mode must not create directories. Only create the directory when not in
dry-run mode.

Fixes test: test_run_default_mode_dry_run_does_not_create_missing_claude_dir
- README.md: change --opencode description to 'Claude setup + OpenCode plugin'
- supported-agents.md: clarify OpenCode installs Claude setup + plugin
- troubleshooting.md: change --opencode label to 'Claude Code + OpenCode plugin'
- main.rs: install Claude even when --opencode is passed (additive)
- mod.rs: update help text to 'Claude setup + OpenCode plugin'

Ports docs edits from rtk-ai#4052 to rtk-ai#4323.
@TechWizard9999

Copy link
Copy Markdown
Contributor Author

@KuSh — ready for review. All CI checks should pass. This fixes #4366 by fixing the tracking baseline for head/tail rewrites to use the actual N-line window size instead of the full file size. Includes regression test.

The function is used in tests but no longer in the main code path after
the tracking baseline fix for --head-lines/--tail-lines.
@rtk-wshm-sync-bot rtk-wshm-sync-bot Bot added bug Something isn't working documentation Improvements or additions to documentation read tracking windows labels Oct 1, 2026
@rtk-wshm-sync-bot

Copy link
Copy Markdown

wshm · Automated triage by AI

📊 Automated PR Analysis

🐛 Type bug-fix
🟡 Risk medium

Summary

Fixes gain token tracking for head -N/tail -N rewrites so the baseline is the actual line-window size rather than the full file (previously miscounted as a full cat), adding a regression test using an in-memory tracker. The diff also bundles unrelated fixes: making rtk init --opencode install the Claude Code setup alongside the OpenCode plugin, ensuring the Claude config directory is created on Windows, and related README/docs wording updates.

Review Checklist

  • Tests present
  • Breaking change
  • Docs updated

Linked issues: #4366, #2519, #4046


Analyzed automatically by wshm · This is an automated analysis, not a human review.

The test was using fields (original_cmd, input_tokens) that don't exist
on CommandRecord. Fixed to use available fields: rtk_cmd and saved_tokens.
- Add back NamedTempFile import (used by other tests)
- Remove unused InitContext import
- Remove unused ctx variable in test
- Remove #[expect(dead_code)] from regular_file_len (used in tests)
The test module imports Tracker and TempDir which are used in test functions,
but clippy with --all-targets incorrectly flags them as unused.
Without stdin inheritance, cmd.output() gives the child an immediate-EOF stdin,
causing curl to read an empty body for -d @-, -T -, --data-binary @-, -F field=@-, -K -.
This mirrors the existing pattern in npm_cmd.rs / wc_cmd.rs (RunOptions::inherit_stdin).

Fixes rtk-ai#4084.
The import was unused because the code used the fully qualified path
std::process::Stdio::inherit() instead of the imported Stdio.
Resolved conflict in src/cmds/system/read.rs test imports.

@KuSh KuSh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 1 of 3 — CHANGES REQUESTED

Claim: rtk read <file> --head-lines N / --tail-lines N books the N-line window, not the whole file, so gain stops claiming savings for rewritten head/tail (#4366).
Scope: accept if narrowed to "rtk read <file> --head-lines N / --tail-lines N is tracked as head -N / tail -N with the N-line window as its baseline, on every path of run()". Rebase on develop with only the read.rs commits: the curl stdin commit duplicates #4085 (and its Fixes #4084 would close that issue on merge), and the init/docs commits are already on develop through #4323. Make it Part of #4366, since the pipeline half of that issue (grep -n sample big.txt | head -3, still 7386 tokens saved) is not in this PR. #4153 changes the same track calls; this narrow fix lands first. (round 1, frozen)
Ran: develop 6f5503d, head 4c292b2 and the merge with develop × 20 rows under the hook (rtk rewrite, then the rewritten command) on the issue's 400-line file, plus 2 typed stdin rows; stdout and exit codes are identical between develop and the head on all 20 rows; invariants: under the hook, non-default targets (CRLF, empty file, .rs, N above the line count), accounting per row, every decision; savings: n/a, this PR fixes accounting, and the correct booking for a head/tail window is 0
Pins: 6 decisions, 6 mutated, 6 survived the suite (fast-path baseline, fast-path label, window-path label → cat, window-path head/tail swap, fallback label → cat, fallback head/tail swap)
Matrix: executed this round; pin check: executed this round
CI @4c292b2f: 3 failing: test (ubuntu-latest), test (macos-latest), test (windows-latest) — test_head_tail_tracking_baseline_is_window_not_full_file panics at read.rs:775 (index out of bounds: the len is 0 but the index is 0); the same panic locally

Blocks merge (2, frozen at round 1)

  1. src/cmds/system/read.rs:67 — a head/tail label over a whole-file baseline (3 track sites in run(), and only the fast path is fixed). On the byte-window path the label now says tail -N <file>, but the baseline is still &bytes, the whole file. The hook reaches this path for every tail -n N <file>.

    Command (under the hook) develop head
    head -5 big.txt cat big.txt 6673 → 82, saved 6591 head -5 big.txt 82 → 82, saved 0 ✓
    tail -n 5 big.txt cat big.txt 6673 → 84, saved 6589 tail -5 big.txt 6673 → 84, saved 6589
    tail -n 5 crlf.txt 298 → 30, saved 268 tail -5 crlf.txt, saved 268
    tail -n 5 big.rs 774 → 26, saved 748 tail -5 big.rs, saved 748

    The PR description's tail row ("83 tokens") is not what the binary books. The filtered and numbered fallback (-l aggressive, -n, typed only) has the same shape: rtk read big.txt -n --head-lines 5 books head -5 big.txt 7473 → 89, and rtk read big.rs --head-lines 5 -l aggressive books head -5 big.rs 774 → 29. That label was cat before, so this PR makes those rows contradict themselves. A fix has to satisfy these rows: every tail -n N row above books input equal to the N-line window (saved 0 at --level none), the fallback's head -N/tail -N rows book the raw N-line window as input, and cat big.txt still books 6673 → 6673.

  2. src/cmds/system/read.rs:758 — unpinned behaviour, and the PR's own test is red on CI. run() records through TimedExecution → Tracker::new(), never into the test's Tracker::new_in_memory(), so recent[0] indexes an empty list on all three runners. Even reading the right DB, rtk_cmd is always "rtk read", so contains("head") cannot pass, and CommandRecord doesn't carry original_cmd, the label this PR changes. Evidence: each of the 6 mutations above leaves the failing set unchanged (26 pass, only this test fails), so nothing pins any of them. The test must run the issue's 400-line file through run() inside test_isolation::with_root once per path (--head-lines and --tail-lines at --level none, plus a filtered or -n window) and once as a plain cat. It must read original_cmd and input_tokens back from test_isolation::db_path() and assert the label and that input equals the window's tokens (the whole file for the cat control). Each of the six mutations, and putting &bytes back as the tail baseline, must turn it red.

Optional — will not hold merge

  • regular_file_len has no production caller left. Delete it and test_regular_file_len_only_answers_for_a_regular_file rather than keeping it under #[allow(dead_code)].
  • #[allow(unused_imports)] on the whole test module hides the duplicate imports (the in-function use lines repeat the module-level ones). Drop the allow and the duplicates, and use expect("…") instead of unwrap() in the test.
  • The head / tail / cat label is built three times. On the byte-window path the head arm can never run, because the fast path returns first for every head read at --level none without -n. One label (and one baseline) computed once would keep the three paths from drifting apart again.

Filed as follow-ups

  • rtk read - --head-lines N / --tail-lines N on stdin still books all of stdin as cat - (stdin) (6673 → 84, saved 6589). It's outside this PR's hunks, and the hook rewrites none of cat f | head -5, head -5 < f or tail -n 5 < f, so only typed calls reach it. Absorbed by the existing #4366, comment added.

Checked and correct — no need to re-verify

  • The --head-lines fast path books the window on every shape tried: 82 → 82 on the issue's file, CRLF 29 → 29, .rs 22 → 22, empty file 0 → 0, and N = 500 on a 400-line file 6673 → 6673.
  • Output is unchanged: stdout bytes, md5 and exit code are identical between develop and the head on all 20 rows, and the merge with develop books the same rows as the head.
  • cat big.txt still books 6673 → 6673, and --max-lines reads keep their cat label.

Your questions

  • None asked yet.

Next

Rebase on develop with only the read.rs change, make it Part of #4366, give the byte-window and fallback paths the window as their baseline (rows in item 1), and replace the test with one that reads original_cmd and input_tokens from the test DB and that the six mutations turn red (item 2).
Rounds: 1/3. Threads: 2 open (waiting on the author), 0 resolved this round.

Comment thread src/cmds/system/read.rs
};
timer.track(
&format!("cat {}", file.display()),
&original_cmd,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocks merge 1: a tail label over a whole-file baseline. This path is what the hook reaches for every tail -n N <file>. The label is now tail -N <file>, but the baseline is still &bytes, the whole file. With the issue's file, tail -n 5 big.txt books 6673 → 84, saving 6589, the same as develop. Only the label moved from cat big.txt to tail -5 big.txt. The baseline here has to be the window (saved 0 at --level none), and the filtered/-n fallback at L128-135 needs the same rule: it now labels head -5 big.txt over 7473 → 89. The rows are in the review body, item 1.

Comment thread src/cmds/system/read.rs
/// not the full file size. When `head -5 big.txt` is rewritten to `rtk read --head-lines 5`,
/// the tracking baseline should be the 5-line window, not the full file.
#[test]
fn test_head_tail_tracking_baseline_is_window_not_full_file() -> Result<()> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocks merge 2: this test is red on all three runners and pins nothing. run() records through TimedExecution → Tracker::new(), so the Tracker::new_in_memory() read here is always empty and recent[0] panics (read.rs:775, index out of bounds). Even with the right DB, rtk_cmd is always "rtk read" and CommandRecord has no original_cmd, so the label can't be checked this way. Run run() inside test_isolation::with_root, read original_cmd and input_tokens from test_isolation::db_path(), and cover --head-lines and --tail-lines at --level none, a filtered or -n window, and a plain cat control. All six mutations listed in the body currently pass the suite; each has to turn this test red.

- Fix fast path (--head-lines): use window.len() instead of file size
- Fix byte-window path: use window.len() and correct original_cmd
- Fix fallback path: use actual command (head/tail/cat) for tracking
- Fix test to use test_isolation::with_root and verify against test DB
- Remove dead code (regular_file_len and its test)
- Clean up test imports (remove unused TempDir)
@TechWizard9999 TechWizard9999 changed the title fix(read): use window size as tracking baseline for --head-lines/--tail-lines (fixes #4366) Part of #4366: fix(read): use window size as tracking baseline for --head-lines/--tail-lines Oct 5, 2026
- Fix byte-window path (--tail-lines): use window.len() instead of full file size
- Fix fallback path: use window for tracking when head/tail lines specified
- Fix test to use test_isolation::with_root and query test DB directly
- Add rusqlite import for test
@TechWizard9999

Copy link
Copy Markdown
Contributor Author

Sorry got corrupted , Tried to trigger CI but couldnt will create an new pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working documentation Improvements or additions to documentation read tracking windows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gain: head -N / tail -N rewrites are tracked as a cat of the whole file, booking savings that don't exist

2 participants