Skip to content

fix(cicd): stop benchmark.sh deleting the tracked scripts/benchmark harness - #3595

Merged
KuSh merged 1 commit into
rtk-ai:developfrom
ousamabenyounes:fix/benchmark-dir-wipes-tracked-harness
Aug 20, 2026
Merged

KuSh merged 1 commit into
rtk-ai:developfrom
ousamabenyounes:fix/benchmark-dir-wipes-tracked-harness

Conversation

@ousamabenyounes

Copy link
Copy Markdown
Contributor

Summary

  • scripts/benchmark.sh deleted the tracked scripts/benchmark/ TypeScript VM-benchmark harness from the working tree on every local run.
  • Wipe only the three gitignored output subdirectories (unix/, rtk/, diff/) instead of the whole $BENCH_DIR.

Follow-up to #3430 — @KuSh asked for this in the merge comment.

Problem

BENCH_DIR="$(pwd)/scripts/benchmark"

if [ -z "$CI" ]; then
  rm -rf "$BENCH_DIR"
  mkdir -p "$BENCH_DIR/unix" "$BENCH_DIR/rtk" "$BENCH_DIR/diff"
fi

.gitignore only ignores the three output subdirectories:

scripts/benchmark/diff/
scripts/benchmark/rtk/
scripts/benchmark/unix/

$BENCH_DIR itself is tracked and also holds the VM-benchmark harness — run.ts, cleanup.ts, rebuild.ts, lib/report.ts, lib/test.ts, lib/vm.ts, cloud-init.yaml. So rm -rf "$BENCH_DIR" removed all 7 of them.

The guard is [ -z "$CI" ], so CI never hits it (fresh checkout anyway) — it only fires locally, i.e. exactly when a contributor runs the benchmark before pushing.

Test verification (RED → GREEN)

There is no shell test harness in this repo, so this is a real observed run of the setup block against a clean checkout.

RED — on unmodified develop (ba7a9ce):

tracked files before: 7 / present on disk: 7
tracked files after:  7 / present on disk: 0
git status:
   D scripts/benchmark/cleanup.ts
   D scripts/benchmark/cloud-init.yaml
   D scripts/benchmark/lib/report.ts
   D scripts/benchmark/lib/test.ts
   D scripts/benchmark/lib/vm.ts
   D scripts/benchmark/rebuild.ts
   D scripts/benchmark/run.ts
output dirs created: scripts/benchmark/diff scripts/benchmark/rtk scripts/benchmark/unix

GREEN — with this patch:

tracked files before: 7 / present on disk: 7
tracked files after:  7 / present on disk: 7
git status:
output dirs created: scripts/benchmark/diff scripts/benchmark/rtk scripts/benchmark/unix

Stale output is still cleared — seeded a STALE.md in each of the three output dirs, ran the block again:

stale files left: 0
tracked harness intact: 7/7

Full local CI replay, CI deliberately unset so the local path is the one exercised:

job develop (ba7a9ce) this branch
test presence PASS PASS
fmt PASS PASS
clippy PASS PASS
test (--all) PASS PASS
build release PASS PASS
benchmark 34 good / 27 warn / 0 negative / 1 fail 34 good / 27 warn / 0 negative / 1 fail
tracked harness surviving the run 0/7 7/7

The benchmark's single remaining ❌ (empty output) is pre-existing and identical on develop.

Test plan

  • cargo fmt --all && cargo clippy --all-targets -- -D warnings && cargo test --all
  • ./scripts/benchmark.sh with CI unset — harness intact afterwards

…arness

`BENCH_DIR="$(pwd)/scripts/benchmark"` is a tracked directory: besides the
gitignored `unix/`, `rtk/` and `diff/` output dirs it holds the TypeScript
VM-benchmark harness (`run.ts`, `cleanup.ts`, `rebuild.ts`, `lib/*.ts`,
`cloud-init.yaml`). `rm -rf "$BENCH_DIR"` therefore wiped all 7 tracked files
from the working tree on every local run (`$CI` unset), which is exactly when
a contributor runs the benchmark before pushing.

Wipe only the three gitignored output subdirectories instead. Stale output is
still cleared between runs; the harness survives.

Follow-up to rtk-ai#3430 (rtk-ai#3430 review).

@KuSh KuSh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks

@KuSh
KuSh merged commit 3034b39 into rtk-ai:develop Aug 20, 2026
27 of 31 checks passed
@rtk-release-bot rtk-release-bot Bot mentioned this pull request Aug 20, 2026
social4hyq pushed a commit to social4hyq/homebrew-core that referenced this pull request Sep 20, 2026
rtk 0.46.0

Created-by: HarmonybrewBot
Commit-by: HarmonybrewBot
Merged-by: HarmonybrewBot
Description: Created by `brew bump`

---

Created with `brew bump-formula-pr`.<details>
  <summary>release notes</summary>
  <pre>## [0.46.0](rtk-ai/rtk@v0.45.0...v0.46.0) (2026-08-26)


### Features

- find: dispatch on find's grammar; compress find output for unmodeled predicates ([#3603](rtk-ai/rtk#3603))
- find: tee tail hint when rtk imposes the result cap ([#3603](rtk-ai/rtk#3603))

### Bug Fixes

- find: never-worse guard, recovery hint, and dispatch on find's grammar ([#3603](rtk-ai/rtk#3603))
- git: don't misdetect a value-taking option's argument as a patch flag ([#3575](rtk-ai/rtk#3575))
- cicd: stop benchmark.sh deleting the tracked scripts/benchmark harness ([#3595](rtk-ai/rtk#3595))
- tee: hash long recovery-file slugs to prevent collisions and shorten hints ([#3266](rtk-ai/rtk#3266))
- benchmark: avoid negative curl/cargo cases that fail the benchmark job ([#3430](rtk-ai/rtk#3430))
- test: accept both Ask and Allow verdicts in rewrite tests ([#3147](rtk-ai/rtk#3147)) — Closes [#3146](rtk-ai/rtk#3146)
- core: decode process output using Windows console code page ([#2717](rtk-ai/rtk#2717)) — Closes [#2452](rtk-ai/rtk#2452)
- git: preserve patch output from log commands ([#2951](rtk-ai/rtk#2951)) — Closes [#2944](rtk-ai/rtk#2944)
- discover: sanitize drive-letter colon so Windows discover finds sessions ([#2952](rtk-ai/rtk#2952)) — Closes [#2919](rtk-ai/rtk#2919)
- stream: decode lossily instead of dropping lines on invalid UTF-8 ([#2997](rtk-ai/rtk#2997)) — Closes [#2994](rtk-ai/rtk#2994)

### Other

- test(find): use the platform temp dir instead of /tmp ([#3717](https://github.com/rtk-ai/rtk/pull/3717))</pre>
  <p>View the full release notes at <a href="https://github.com/rtk-ai/rtk/releases/tag/v0.46.0">https://github.com/rtk-ai/rtk/releases/tag/v0.46.0</a>.</p>
</details>
<hr>

See merge request: Harmonybrew/homebrew-core!17826
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants