test-perf: diagnose and cut the wall clock of the local test and gate corpus, especially on Windows Git Bash

[kyle-sexton]

Summary

Running the local test suites and the pre-PR gate scripts takes long enough on a Windows Git Bash host that a single phase close-out spends most of its wall clock waiting on them, and several individual scripts exceed the Claude Code Bash tool's two-minute default timeout on their own. The repository already documents the cause in scripts/affected-tests.sh: "The full corpus is tens of minutes of wall clock on a Windows box (Git Bash pays ~140ms per process spawn, and these suites are spawn-bound), which is long enough that nobody runs it locally and regressions reach CI." This issue asks for a measured program to troubleshoot, diagnose, and optimize test and gate performance so the corpus is as fast as it can be, with Windows as the primary target.

Observed (2026-09-04, Windows 11, Git Bash, worktree on the feat/handoff-prompt-qol branch)

• Corpus size: 354 *.test.sh suites plus 42 Python test files under plugins/.
• scripts/check-purged-em-dashes.sh exceeded the 120-second tool default on its first run (killed, exit 143) while scanning 105 files; it passed on a rerun under a 10-minute timeout.
• scripts/check-changed-skills.sh origin/main exceeds 120 seconds for six touched skills and has to be backgrounded.
• scripts/affected-tests.sh --run on a 56-file diff takes several minutes per run; two runs were needed this session.
• plugins/session-flow/scripts/save_point.test.sh: 52 pytest cases take 10 to 14 seconds; plugins/session-flow/skills/retro/scripts/parse-transcript.test.sh: 38 cases take 2.6 to 5.4 seconds. Both are dominated by interpreter startup and per-case subprocess spawns, not by the assertions.
• scripts/run-plugin-tests.sh keeps --jobs 1 as the default on Windows because a parallel run measured sublinear (the box saturates on process creation), and tool-honesty and jira conformance suites fail under run-plugin-tests --jobs 4, pass serially #3694 shows two suites fail under --jobs 4.
• Side observation while diagnosing: with a merge in progress in the worktree, affected-tests.sh resolves its diff base to the fork point rather than the merge target, so files changed only on origin/main are reported as unmapped and the run exits 1 before executing any suite. Worth a guard or a clearer message.

Related

• Hook performance program: measure and cut every runtime-cost surface on the Windows host #3623 (hook performance program on the Windows host) and perf(hooks): Windows process-creation tax makes the hook set unusable - 1,778 timeouts / 20.6 hours blocked over 9 days #3508 (Windows process-creation tax on hooks): the same root cost, on the hook surface rather than the test surface.
• perf(host): melo-desk-001 process creation costs 1.4 to 4 s system-wide; 3.81M leaked kernel Token objects (~10 GB paged pool) since boot #3715 (host-level process creation of 1.4 to 4 seconds system-wide): measurements for this issue should be taken on a host that does not carry that condition, or at least record it, so the baseline reflects the scripts rather than the machine.
• ci: push-to-main wall time rose from 6 to 21 minutes after the six-job consolidation; parallelise test-linux on the push path #3705 (CI push-to-main wall time) and tool-honesty and jira conformance suites fail under run-plugin-tests --jobs 4, pass serially #3694 (suites that fail under --jobs 4): the CI-side and parallelism-side halves of the same problem.
• Five suites fail only on a Windows Git Bash host; give them the host-skip pattern or fix them #3683 (five suites fail only on Windows Git Bash).

Proposed work

1. Instrument first. Add a timing mode to scripts/run-plugin-tests.sh and scripts/affected-tests.sh that records per-suite wall clock and a process-spawn count (a DEBUG trap or bash -x line count is enough on Git Bash where strace is unavailable), and commit a ranked baseline table so every later change is measured against it.
2. Diagnose the top offenders by mechanism: command substitutions inside per-item loops, grep/sed/jq/git invoked once per file or per line, find walks repeated per suite, mktemp -d per case, cygpath and uname per call, npx cold starts, and one pytest interpreter start per wrapper.
3. Optimize by the same pattern lib/hook-utils.sh already used (perf(hook-utils): build the telemetry envelope and read file_path with builtins #3678): builtins over spawns, one batched git call over many, one jq parse reused, one find walk hoisted, and one pytest invocation for co-located Python suites (with -p no:cacheprovider so no .pytest_cache is left for markdownlint to trip over).
4. Make parallelism safe on Windows: fix the suites in tool-honesty and jira conformance suites fail under run-plugin-tests --jobs 4, pass serially #3694 that fail under --jobs, then measure whether --jobs 2 or --jobs 4 pays on a healthy host and set the default accordingly.
5. Guard the result: a per-suite wall-clock budget lane (warn, then fail above a documented ceiling) so a regression is caught locally, and a documented fast local loop (the smallest command that covers a typical diff).

Acceptance

• A committed baseline table of per-suite wall clock on Windows and on the Linux CI runner.
• No single gate script in scripts/ exceeds 60 seconds on a healthy Windows host.
• scripts/affected-tests.sh --run for a typical 50-file diff completes in under 2 minutes on Windows.
• The full scripts/run-plugin-tests.sh corpus completes in under 10 minutes on Windows, with the number recorded.
• Each optimization is behavior-preserving and shown so by the existing suites passing unchanged.

[kyle-sexton]

This was generated by AI during triage.

Merged from #3717 (duplicate)

#3717 was filed the same day, by the same author, on the same Windows Git Bash host, against the same root cost: per-process spawn overhead making the local gate and test corpus slow enough that a per-edit check becomes a per-phase check. Closing it as a duplicate of this issue, which carries the acceptance criteria. Carrying over its measurements, which are the per-script numbers this issue asks for in step 1 and did not yet have:

Script	Invocation	Measured
plugins/skill-quality/scripts/check-skill.sh	--require-evals visualize (one skill, markdownlint skipped)	> 120 s foreground timeout; finished under a 540 s background ceiling (two independent runs)
scripts/check-changelog-parity.sh	--check-bump origin/main	1248 s (one run); --check > 180 s; --check-preserved origin/main 16 s
scripts/affected-tests.sh	--run after an evals/evals.json edit	selection alone > 120 s; the basename reference rule selects ~148 suites for one evals file
scripts/check-purged-em-dashes.sh	bare	did not finish within a 600 s background window on one run (timing otherwise unverified)
plugins/planning/scripts/check-open-questions.sh	--ledger … --brief …	> 120 s foreground timeout; the --ledger-only form returns instantly

Conditions for those numbers: branch feat/visualize-absorb-show-me, base origin/main 8eae8e8, Claude Code 2.1.260, Windows 11, Git Bash, no node_modules until npm ci.

Suspected mechanisms #3717 named, for the step 2 diagnosis:

• check-skill.sh is 1630 lines and counts codepoints through iconv, one spawn per file.
• check-changelog-parity.sh is 728 lines and appears to diff every plugin's manifest and changelog against the base ref, so --check-bump scales with the fleet rather than the diff.
• affected-tests.sh's basename reference rule maps any evals.json edit to every suite mentioning evals.json, hence ~148 suites for one plugin's file. This is a selection defect as much as a performance one.
• check-open-questions.sh --brief reads the same ledger the fast path reads, so the cost is in the Brief extraction (extract_section), not the register parse.

Two scope items this issue's acceptance should absorb:

1. The acceptance criterion "no single gate script in scripts/ exceeds 60 seconds" does not reach two of the worst offenders above, which live in plugin trees: plugins/skill-quality/scripts/check-skill.sh and plugins/planning/scripts/check-open-questions.sh. Worth widening the criterion to plugin-scoped gates, or they get optimized last.
2. perf(scripts): local gate scripts take minutes on Windows Git Bash (check-skill, changelog-parity --check-bump, affected-tests, em-dash purge, check-open-questions --brief) #3717 asked for a wall-clock budget per gate published in each script's --help or the CI docs, so a contributor knows when to background a run rather than waiting on it. That complements this issue's step 5 budget lane: one is the enforcement, the other is the contributor-facing statement.

Also from #3717, worth stating in the framing: AGENTS.md tells every contributor to validate with scripts/affected-tests.sh --run. At these timings the honest path is "defer to CI", which makes this a documented-contract problem, not only a comfort problem.

---

Generated by Claude Code

[kyle-sexton]

Additional measurements (2026-09-07, Windows 11, Git Bash, perf/hook-performance-program)

Session: comment-cleanup pass touching 10 files (ruff.toml, plugins/code-tidying/**, one claude-ops script).

• scripts/affected-tests.sh --run --explain over that diff ran 40+ minutes with no completion and was killed.
• Selection alone, no --run, takes 54 seconds wall clock (real 0m54.474s, user 0m0.962s, sys 0m3.860s): almost all of it is process-spawn overhead, not CPU.
• plugins/claude-ops/skills/plugins/scripts/fleet-state.test.sh (2369 lines) exceeds the 600-second Bash tool maximum on its own, so no tool call can ever run it to completion here. It reached 15 PASS before the cap.
• plugins/disk-hygiene/skills/clean/scripts/test_hygiene.py: a -k subset of 206 cases takes 65 seconds.

Over-selection compounds the cost

The reverse lookup matches on basename, so a change to plugins/code-tidying/skills/dissolve-comments/reference/safety.md selected three unrelated suites:

select: plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_merge.py  (references safety.md)
select: plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_merge_review_settle.py  (references safety.md)
select: plugins/source-control/skills/babysit-prs/scripts/tests/test_skill_contract.py  (references safety.md)

Those tests reference the babysit-prs safety.md, not the code-tidying one. A 10-file diff selected 23 suites. Matching on the repository-relative path, or at least on <skill>/<file>, would cut selection sharply for the common case of same-named reference files across skills.

Asks, in addition to the program above

1. Path-qualified reverse lookup instead of basename matching.
2. A per-suite wall-clock budget in CI that fails a suite exceeding it, so a suite cannot grow past what a developer can run locally. fleet-state.test.sh is already past 600 seconds.
3. A documented local escape hatch for the case where the affected set is known to be unrunnable, so a developer can record "NOT RUN locally, CI covers" without the change being reported as validated.

[kyle-sexton]

Same session, one more: scripts/check-changelog-parity.sh --check-order alone exceeded the 600-second Bash tool maximum on this host (2026-09-07, Windows 11, Git Bash) and was killed. A combined --check plus --check-bump main run earlier in the session finished, but only when backgrounded past the 180-second mark. Both are pure read-only gates over CHANGELOG.md files; they belong in the same spawn-count audit as the test suites.