Repository navigation
scripts/run-gauntlet.sh flaky under concurrent shared-machine load #158
Description
Problem
bash scripts/run-gauntlet.sh, run against a clean pinned commit, produced a real exit 1
("FAIL tests") on its first run, then passed clean (exit 0) on 6 immediate reruns of the exact
same shape (same GIT_* unset, same three-layer parallel execution) against the same commit.
Every one of the 13 individual test files also passed cleanly in isolation. No code-level defect
was found — the failure did not reproduce.
Context
Surfaced during /mh:compliance-audit GH #151 (2026-09-07), auditing historical commit
4bb93955 in an isolated git worktree add --detach checkout. The runner's own 40-line tail
(scripts/run-gauntlet.sh:73-80) hid which specific test failed — every line in the visible tail
was a PASS, meaning the actual failure occurred earlier in run_hook_tests()'s file-iteration
order than the tail window covers.
Evidence for the likely cause: concurrent-session resource contention
scripts/run-gauntlet.sh runs three layers as parallel background jobs, each writing to its own
file under one shared mktemp -d, then unsets GIT_* env vars specifically because a caller
invoking it from a linked worktree (a git-hook context, or exactly this audit's own detached
worktree) can otherwise leak GIT_DIR etc. into the fixture git init calls the test suite runs.
While forensically searching ~/.Trash and leftover /var/folders/.../T/ dirs for the original
failing run's untruncated log (the trash "$LOG" EXIT trap doesn't always fire — a second,
matching-fingerprint, all-pass leftover LOG dir was found still present, confirming the trap can
silently no-op), a run-gauntlet.sh temp-log directory was found actively being written by a
different, concurrent session on the same machine at that same moment — this repo's tree is
routinely used by multiple concurrent Claude Code sessions (documented pattern:
docs/reference/branching-model.md). This repo's own git-hooks/pre-push runs the same gauntlet
on every push; a peer session pushing at the same time is a real, ordinary trigger for this.
The original failing run's raw output was never recovered — none of the ~140 leftover logs
swept matched this worktree's exact fingerprint (13 test files, 29 eval cases) with a failure.
Not confirmed
- The exact mechanism (temp-dir/
mktempcollision under load vs. aflock-based test racing a
peer's own lock vs. something else) — not isolated, only circumstantially evidenced. - Whether this is unique to running from a linked/detached worktree, or would also flake from the
main tree under the same concurrent load.
Suggested next step
Not urgent — 6/7 runs clean, no reproducible code defect, and the pre-push hook already exists as
a backstop (a flaky pass-then-real-push is the risky direction; a flaky fail-then-retry is just
noise). Worth investigating if it recurs: whether mktemp -d collisions or a specific test's
locking assumption breaks under real concurrent gauntlet runs on one machine.
Activity
fix(scripts): preserve full gauntlet log on failure (GH #158)
docs(tests): explain the pid-0 FAIL trick in the #158 regression test
fix(scripts,tests): close trash("") cwd-deletion exposure in the #158 …
fix(hooks,scripts): deep-audit follow-ups on the #117 marker and #158 …
Merge pull request #304 from wasikarn/claude/gauntlet-158-retry-timin…
Merge pull request #481 from wasikarn/claude/gauntlet-lock-158
wasikarn commentedon Oct 5, 2026
Status: per-user pre-push lock merged (#481, released 1.1.184) so pushes queue instead of running gauntlets in parallel; sgg_rc8 already retries 3 times (#470). The 8 s rows are 7 checks in tests/hooks/test-subagent-git-guard.sh, all via sgg_rc8; two more elapsed-time checks sit near lines 440 and 542 of the same file. Not started: ratio test / best-of-N, deliberately, until a push still flakes with the lock on. Lock covers only pushes whose hook has gauntlet-lock.sh (worktree pushes use the main checkout's hook when core.hooksPath is absolute). Keep open and watch.
wasikarn commentedon Oct 5, 2026
More data points from 2026-10-04/05, all on one shared machine with several live Claude sessions:
tests/hooks/test-subagent-git-guard.sh. The lane log showedGH #307 allow shape finishes inside 8 s (rc 124)at a load average of about 4-8. The same suite passed 797/0 when run alone.tests/hooks/test-gates.sh, with the load average at 64. Run alone it passed 1461/1461, and the next push at a load of about 9 passed every layer. I did not capture which case intest-gates.shfailed, so this is a single unexplained sample, not a diagnosis.Suggestion, not a request: have the gauntlet print the failing test's name and its last failing line to the console on failure, so a flake can be told from a regression without a re-run.
Merge pull request #503 from wasikarn/claude/gauntlet-failed-at-158
Merge pull request #632 from wasikarn/claude/stray-proc-sensor
wasikarn commentedon Oct 26, 2026
Status 2026-10-26: the gauntlet now names a stray-process cause of load flakes (#632, with tests tightened in #666).
yesprocesses left over from a stress run held the load near 6 for about 4 h. Every push's timing rows failed until someone found them. An earlier flood (2026-10-05) left 289 workers at load ~248.scripts/_lib/stray-procs.sh, called byrun-gauntlet.shbefore its layers start and again on failure, above thefailed-at:/failing:lists. It groups your own processes older than 30 min by command and names any group whose %CPU sums to 80% or more, along with the load. It is warn-only: it never kills anything and never fails the run.test-gates-other.sh, 5 s against 4 s, load 11.28).repo-gotchas.mdnow has "Background load runs go in a bash file, with explicit PIDs", because the check only catches a stray after the fact.wasikarn commentedon Oct 31, 2026
Re-checked 2026-10-31: load flakes are now fenced. Pre-push is lint-only (#735/#740), the full gauntlet runs once in merge-pr.sh with a tree-hash pass cache and a load wait capped at the core count (af10f29), gauntlet levels exist (GH #749), and stray load is named (#632). The timing rows themselves are still load-sensitive, so this stays open until one more flake is seen with these in place.
wasikarn commentedon Nov 4, 2026
Operator triage, 2026-11-04: close this if no load flake is seen by 2026-11-11.