bun test is non-deterministic on a clean clone: 3 failures then 0, same command #2536

Activity

1. CarringtonCreative commented on Aug 12, 2026

   

   CarringtonCreative

   on Aug 12, 2026

   ContributorAuthor

   More actions

   Ran the identical command 5 more times on the same pristine clone. 7 samples total, and the data narrows this a lot — including away from the hypothesis I posted above.

   Frequency

   Run	Fails	Which
   1	3	--list > handles missing proposals file (5.0s) · gstack-community-dashboard > connects to Supabase (6.9s) · D5 detect-first > --dry-run falls through to fresh clone (7.4s)
   2	0
   3	1	D5 detect-first > --dry-run reuses a pre-existing ~/git/gbrain-shaped clone (0.67s)
   4	0
   5	0
   6	1	D5 detect-first > rejects a pre-existing path that lacks a valid gbrain package.json (0.70s)
   7	0

   3 of 7 runs failed (43%). gstack-gbrain-install D5 detect-first is in every failing run — a different one of its three tests each time. The file passes 16/16 in isolation, repeatedly.

   Correcting my hypothesis above

   I guessed resource contention from the multi-second durations. That looks wrong for the dominant case: runs 3 and 6 failed in ~0.7s, which is a logical failure, not a timeout. And run 1 is the least trustworthy sample — I had other work running on the machine concurrently, which likely explains the 5-7s outliers there. distill-apply and telemetry each appeared only in that run and never again in 6 further samples.

   Actual cause: PATH leaks into the D5 tests

   test/gbrain-detect-install.test.ts defines a deliberate guard:

   // Minimal PATH with POSIX tools + homebrew (for jq/git/curl) but no user-bin
   // dirs — this keeps `gbrain` out of PATH deterministically across dev machines
   const SAFE_PATH = '/usr/bin:/bin:/usr/sbin:/sbin:/opt/homebrew/bin:/usr/local/bin';

   But run() doesn't apply it:

   const env = {
     ...process.env,          // <-- PATH comes through untouched
     GSTACK_HOME: tmpHome,
     HOME: tmpHomeReal,
     ...(opts.env || {}),
   };

   GSTACK_HOME and HOME are pinned, so those are safe. PATH is not — and all three D5 tests call run(INSTALL, ['--dry-run']) with no opts.env, so none of them get SAFE_PATH. Meanwhile there are ~20 process.env.PATH = ... assignments across the suite's other test files. If those share a process with this one, gstack-gbrain-install sees a PATH the test never intended, finds (or stops finding) a gbrain binary, and takes a different branch.

   That fits every observation: isolation-clean, fails only alongside other files, a different test in the block each time, sub-second logical failures.

   I'll flag the one thing I have not proven: that bun shares process.env across concurrently-executing test files. I haven't caught a specific polluting file in the act. The circumstantial case is strong but that link is inferred.

   Suggested fix

   Default SAFE_PATH in run() so every test gets the guard the file already intends:

   const env = {
     ...process.env,
     PATH: SAFE_PATH,
     GSTACK_HOME: tmpHome,
     HOME: tmpHomeReal,
     ...(opts.env || {}),
   };

   opts.env still wins, so the tests that deliberately prepend a fake-gbrain dir keep working.

   Happy to submit that, validated across ~10 full-suite runs to show the 43% goes to 0 — say the word and I'll open it.
2. CarringtonCreative commented on Aug 12, 2026

   

   CarringtonCreative

   on Aug 12, 2026

   ContributorAuthor

   More actions

   Found the actual cause, and it is not the PATH leak I proposed above. Retracting that — the suggested SAFE_PATH default would also have broken these tests on any machine where bun lives at ~/.bun/bin (the official installer's location), since SAFE_PATH doesn't include it. Glad I checked before sending it.

   The real cause: --dry-run makes a network call

   The failing assertion is expect(r.status).toBe(0) receiving 3. In bin/gstack-gbrain-install, fail() exits 3, and one of its callers runs before any dry-run early-exit:

   if ! $VALIDATE_ONLY; then
     check_prereq bun ...
     check_prereq git ...
     if ! curl -s --head --max-time 10 https://github.com >/dev/null 2>&1; then
       fail "cannot reach https://github.com. Check your network and try again."
     fi
   fi

   --dry-run sets DRY_RUN, not VALIDATE_ONLY, so a dry run performs a live request to github.com — then exits without ever cloning.

   Direct proof

   60 concurrent --dry-run invocations against temp HOME/GSTACK_HOME, outside the test harness entirely:

   exit=3 :: gstack-gbrain-install: cannot reach https://github.com. Check your network and try again.
   ...  (9 of 60)
   

   ~15% failure rate, no test runner involved. Under a full bun test the suite spawns plenty of concurrent processes, the curl loses a race for sockets/DNS, and whichever D5 test is executing at that moment goes red. That explains everything: passes in isolation, different test each time, sub-second failures, ~43% per suite run.

   Fix

   Skip the reachability check under --dry-run — it plans, it never clones, so it doesn't need the network:

   if ! $DRY_RUN && ! curl -s --head --max-time 10 https://github.com >/dev/null 2>&1; then

   Same 60-concurrent experiment with the guard: 0 of 60. Real installs still get the fail-fast offline check.

   Running the full suite 10x to confirm the 43% goes to zero; PR to follow.

   One broader note

   telemetry > connects to Supabase when config exists — one of the other failures in my first sample — is likely the same shape: a real network call inside the free Tier-1 suite. I haven't investigated it and am not claiming it, but it may be worth a look while this is fresh.
3. CarringtonCreative commented on Aug 12, 2026

   

   CarringtonCreative

   on Aug 12, 2026

   ContributorAuthor

   More actions

   Validation done, PR up: #2540.

   Documented Tier 1 gate on a clean clone — 10 runs, 0 failures, against the 43% (3 of 7) measured on the same clone before the change. Combined with the 9/60 → 0/60 direct experiment, that's the mechanism, a causal test, and a statistical result all pointing the same way.
4. garrytan

   added a commit that references this issue on Aug 17, 2026

   v1.67.0.0 fix: the tracker wave — XProtect self-heal, complete instal…

   ...

   ae8914a
5. garrytan

   closed this as completedin #2604on Aug 17, 2026