Skip to content

macOS: 3 tests in test/gstack-design-detect.test.ts fail deterministically — $TMPDIR vs its realpath (/var -> /private/var) #2841

Closed

Description

@michaelrgeary

Summary

Three tests in test/gstack-design-detect.test.ts fail deterministically on macOS because the test
builds its expected path from $TMPDIR while the code under test has already resolved that path to
its real location. On macOS $TMPDIR is /var/folders/… and /var is a symlink to /private/var,
so the two strings differ by that one prefix even though they name the same directory.

This is not flaky — it fails on gstack's own serial flaky-retry as well.

Environment

  • gstack 1.84.1.0 (71f6048e)
  • macOS 26.6.2, arm64
  • bun 1.3.14

Failing tests

From bun run test:free (shard 5/7 failed with 3; the GSTACK_FREE_RETRY_FLAKY=1 serial retry
failed with 4):

  • probe > ignored rules and files are the union of config.json and config.local.json; malformed config is reported
  • scan: option-like bases and page-controlled inline ignores > from HOME (no repository) HOME-rooted installs are READY, HOME files are refused as targets, and a dump still scans without inline ignores
  • scan: option-like bases and page-controlled inline ignores > the whole-scan budget stops a huge target set instead of grinding batch after batch (retry only)
  • install: the one download gstack makes, after consent > downloads from a mirror, verifies the checksum, installs under IMPECCABLE_HOME, receipts the fetch first, and the probe finds it

Reproduction

$ echo $TMPDIR
/var/folders/qp/…/T/
$ cd "$TMPDIR" && pwd -P
/private/var/folders/qp/…/T

Any assertion that joins $TMPDIR into an expected path and compares it against output the
implementation has realpath'd will differ by the /private prefix.

Confirmation

Running that test file with a TMPDIR that is already its own realpath:

TMPDIR="$(cd "$TMPDIR" && pwd -P)" bun test test/gstack-design-detect.test.ts
before after
test/gstack-design-detect.test.ts 62 pass, 3 fail 65 pass, 0 fail

No other test file changes result. I specifically A/B'd the path-security tests, since a naive
"resolve all temp paths" change could plausibly weaken them — they are unaffected:

before after
browse/test/path-validation.test.ts 29 pass, 0 fail 29 pass, 0 fail
browse/test/temp-dirs.test.ts 7 pass, 0 fail 7 pass, 0 fail
browse/test/data-platform.test.ts 15 pass, 0 fail 15 pass, 0 fail

(One caveat worth knowing if you reproduce this: browse/test/path-validation.test.ts and
temp-dirs.test.ts resolve paths relative to cwd and treat the temp dir as allowlisted, so running
them from a checkout that itself lives under /tmp produces three unrelated failures. Clone under
$HOME to reproduce cleanly — that cost me an hour of chasing a phantom.)

Suggested fix

Resolve $TMPDIR to its real path where the tests construct expected paths (or have the tests
compare realpath-to-realpath rather than raw strings). Happy to send a PR if you'd like it in a
particular shape — I didn't want to guess at your preference between fixing it in the individual
assertions vs. a shared test helper.

Why this matters beyond one machine

CI presumably runs on Linux, where /tmp has no such symlink, so this is invisible there and hits
every macOS user. On my side it was surfaced by an auto-updater that runs the free suite as a gate
before accepting a new gstack — with this bug, every release rolls back on macOS.

Activity

  1. v639dragoon commented on Sep 13, 2026

    @v639dragoon
    Contributor

    Independent confirmation on a second machine, same three tests, same cause.

    • gstack 1.84.1.0 (71f6048e), pristine origin/main worktree, bun install --frozen-lockfile + bun run gen:skill-docs
    • macOS (Darwin 25.6.0), arm64, bun 1.3.11
    • bun test test/gstack-design-detect.test.ts → 62 pass, 3 fail, deterministic across three runs (also reproduced inside the full sharded bun run test)

    Failing assertions and the exact prefix drift:

    1. probe > ignored rules and files are the union … (test/gstack-design-detect.test.ts:315)
      expected IMPECCABLE_CONFIG_UNREADABLE: /var/folders/…/repo/.impeccable/config.local.json
      received IMPECCABLE_CONFIG_UNREADABLE: /private/var/folders/…/repo/.impeccable/config.local.json
    2. scan … HOME-rooted installs are READY … (:984)
      expected IMPECCABLE_READY: /private/var/folders/…/.impeccable/bin/0.1.3/impeccable (this one is already realpath'd on the test side)
      received IMPECCABLE_NOT_AVAILABLE — so here the drift is on the other side: the fake HOME is handed to the probe un-realpath'd and the install under it is not found.
    3. install … the probe finds it (:1131)
      expected IMPECCABLE_INSTALLED: /var/folders/…/bin/0.1.3/impeccable version=0.1.3 sha256=… bytes=2180
      received the same line with /private/var/…

    Everything else in the file passes. Happy to test a fix branch on this box.

  2. capy-ai commented on Oct 3, 2026

    @capy-ai
    Contributor

    Fixed in v1.91.16.0 (#3014, ac20ef1): fixed in the tests (macOS's symlinked temp root and process-group EPERM), proven on the new macos-named-regressions CI job.

    Thanks for the report! If it still happens on v1.91.16.0 or later, reply here and we'll reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions