Skip to content

test:free runs the real gbrain when it is on PATH; brain-cache tests hit a remote brain and trip the 30s exec timeout #2829

Closed

Description

@michaelrgeary

Summary

bun run test:free is not hermetic on a machine that has gbrain installed and pointed at a remote (Supabase) brain. Several tests reach lib/gbrain-exec.ts through the real binary on PATH, so each call is a network round-trip (~1.5 s here). Under the free-suite shard load, one brain-cache test crossed the 30 s default exec timeout and failed. gstack's own CI never installs gbrain, so the suite is green upstream and the failure only shows up on a user's machine — in my case inside a launchd auto-updater that runs ./setup && bun run test:free and rolls back on red.

Environment

  • gstack 1.81.0.0 (0530392), macOS 15, bun via ~/.bun/bin
  • gbrain 0.48.x installed globally, configured against a remote Supabase brain (not PGLite)
  • Suite run unattended (launchd), 6 shards

Observed

(fail) brain-cache malformed _meta.json (#1879) > cmdGet does not throw on null / array / primitive _meta.json [30759.31ms]

test/brain-cache-roundtrip.test.ts writes a malformed _meta.json under a temp HOME, so every entity is stale; cmdGet in bin/gstack-brain-cache then refreshes via execGbrainJson / spawnGbrain (lib/gbrain-exec.ts, timeout: opts.timeout ?? 30_000). The temp HOME isolates the cache dir but not the binary: gbrain still resolves from PATH and talks to the user's real brain. Four such lookups at ~1.5 s each fit comfortably in isolation but not under shard contention, and the whole test is then reported as a failure of the assertion under test rather than of the environment.

Runs of the same suite with gbrain removed from PATH pass every time (GSTACK_FREE_RETRY_FLAKY=1 did not rescue the red run because the failure reproduces serially once the brain is slow).

What does not work as a workaround

A stub gbrain that exits non-zero first on PATH is not equivalent to "absent": test/gbrain-sync-voyage-code-3-integration.test.ts gates on which gbrain (plus VOYAGE_API_KEY) and then fails 3 tests against the stub, reproducibly. So the only green state on a dev machine with gbrain installed is to build a PATH without it, which is what I ended up doing in my updater (mirror every PATH dir that contains gbrain into exec-wrappers minus gbrain, run the suite on that).

Suggested fix (either would do)

  1. In the brain-cache tests, stub the exec layer (lib/gbrain-exec.ts) or set whatever env makes bin/gstack-brain-cache treat gbrain as unreachable, so cmdGet exercises the malformed-meta path without a real spawn. Those tests are about _meta.json handling, not about gbrain.
  2. More generally, have test:free refuse to spawn a real gbrain unless an explicit opt-in (e.g. GSTACK_TEST_REAL_GBRAIN=1) is set, mirroring how the voyage integration test already gates on env. Then a developer's installed gbrain can't leak into the "free" suite.

Happy to send a PR for (1) if you'd take it.

Activity

  1. capy-ai commented on Oct 3, 2026

    @capy-ai
    Contributor

    Fixed in v1.91.16.0 (#3014, ac20ef1): free tests no longer call a real gbrain on PATH.

    Thanks for the report! If it still happens on v1.91.16.0 or later, reply here and we'll reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions