Test-suite consult runs escape isolation: real agy spawns (login-window burst) + user-global metrics DB pollution

[waleedkadous]

Observed (2026-08-01, ~05:56-05:59 local)

A builder (aspir-1286) ran the unit + CLI-integration suites plus manual real-agy verification. Within the same two-minute window the user got a burst of dozens of agy login browser windows — the #1077 forkbomb, back through a path the #1250 fix explicitly does not cover.

Evidence from the user-global consult metrics DB (consult stats --json):

• 8 gemini + 2 codex invocations timestamped within ~1.5s (12:58:57-58Z), all 0.0s duration
• workspace_path values are throwaway test dirs: /var/folders/.../codex-test-mmxOf7, .../codev-consult-test-1785589137328
• one gemini row carries error_message: "agy timed out producing the review" — a real agy spawn attempt from a test process
• An identical fingerprint exists from 2026-07-25 06:11 (8× gemini 0.0s + 2× codex 0.0s), so this happens on every suite run that exercises these paths; it only becomes visible when agy happens to want login (agy updated to 1.1.9 on 07-31, which triggered re-auth)

This also matches the earlier report on #1077 of a tab burst while running pnpm --filter @cluesmith/codev test:e2e:cli.

Why the #1250 fix doesn't help here

Two deliberate design points of the auth cache compose badly in the test path:

1. agyAuthCacheDisabled() returns true under VITEST unless CODEV_AGY_AUTH_CACHE_DIR is set — correct for keeping unit tests off the developer's real ~/.cache/codev, but it also means burst protection is off for any test that reaches the real lane. Spawned codev consult child processes inherit VITEST from the runner's env, so CLI-integration/e2e children are inert too.
2. Tests that don't pin a fake agy resolve the real binary from PATH. Pre-flight disabled + real binary = unconditional spawn per test = one login window per spawn when agy is unauthenticated.

Separately, those same child processes record into the real user-global metrics DB (rows id ~35478-35487 and counting) — every suite run appends ~10 junk rows with test-temp workspace paths, silently skewing consult stats.

Fix directions (for the builder to evaluate)

• The test harness (unit setup + the helpers that spawn codev consult subprocesses in CLI-integration/e2e) should pin CODEV_AGY_BIN (or equivalent) to a fake agy unconditionally, so no test can resolve the real binary by omission.
• Consider a belt-and-braces guard in runAgyConsultation: refuse to spawn a real agy when running under a test environment (VITEST/CODEV_TEST-style detection) without an explicit opt-in — misconfigured future tests should fail loudly, not open browser windows.
• Metrics isolation: same treatment as the auth cache — tests pin a metrics DB path into their temp dir (or metrics writes are inert under test env unless a path is pinned). Consider whether the existing junk rows warrant a one-off scrub.
• Builders doing deliberate real-agy manual verification (as aspir-1286 legitimately did) are out of scope — the issue is tests spawning real agy by omission.

Acceptance criteria

• No test in unit, CLI-integration, or e2e suites can spawn the real agy binary (verify with a canary: put a logging fake at the real agy's PATH position, run full suites, assert zero invocations)
• Suite runs add zero rows to the user-global consult metrics DB
• A test that reaches the agy lane without pinning a fake bin fails loudly rather than spawning
• Existing [Bugfix #1077] Pre-flight agy auth cache to stop the OAuth tab burst #1250 burst/regression tests still pass unchanged

Related

• consult: pre-flight auth-state cache for agy to prevent N-spawn browser-tab burst on unauthenticated state #1077 / PR [Bugfix #1077] Pre-flight agy auth cache to stop the OAuth tab burst #1250 — the production-path fix (pre-flight auth cache); this issue is the test-path escape it explicitly scoped out
• consult: pre-flight auth-state cache for agy to prevent N-spawn browser-tab burst on unauthenticated state #1077 comment by mohidmakhdoomi — earlier sighting via test:e2e:cli
• agy 1.1.9 (2026-07-31 update) triggered the re-auth state that made this visible; the isolation gap exists regardless of agy version

[waleedkadous]

On it! Working on a fix now.