CI: track flaky tests on main and PRs #2327

Several tests fail intermittently in CI, both on main and in PRs, and then pass on rerun with no code change. This issue tracks them in one place so they can be fixed and checked off. Where a dedicated issue already exists, the entry links to it, and the fix should land there.

Data: CI workflow runs from 2026-10-08 to 2026-10-14: 400 runs, including first attempts of reruns, plus earlier main runs where noted. Rates are counted against jobs that actually ran the relevant suite. They are approximate lower bounds, because some runs reuse cached Go test results.

Shared context for items 1 and 2. On 2026-10-12, two root-module test packages became much slower under -race. deploy/helm/mecatl-execution went from 7–15s to 137–233s. deploy/mecatl-execution-kind went from 2–4s to 87–155s. The jump is first visible at the #2219 merge commit (run: 172s and 117s). Both packages run alongside every other root-module package in the single Test (race) job (task --concurrency 3 test:race, which passes no -timeout). Items 1 and 2 both first appeared that day. This is a correlation, not an established cause.

Flaky tests

• 1. cmd/mecatui/ui: the package exceeds the default 10m go test timeout (Test (race))

  • Failure: panic: test timed out after 10m0s then FAIL github.com/stacklok/mecatl/cmd/mecatui/ui 601s. The test still running at the deadline is only 0–3s old and is different each time (TestApprovalSurfaceStructuralBoundary, TestModalFrameLifecycleSharedAcrossPlacements/fill, TestSteer_TUIRendersAuthoritativeState/…, TestMecatuiSlashPaletteBoundedList_Scenario1_…). The whole package runs out of time; no single test hangs.
  • Runs: main, 2026-10-14 · main, 2026-10-14 · #2292, attempt 1 (attempt 2 passed, with the package at 460s)
  • Frequency: 13 of 82 suite-running Test (race) jobs since 2026-10-12 12:00 UTC (~1 in 6; 2 of them on main). It did not occur in the 80 suite-running jobs from 2026-10-08 up to then.
  • Evidence: in passing race runs the package takes 192–594s (median about 380s). One passed at 593.9s (#2291, attempt 2). See shared context above.
  • Flaky: yes. It passes on a same-SHA rerun.

• 2. internal/adapter/acp TestServeOverLimitFrameErrorsNotHang (Test (race))

  • Failure: conn_test.go:279: Serve hung on an over-limit frame instead of returning the error
  • Runs: main, 2026-10-14 · #2291, attempt 1 · #2219, attempt 1 (both PR reruns passed)
  • Frequency: 19 of 82 suite-running Test (race) jobs since 2026-10-12 12:00 UTC (~1 in 4). Zero before 2026-10-12. It was first seen on feat: add execution templates with platform-managed identity #2219's PR run.
  • Evidence: the test gives Serve a fixed 2s wall-clock budget to read and reject a maxFrameBytes+1 (16 MiB + 1 byte) line. In every sampled run before 2026-10-12 (23 runs) the package finished in 6.5–10.8s. Since then many runs take 16–30s, and every failure is one of those slow runs. In 10 of the 13 item-1 jobs, this test also failed.
  • Suspected cause: CPU starvation of a fixed wall-clock budget; see shared context. Not proven.
  • Flaky: yes.

• 3. cmd/mecated TestRunMCPLoginProductionOptionsKeepDCRPriorityThroughRealRuntime (Test (race), test:microvm-dev-tag task)

  • Failure: mcplogin_test.go:405: context deadline exceeded (last callback rejected: callback state did not match the authorization request)
  • Runs: main, 2026-10-14 · main, 2026-10-14 · #2310
  • Frequency: 3 of 10 suite-running Test (race) jobs since the test landed with fix(mcp/oauthlogin): pin mcp login's callback path, keep the port ephemeral #1377 (2026-10-13 21:19 UTC). Each time, cmd/mecated passed in the root-module run and failed in the -tags=microvm_dev run of the same job.
  • Suspected cause (supported by the code and message): the test sends the OAuth callback (state=s) from a goroutine (cmd/mecated/mcplogin_test.go:394) concurrently with present(ctx, ".../authorize?state=s") (:401). The expected state is only set inside present (mcp/oauthlogin/runtime.go:280). If the callback arrives first, callbackFlow.validate rejects it as a wrong-state probe (mcp/oauthlogin/callback.go:244). The test sends no second callback, so the 3s context expires with exactly this message.
  • Flaky: yes. The same test passes and fails within one job on the same SHA.

• 4. environment/microvm TestRepositoryRuntimeHostedSocketSuffix (Test (race))

  • Failure: snapshot repository Git objects: open object entry "maintenance.lock": no such file or directory (reported from repository_composition_test.go:569 or :587)
  • Runs: #2265, attempt 1 · #2291, attempt 1 (both reruns passed) · #2215
  • Frequency: 3 of 162 suite-running Test (race) jobs since 2026-10-08 (~2%).
  • Suspected cause: copyObjectDirectory lists .git/objects with ReadDir, then opens each entry (environment/microvm/repository_object_snapshot.go:110, :130). An entry that disappears in between is a hard error. The fixture commits right before the snapshot (repository_lifecycle_test.go:589), and gitexec does not disable Git auto-maintenance, so Git's transient objects/maintenance.lock can disappear mid-walk. This is the same race class as flaky test: TestMergerTextconvDoesNotFire races with git's own maintenance.lock cleanup #669, which internal/adapter/forker handles with isTransientGitMaintenanceLock (forker.go:711). The microVM snapshot walker has no equivalent.
  • Related, seen once: internal/adapter/forker TestForkDirtyOverlayConcurrent, error copy base tree: open …/.git/worktrees/mecatlfork-conc-1-…/locked: no such file or directory (#2286, attempt 1; rerun passed).
  • Flaky: yes.

• 5. internal/app TestMicroVMChildRetainsParentSourceAfterRetirement: tracked in flake: MicroVM child retained-source test can deadlock during provider handshake #2240 (Test (race))

  • Failure: panic: test timed out after 10m0s with TestMicroVMChildRetainsParentSourceAfterRetirement (8m41s–9m29s) still running.
  • Runs: main, 2026-10-09 · main, 2026-10-05 · #1935, 2026-10-14
  • Frequency: 8 of 162 suite-running Test (race) jobs since 2026-10-08 (~5%), plus 2 main runs on 2026-10-05. Most recent: 2026-10-14.
  • Suspected cause: the circular wait described in flake: MicroVM child retained-source test can deadlock during provider handshake #2240. The arrived <- req before calls.Add(1) ordering is still present at internal/app/microvm_harness_context_test.go:375-377.
  • Flaky: yes (reruns pass).

• 6. "TypeScript SDK codegen and offline e2e" (task sdk:e2e)

  • 6a. e2e/mcp-authorization.e2e.test.ts > "MCP authorization works over gRPC TCP UDS and HTTP SSE": tracked in test(sdk): investigate flaky gRPC MCP authorization premature close #1785
    • Failure: grpc tcp|uds authorization lifecycle failed … ConnectError: [unknown] Premature close
    • Runs: #2219, gRPC TCP · #2266, gRPC UDS · #2243, gRPC UDS · main, 2026-10-05, gRPC TCP
  • 6b. e2e/run-controls.e2e.test.ts > "run controls work over grpc tcp uds and http": no existing issue found
    • Failure, variant 1: TransportError: The request could not reach the mecatl server caused by ConnectError: [unknown] Premature close (ERR_STREAM_PREMATURE_CLOSE) (#2277)
    • Failure, variant 2: steer outcome mismatch at run-controls.e2e.test.ts:211: got outcome: "accepted", expected "appended", for messageId: "media-2" (#2263)
  • Frequency: 5 flaky failures in about 180 runs of the job between 2026-10-08 and 2026-10-14 (~3%), not counting runs that failed for branch-specific reasons. All 4 that were rerun passed.
  • Flaky: yes.

• 7. internal/app TestFireProvisionalAdmissionDeadlineKeepsPendingUntilRecovery (Test (non-race draft))

  • Failure: scheduler_liveness_test.go:463: admission recovery did not report actual failure: {…} (2×). One run instead failed with scheduler_liveness_test.go:461: deadline fixture gate not reached, followed by goleak found unexpected goroutines … (*Service).renewLoop.
  • Runs: #2297, attempt 1 (rerun passed) · #2297 · #2286
  • Frequency: 3 of at most 23 Test (non-race draft) jobs since the test landed with fix(scheduler): keep polling behind stuck fires without releasing ownership #2277 (2026-10-13 13:43 UTC). It hasn't been seen in Test (race).
  • Suspected cause: not established. The test uses the wall clock and a 1s FireTimeout.
  • Flaky: yes.

Seen once in this window (watch list)

• cmd/mecatequi TestTelemetryShutdownTimeoutBounded: realMain against a stalling collector took 716.605464ms; … should keep it under 600ms (#2286, attempt 1; rerun passed)
• cmd/mecatui/customization TestStatusLine_Scenario4_DebounceTemplateRefreshAndClock: lifecycle_test.go:36: template clock did not refresh on its interval (#2121)
• internal/adapter/server TestHeadlessAutoApproveUsesExactOwnership: data race, with a write in (*Service).promoteRunAdmission (service.go:8785) against a read in (*Service).completeRelay (#2234)

Other open flake issues

• fix(app): flaky startup data race between attempt recovery and Build config mutation #2268: startup data race in TestCloudNativeLearning_Scenario3_ExplicitProcedureAttemptSurvivesRestart. Seen once in this window (#2266); a fix is open in fix(app): eliminate recovery startup Config race and order worker lifecycle #2310.
• Flaky mecak8s Prometheus metrics listener test under race suite #2074 (mecak8s Prometheus listener), test(learning): diagnose intermittent learned-skill publication failure #1088 and test: diagnose intermittent JSONL inventory catalog preparation failure #1223 (TestUsableAutoSkillsStockBuildPolicyMatrix): not seen in this window.

Checked and not included

• deploy/helm/mecak8s TestMecatlStudioHelmChart_ReleaseMetadataMatchesVersion (main): a merge-skew regression, not a flake. Release v0.0.48 (Release v0.0.48 #2294) bumped VERSION, then feat(studio): add the mecatl-studio Helm chart #2291 added the Studio chart at 0.0.47. fix(studio): align the mecatl-studio chart with VERSION 0.0.48 #2295 fixed it.
• Studio global-search.test.tsx > "checks fresh identity before reopening cached private results" (failed on main on 2026-10-10 and 2026-10-12): appears fixed by feat(studio): add the @/ alias and port the prototype's foundation primitives and utilities #2225, which waits for the palette to re-key before clicking. No failures since.

How to verify a fix

Several of these depend on load, so one local pass proves little. Run each repeatedly, add CPU pressure where noted, and then confirm the signature stops appearing across several full CI runs.

# 1. Should finish well under 600s even with reduced parallelism; also watch "ok ... cmd/mecatui/ui" in CI
GOMAXPROCS=2 go test -race -count=1 ./cmd/mecatui/ui

# 2. Fixed 2s budget; -cpu 1,2 adds scheduling pressure
go test -race -count=50 -cpu 1,2,4 -run '^TestServeOverLimitFrameErrorsNotHang$' ./internal/adapter/acp

# 3. CI runs both variants
go test -race -count=100 -run '^TestRunMCPLoginProductionOptionsKeepDCRPriorityThroughRealRuntime$' ./cmd/mecated
go test -race -count=100 -tags=microvm_dev -run '^TestRunMCPLoginProductionOptionsKeepDCRPriorityThroughRealRuntime$' ./cmd/mecated

# 4. environment/microvm is its own module
(cd environment/microvm && GOWORK=off go test -race -count=50 -run '^TestRepositoryRuntimeHostedSocketSuffix$' .)

# 5. See #2240
GOMAXPROCS=2 go test -race -count=50 -timeout 300s -run '^TestMicroVMChildRetainsParentSourceAfterRetirement$' ./internal/app

# 6. Build once, then repeat the two e2e files
task sdk:e2e
(cd sdk/typescript && for i in 1 2 3 4 5 6 7 8 9 10; do pnpm exec vitest run --config vitest.e2e.config.ts e2e/mcp-authorization.e2e.test.ts e2e/run-controls.e2e.test.ts || break; done)

# 7. It failed in the non-race job; check both modes
go test -count=50 -run '^TestFireProvisionalAdmissionDeadlineKeepsPendingUntilRecovery$' ./internal/app
go test -race -count=50 -run '^TestFireProvisionalAdmissionDeadlineKeepsPendingUntilRecovery$' ./internal/app