[nightly-verify] main is red #1092

Activity

1. github-actions

   added

   status:readyReady to be worked onReady to be worked on

   type:bugBug fixes, error corrections, or issue resolutionsBug fixes, error corrections, or issue resolutions

   priority:highHigh priorityHigh priority

   on Aug 9, 2026
2. github-actions commented on Aug 10, 2026

   

   github-actionsBot

   on Aug 10, 2026 – with GitHub Actions

   Author

   More actions

   The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

   Field	Value
   verify job result	failure
   make verify-fmt	success
   make verify-clippy	success
   make verify-test	failure
   Trigger	schedule

   Duration of the verify job: 2m7s.

   Commit: 45dea248926c5d0a8f09bdfb2ce1d21aed8d504a
   Run: https://github.com/lablup/mlxcel/actions/runs/31418512903

   An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

   Reproduce locally with make verify (or the single failing target above).
   This issue is reused by subsequent nightly reports of the same kind while it stays open.
3. github-actions commented on Aug 11, 2026

   

   github-actionsBot

   on Aug 11, 2026 – with GitHub Actions

   Author

   More actions

   The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

   Field	Value
   verify job result	failure
   make verify-fmt	success
   make verify-clippy	success
   make verify-test	failure
   Trigger	schedule

   Duration of the verify job: 22m22s.

   Commit: 6565d3a208aee7507d10d4be974af180ff75a49f
   Run: https://github.com/lablup/mlxcel/actions/runs/31522220689

   An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

   Reproduce locally with make verify (or the single failing target above).
   This issue is reused by subsequent nightly reports of the same kind while it stays open.
4. github-actions commented on Aug 12, 2026

   

   github-actionsBot

   on Aug 12, 2026 – with GitHub Actions

   Author

   More actions

   The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

   Field	Value
   verify job result	failure
   make verify-fmt	success
   make verify-clippy	success
   make verify-test	failure
   Trigger	schedule

   Duration of the verify job: 26m22s.

   Commit: 58a92914a739fd938c03308d1ced95f4f6845447
   Run: https://github.com/lablup/mlxcel/actions/runs/31627206198

   An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

   Reproduce locally with make verify (or the single failing target above).
   This issue is reused by subsequent nightly reports of the same kind while it stays open.
5. github-actions commented on Aug 13, 2026

   

   github-actionsBot

   on Aug 13, 2026 – with GitHub Actions

   Author

   More actions

   The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

   Field	Value
   verify job result	failure
   make verify-fmt	success
   make verify-clippy	success
   make verify-test	failure
   Trigger	schedule

   Duration of the verify job: 22m12s.

   Commit: 25f8eb51da47074d2417b3ea4cfa686433913257
   Run: https://github.com/lablup/mlxcel/actions/runs/31730309892

   An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

   Reproduce locally with make verify (or the single failing target above).
   This issue is reused by subsequent nightly reports of the same kind while it stays open.
6. github-actions commented on Aug 16, 2026

   

   github-actionsBot

   on Aug 16, 2026 – with GitHub Actions

   Author

   More actions

   The nightly make verify run on self-hosted-macos-26-arm64 failed. Check the first failing step in the run log before reading the gate table below: a setup step that fails leaves the gates reporting failure for want of a toolchain rather than because the code is red.

   Field	Value
   verify job result	failure
   make verify-fmt	success
   make verify-clippy	success
   make verify-test	failure
   make verify-test-video	success
   Trigger	schedule

   Duration of the verify job: 30m53s.

   Commit: ffde948ee094643fd2f3bfd7933e4daa22f1a6fa
   Run: https://github.com/lablup/mlxcel/actions/runs/31963604307

   An unknown step outcome means the job published no outcome for that step. A timeout-minutes cancellation still publishes them, so unknown points at the runner being lost outright rather than at the budget.

   Reproduce locally with make verify (or the single failing target above).
   This issue is reused by subsequent nightly reports of the same kind while it stays open.
7. inureyes commented on Aug 17, 2026

   

   inureyes

   on Aug 17, 2026

   Member

   More actions

   Still red as of the 2026-08-16 nightly, but with a different signature, and it does not reproduce locally

   The 08-14 and 08-15 nightlies were green, so the failure this issue was opened for (2026-08-13, 608c7a6e) had cleared. The 2026-08-16 run at ffde948e failed again, and it is not the same thing.

   What failed

   field	value
   failing step	cargo test (test-fast profile, metal,accelerate)
   failing target	-p mlxcel-core --lib
   last line before the stop	test ffi_tests::test_compiled_softcap_sdpa_gqa_shape ... ok
   test result line for that target	absent
   panic message, failure list	absent

   The binary stopped without reporting a result. That is a hard abort, not an assertion failure, so --no-fail-fast had nothing to collect and cargo reported 1 target failed.

   Everything else in the same run was green: the root crate lib reported 5706 passed; 0 failed; 126 ignored, and the new make verify-test-video gate from #1183 reported 37 passed; 0 failed on its first nightly outing.

   It is a different failure from the one this issue was filed for

   The 2026-08-13 run failed with test result: FAILED. 5573 passed; 2 failed in the root crate lib: real assertions, named tests, a collectable failure list. The 08-16 run has none of that and fails in a different crate.

   It does not reproduce on an M1 Ultra

   Same profile and flags as CI, on main at 4f65e30c:

   cargo test --profile test-fast --features metal,accelerate -p mlxcel-core --lib --no-fail-fast
   test result: ok. 1475 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 20.83s
   

   What changed in mlxcel-core between the last green nightly and this one

   0adb53c5..ffde948e touches src/lib/mlxcel-core/ in exactly two commits: #1182 (the Qwen 3.5 MTP drafter, which adds drafter/qwen3_5_mtp/) and #1170. The four commits merged on 08-16 after #1182 (#1183, #1189, #1192, #1193) are root-crate and tests/ only, so they cannot be the cause of a crate-local abort.

   Reading

   An abort with no panic inside an MLX-backed test binary is the shape this repository already guards against with MLXCEL_ALLOW_CONCURRENT_GPU_TESTS, whose own documentation says two suites on one device corrupt each other and can report failures that do not exist (#1008). That guard fires when a second mlxcel-core test binary is detected; it did not fire here, which either means nothing else was running or means the detection missed it.

   The next nightly is the discriminator, and it is cheap to wait for:

   • Recurs at the same test boundary means it is deterministic and feat(inference): support the qwen3_5_mtp drafter for Qwen 3.5 MTP speculative decoding #1182 is the thing to bisect, since it is the only mlxcel-core change in range.
   • Does not recur puts it in the GPU-contention class, and the useful follow-up is why the exclusivity guard stayed quiet rather than why the test aborted.

   Leaving this open either way: main is red today, which is what this issue tracks.

   For the record on the sibling issue: #1084 tracks a run that did not finish, and the 08-16 run finished in 37 minutes, so that symptom has not recurred.
8. inureyes commented on Aug 17, 2026

   

   inureyes

   on Aug 17, 2026

   Member

   More actions

   The current red is a SIGSEGV, not a test failure, and it is not specific to the runner

   The 2026-08-16 run (31963604307, ffde948e) failed in cargo test (test-fast profile, metal,accelerate) with:

   process didn't exit successfully:
     .../test-fast/deps/mlxcel_core-18654e264ec885e9 (signal: 11, SIGSEGV: invalid memory reference)
   

   No assertion, no test result: FAILED for that binary. The process died, so libtest never printed a summary and every test after the crash point simply did not run.

   This is a different cause from the earlier reds this issue has collected. The 08-11, 08-12 and 08-13 runs each failed with 2 genuine test failures in the mlxcel lib binary (5532 to 5573 passed), and 08-14 and 08-15 were green. Reusing one issue across nightly reports of "the same kind" has merged two unrelated failure modes; the assertion failures are resolved and the crash is what is live.

   Reproduced off the runner

   I hit the identical crash on an Apple M5 Max (40 GPU cores, macOS 26.6.1) during a whole-workspace cargo test --release --features metal,accelerate --workspace: the mlxcel-core binary took SIGSEGV 198 tests in, around cache::paged_detach::tests, and the run reported -p mlxcel-core --lib as a failed target with no summary.

   The same binary run on its own is clean and deterministic:

   invocation	result
   -p mlxcel-core --lib, 5 consecutive runs	1472 passed / 4 failed, every time
   inside --workspace	crashed once out of several attempts

   So it is not a specific test being broken, and it is not the self-hosted runner's hardware. It is intermittent and it correlates with whole-workspace execution, where cargo runs several test binaries concurrently and more than one of them drives Metal.

   Why that shape is suspicious

   This tree already has the CUDA precedent for exactly this failure mode. CONTRIBUTING.md documents that driving MLX from the many host threads libtest spawns takes the CUDA process down with SIGABRT partway through the suite, at a different test and with a different error each run, which is why make verify-test-cuda requires --test-threads=1 and why mlxcel-core carries a the_cuda_test_suite_must_run_single_threaded guard.

   The Metal case differs in where the concurrency lives. --test-threads bounds threads within one binary; here the contention is between binaries, which cargo schedules in parallel and --test-threads does not touch. That points at cargo test --jobs 1 (or a serialized nextest profile) as the lever rather than the existing flag, but the cost in wall clock has to be measured before it is imposed on a 44-minute nightly.

   Suggested next steps

   1. Reproduce under a debugger or with a crash report captured on the runner, so the failing frame is named instead of inferred. The crash point moves, so a single backtrace is worth more than more repetitions.
   2. Measure --jobs 1 wall clock against the current nightly budget.
   3. Split this issue: the assertion-failure reds are resolved and the crash deserves its own tracking, or the auto-reporter will keep appending unrelated causes to one thread.

   Also worth noting for whoever picks this up: a whole-workspace run on this hardware is not a stable comparison basis for anything else either. While baselining the MLX pin bump (#1198) I found models::phixtral::tests::the_prefill_is_causal_without_being_handed_a_mask passing under whole-workspace parallelism and failing in isolation, on both pins. Per-target runs were the only way to get a set that reproduced.
9. inureyes commented on Aug 17, 2026

   

   inureyes

   on Aug 17, 2026

   Member

   More actions

   A second data point from the CUDA gate, weaker than the macOS one

   make verify-test-cuda at fa260a5d on NVIDIA hardware reported two failing targets on its first run:

   error: 2 targets failed:
       `-p mlxcel --lib`
       `-p mlxcel-surgery --test add_integration`
   

   -p mlxcel --lib is explained and not a harness problem: three numeric-agreement tests that pass under MLX_ENABLE_TF32=0, tracked on #1065.

   add_integration is the interesting one. It did not reproduce:

   • standalone, -p mlxcel-surgery --features mlxcel-core/cuda --test add_integration -- --test-threads=1: 3 passed
   • a second full make verify-test-cuda: only -p mlxcel --lib in the failure summary

   So it failed once inside a whole-workspace run and passed every time since, including inside a whole-workspace run.

   I hypothesised this was the same cross-binary concurrency problem as the macOS SIGSEGV above, and the evidence does not support saying so. There is no signal: line in the second run's output, so nothing shows the mechanism, and the first run's log no longer distinguishes an abort from an ordinary failure. What is known is that the target is intermittent under --workspace and stable when run alone. That is consistent with the concurrency hypothesis and also with several other things.

   Recording it because two gates on two platforms now both have a target that fails intermittently only under whole-workspace execution, and because the next person to see add_integration red should know it has passed on every re-run so far rather than start by suspecting the surgery code. If the --jobs 1 experiment this issue already needs gets run, running it on the CUDA gate too would cost nothing extra and would settle whether these are one problem or two.
10. inureyes commented on Aug 18, 2026

    

    inureyes

    on Aug 18, 2026

    Member

    More actions

    PR #1210 has the fix and the measurements.

    Short version for anyone reading this thread before the PR: the --jobs 1 step I suggested above is based on a wrong premise and should not be run. Cargo 1.97.1 completes the whole build before the first test binary starts and runs the binaries one at a time, measured against a three-crate probe workspace, so --jobs cannot affect anything at the time a test is executing. The Makefile's existing claim was the correct one.

    The concurrency is inside a single binary. The macOS crash report for the 08-17 local repro survived on disk and its UUID matches the binary, so the frame picture is direct rather than inferred: 18 named libtest workers live at the fault on an 18-core M5 Max, all of them in MLX-backed cache tests, two inside iokit_user_client_trap and two inside the allocator, faulting on an address in no mapped region. libtest defaults to one thread per logical CPU and the Metal gate never bounded it, while verify-test-cuda has since #1048. Serializing the whole workspace costs +7.2s warm (69.17s to 76.39s, 101 binaries, 8128 tests).

    One measurement trap, since it inverted my first attempt: a cold first run pays about 50s of one-time Metal shader compilation. Compare warm arms only.

    Also note for the local gate, unrelated to the crash: tests/granite4_vision_parity.rs::text_only_forward_produces_finite_logits fails 6/40 parallel and 4/40 serialized against a real checkpoint, so it is thread-count independent and gets its own issue. It does not reach this job, which carries no weights.
11. inureyes

    added a commit that references this issue on Aug 18, 2026

    fix(ci): run the Metal test gate single threaded, like the CUDA one (#…

    ...

    b32059c
12. inureyes

    closed this as completedin #1210on Aug 18, 2026
13. inureyes

    added a commit that references this issue on Sep 2, 2026

    docs: sync PR template clippy and test commands with CONTRIBUTING.md (#…

    ...

    f5bd3e1
14. inureyes

    added

    status:doneCompletedCompleted

    and removed

    status:readyReady to be worked onReady to be worked on

    on Sep 6, 2026
15. be-free-snowboy

    added a commit that references this issue on Oct 27, 2026

    fix(ci): run the Metal test gate single threaded, like the CUDA one (l…

    ...

    3058c72