fix(test): mlxcel-core release test binary intermittently hangs forever, blocking later runs via the cargo lock

Summary

The mlxcel-core release test binary intermittently hangs and never terminates, spinning at 100% CPU indefinitely. It has been observed three times over three days on the same machine. Because the hung process keeps the cargo file lock for its target directory, every subsequent cargo test in that working tree blocks behind it, so the failure also silently prevents later test runs from executing at all.

This matters more than an ordinary flake because this repository's CI does not run the test suite. A local full-suite run is the project's only real correctness gate, and a gate that intermittently hangs forever is not a gate.

Observations

Three occurrences, all the same test binary (target/release/deps/mlxcel_core-693b1aae6e646b8d), all pinned at approximately 100% CPU, all requiring kill -9:

elapsed when found	working tree
2 days 02:44:59	`mlxcel-2`
2 days 11:34:45	`wt-epic-909-issue-899` (worktree, since removed; the process held the deleted inode)
7 hours 43:44	`wt-epic-909-issue-904`

Spinning rather than blocked: the processes consumed a full core each for their whole lifetime, which distinguishes this from a deadlock on a mutex.

Independently, during the same period an implementer working on #902 reported a full-suite run stalling on layers::tests::causal_attention_* while pgrep showed test binaries from two different working trees competing for the GPU. That is the only named lead so far and has not been confirmed.

Conditions

Every observed occurrence happened while the machine was under heavy parallel load: several git worktrees of this repository building and running tests concurrently, load average between 18 and 25 on a 20-core M1 Ultra. The suite completes normally in about 20 to 30 seconds of test time when the machine is quiet, and repeated quiet-machine runs during the same period all completed.

That points at a contention-dependent failure rather than a deterministic one, and it means reproduction likely requires inducing GPU or memory pressure rather than simply re-running the suite.

Impact

Two effects, the second easy to misdiagnose:

A full core is consumed indefinitely. Two of these ran for over two days before being noticed, which also silently degraded every performance measurement taken on the machine during that window.
The hung process holds the cargo lock for its target directory. A later cargo test in the same tree blocks waiting for it and produces no output, which reads exactly like the new run itself hanging or dying silently. During epic Epic: Fused paged decode, sorting-free sampling, and serving performance techniques #909 this was initially misdiagnosed as shell pipe buffering; the actual cause was a new run queued behind a hung old one.

Suggested investigation

Reproduce by running cargo test -p mlxcel-core --release --features metal,accelerate in two or more worktrees simultaneously, ideally with additional GPU load, and watch for a test binary that outstays the normal 20 to 30 second runtime.
When one is caught live, sample <pid> gives the stack. A Metal command buffer that never completes, or a wait inside MLX's stream worker, would both present as a spinning process here.
layers::tests::causal_attention_* is the only named candidate and should be checked first, but it is a single unconfirmed report.
A per-test timeout in the harness would convert an indefinite hang into a failed test, which is strictly better for a gate even if the root cause takes longer to find.

Environment

Apple M1 Ultra, 20 cores, 128 GB, macOS 26.5.2 (Darwin 25.5.0), Metal backend, --features metal,accelerate, release profile, rustc 1.93.1. Observed across commits spanning epic #909 (14412c13 through 78c3d6e6).

Related: #964 is a separate pre-existing failure in the same suite.

Reproduced, and the mechanism is not what the issue first suggested

Running the suite in two working trees at once reproduces failure reliably, but the dominant symptom is an abort rather than the hang, and the cause is not test-thread count.

Method. cargo test -p mlxcel-core --release --features metal,accelerate launched simultaneously in mlxcel-2 and a sibling worktree, six rounds.

Result: 7 of 12 runs aborted (58%).

round1 a: completed 1368 passed; 1 failed        round1 b: (no result line)
round2 a: CRASH  Command buffer execution failed round2 b: completed 1399 passed; 3 failed
round3 a: CRASH  Failed to make new command queue round3 b: CRASH Command buffer execution failed
round4 a: CRASH  Command buffer execution failed round4 b: CRASH Command buffer execution failed
round5 a: CRASH  Command buffer execution failed round5 b: CRASH Command buffer execution failed
round6 a: (no result line)                       round6 b: completed 1398 passed; 4 failed

The two error forms:

libc++abi: terminating due to uncaught exception of type std::runtime_error:
  [metal::CommandEncoder] Failed to make new command queue.
libc++abi: terminating due to uncaught exception of type std::runtime_error:
  [METAL] Command buffer execution failed: Internal Error (00000206:Internal Error).

Both abort with SIGABRT.

The more damaging finding: runs that do not crash report false failures

round2 b finished with 3 failed and round6 b with 4 failed, against the single known pre-existing failure (#964) that a quiet-machine run produces. So a concurrent run does not merely risk aborting, it can complete and report regressions that do not exist. That is worse than a crash, because a crash is obviously invalid while a failure count is not, and someone will spend time investigating a test that is fine.

Test-thread count is not the cause

The first hypothesis was command-queue exhaustion from Rust's default test parallelism (20 threads here), since new_thread_local_stream_gpu() creates a GPU stream per call and a Metal stream is a command queue. That is wrong. Repeating the concurrent experiment with an explicit cap still aborts:

`--test-threads`	tree A	tree B
4	CRASH, command buffer execution failed	CRASH, command buffer execution failed
8	CRASH, command buffer execution failed	no result line

So bounding parallelism inside one process does not help. The conflict is between processes sharing one Metal device, not between threads inside one process. Note that make test already passes --test-threads=1, which addresses intra-process ordering and does nothing for this.

Relationship to the reported hang

The three hangs in the original report and these aborts share the same trigger, concurrent suites on one GPU, and are most likely two outcomes of the same contention. The hang is the worse outcome because it holds the cargo lock indefinitely and silently blocks later runs in that tree.

Suggested direction

Making two suites coexist on one GPU is probably not the right goal. The actionable defects are that the failure is opaque, that it can masquerade as real test regressions, and that it sometimes hangs instead of failing.

Detect at test-suite start that another mlxcel-core test binary is running, and say so loudly in the output, so a 3-failure run is not mistaken for a regression. This is the cheapest change and addresses the most damaging symptom.
Consider an advisory cross-process lock so a second suite waits rather than competing. It needs stale-lock handling, since a hung holder is exactly the situation being defended against.
A per-test timeout would convert the hang into a failed test.

Environment

Apple M1 Ultra, 20 cores, 128 GB, macOS 26.5.2 (Darwin 25.5.0), Metal, release profile, rustc 1.93.1, at 78c3d6e6 and a sibling worktree.

fix(ci): widen the verify gate to all four workspace members (#1021)

docs: document the CUDA test guards' escape hatches and trim a repeat

fix(test): make the CUDA merge gate runnable and serialized (#1052)

test(core): fail loudly when another mlxcel-core test binary shares t…

fix(internlm): stop doubling RoPE positions under dynamic NTK scaling (…

test(turbo): exercise the steel kernel and classify #1712 failures (#…

Navigation Menu

Repository navigation

fix(test): mlxcel-core release test binary intermittently hangs forever, blocking later runs via the cargo lock #1008

Summary

Observations

Conditions

Impact

Suggested investigation

Environment

Reproduced, and the mechanism is not what the issue first suggested

The more damaging finding: runs that do not crash report false failures

Test-thread count is not the cause

Relationship to the reported hang

Suggested direction

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fix(test): mlxcel-core release test binary intermittently hangs forever, blocking later runs via the cargo lock #1008

Description

Summary

Observations

Conditions

Impact

Suggested investigation

Environment

Activity

inureyes commented on Aug 3, 2026

Reproduced, and the mechanism is not what the issue first suggested

The more damaging finding: runs that do not crash report false failures

Test-thread count is not the cause

Relationship to the reported hang

Suggested direction

Environment

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions