[inureyes]

Refs #1811, #1801.

Nothing in CI compiles the rocm feature at all: the cargo check jobs do not carry it and the CUDA jobs point at NVIDIA runners. Since the ROCm half of every MLX pin bump lives in src/lib/mlx-cpp/patches-rocm/, a bump that breaks the overlay reaches main unseen.

Why build, link and generate rather than just check

Each step catches a class the one before it cannot.

cargo check never invokes the linker. That is how the CUDA link regression in #1274 passed on a tree that could not link a single integration test. A kernel can compile and link while launching nothing, which is the shape the missing primitives in #1825 had, so the job runs a real forward pass. LD_LIBRARY_PATH is unset for that step, because the binary must find the ROCm libraries through the rpath its build script emits, and #1802 shipped without that rpath on two crates whose test binaries then failed to start with libamdhip64.so.7: cannot open shared object file.

The output assertion was hollow at first, and that is worth reading

The first draft counted word-like tokens over the whole log and required eight. It passed on a run that generated nothing, because the diagnostics alone carry 143 of them:

my grep counts: 143 word-like tokens
...but the content channel is EMPTY. Those words are diagnostics:
mlxcel custom kernel backend rocm no custom kernel

Two causes. Merging stderr into stdout interleaves the [mlx-rocm] device lines into the generated text, on the same physical line as the echoed prompt, so no line filter removes them. And a reasoning-capable checkpoint sends every token to the hidden channel on a prompt this short, leaving the content channel legitimately empty on a healthy run.

The step now keeps the streams in separate files, passes --show-reasoning, and counts only the text between the Generating... banner and the summary line. Verified against three logs before it went in: a good run passes, a run whose decode loop finished with no text is rejected, and a log with no ROCm backend line is rejected. It also asserts that no CUDA compute capability is printed, so the #1805 defect cannot come back unnoticed.

The job is parked, not left to queue

The first version of this PR targeted the self-hosted runner unconditionally, and opening it showed why that is wrong. GitHub does not skip a job whose runner is absent, it queues it, for up to 24 hours. While it sits there the entire workflow run stays in progress, so no job in that run has readable logs, and the check reads as "still running" to a reviewer. Every PR touching the HIP overlay would have paid that until a runner appeared.

So rocm-build is gated on vars.ROCM_CI_ENABLED == 'true' as well as the path filter. It is genuinely inert until someone registers the runner and sets the repository variable, at which point one flip activates it with no further change here.

Parking it silently would be its own trap: a reviewer would see no ROCm check at all on an overlay change and reasonably conclude one ran. ROCm CI status always reports which state the gate is in, and carries administration: read so it can say whether the runner is already registered rather than only "cannot tell". It never fails a PR, because whether a runner is attached is an infrastructure fact and not a defect in the change under review.

Documentation

Host requirements for the runner are in this workflow's own comments, next to the job, which is where the GB10 runner's are. docs/installation.md is for building mlxcel, not for operating CI.

It does correct what merged work had already invalidated there. Three status rows: affine MoE no longer needs MLXCEL_FUSED_MOE=0 (#1803), memory estimation on UMA hosts is correct rather than broken (measured in #1805), and the diagnostics report the AMD device rather than a CUDA compute capability (#1883). And three places that still told a reader to set that flag: the documented mlxcel-server command, the troubleshooting entry for the CUDA-backend abort, and the throughput paragraph.

Both flag removals were verified rather than inferred from the merge. mlxcel generate on Qwen3-30B-A3B-4bit runs without it, and the documented scripts/server_chat_smoke.sh passes against a server started without it:

health          200 {"status":"ok"}
models          200 Qwen3-30B-A3B-4bit
chat            200 finish_reason=stop channel=content text='Paris.'
chat stream     200 frames=221 done=True finish_reason=stop channel=content text='...'
OK

The 55 tok/s figure in the throughput paragraph is left as it stands, with its provenance stated, rather than replaced by my own 24-token ad-hoc run at 49 tok/s, which is not a comparable measurement. The troubleshooting entry now names the one case that still aborts this way, the sampler entry points in #1885.

Not done here

Registering the runner needs org admin, which this account does not have: the repository has no self-hosted runners and the GB10 runner is org-level. Two steps activate the gate once a runner exists: register it with the labels self-hosted and rocm-gfx1151, then set the repository variable ROCM_CI_ENABLED to true.

Until then three of the issue's acceptance criteria stay unchecked, namely that a PR touching the overlay builds and generates, that the log shows a GPU run, and that a deliberately broken overlay file fails the job. The fourth, that unrelated PRs do not trigger it, follows from the path filter and is visible on this PR. The issue stays open for those.