chore(rocm): mlxcelverse ROCm fork sync script, MLX pin-bump procedure, and upstreaming local fixes #1813

Part of #1801. Phase 3. Depends on #1802 (closed, landed in PR #1818).

Status (2026-09-29, origin/main d8d34e2b)

• Exists: src/lib/mlx-cpp/patches-rocm/UPSTREAM and LOCAL_FIXES.md (PR feat(rocm): vendor the ROCm backend into mlxcelverse, add rocm feature #1818, kept current by every overlay change since), a short pin-bump procedure in src/lib/mlx-cpp/patches-rocm/README.md:18-24 (PR feat(rocm): vendor the ROCm backend into mlxcelverse, add rocm feature #1818) and a ROCm paragraph in CONTRIBUTING.md:150 (PR docs(rocm): document the experimental AMD ROCm build and mlxcelverse #1819).
• Does not exist: scripts/mlxcelverse/ (neither the sync script nor the drift check), any upstream PR against NripeshN/mlx, and any upstream link in LOCAL_FIXES.md. Nothing under scripts/, Makefile or docs/ automates the fork sync; docs/installation.md and docs/architecture.md only describe the overlay.
• The overlay's recorded source is still NripeshN/mlx@rocm-support commit 75915908dfe5028335d318b10340313744fd3a8d, fork merge base 39886de4, retargeted to the MLX pin 81ba1c6a, which is still the current pin (UPSTREAM; scripts/ci/mlx_pinned_commit.sh).

Context

The ROCm part of mlxcelverse has two moving upstreams:

1. ml-explore/mlx, which mlxcel bumps regularly. Each bump can change the 15 core overlay files and the APIs the 106 backend files compile against (108 on d8d34e2b: hadamard.hip and fft.hip were added locally; patches-rocm/README.md:9 still says 106). The retarget in epic: AMD GPU (ROCm) backend on Linux via mlxcelverse #1801 needed six fixes after 360 upstream commits (a host-only isnan template hiding device overloads, a changed return tuple, constructor argument changes, a new force_fused parameter, new primitives needing NO_GPU stubs, and Event::error). Because libmlx is static in mlxcel, a missing eval_gpu shows up at link time.
2. NripeshN/mlx@rocm-support, where ROCm kernels improve quickly (dozens of commits in July 2026 alone: WMMA flash attention, MoE, allocator).

Whole-file overlays carry a known review risk: the existing bump check (the overlay's +/- line counts against the new upstream base must equal those against the old base) does not catch an overlay that was already wrong before the bump. A Metal overlay once carried only half of an upstream fix for months until a reviewer compared it line by line.

Local fixes made while vendoring should flow back to the fork so the overlay does not drift from its source.

Inventory: overlay changes since PR #1818

git log origin/main -- src/lib/mlx-cpp/patches-rocm/ shows five commits after the vendoring commit bec64748 (PR #1818). All of them touch only mlx/backend/rocm/ (plus LOCAL_FIXES.md), none of the 15 core files, and each is recorded in LOCAL_FIXES.md as "Applies to the fork; to be proposed there". These, with items 8 to 11 that arrived with PR #1818 itself, are the upstreaming backlog and the fix list the sync script must re-apply.

• b3a2226e PR perf(rocm): route f16 gather_qmm to the warp-shared kernel #1822: f16 activations routed to the warp-shared gather_qmm kernel (quantized/qmm.hip; LOCAL_FIXES.md item 12).
• 20cf77ce PR fix(rocm): stop the tiled qmv path from skipping f32 activations #1823: f32 activations no longer skip every launch in the tiled qmv path (quantized/qmm.hip; item 13).
• 39c597b0 PR fix(rocm): make scatter apply every update, check kernel arg sizes #1824: Scatter passes upd_post_idx_size as int64_t, and add_kernel_node statically checks argument sizes (indexing.hip, device.h; item 14).
• 9ead2b7d PR feat(rocm): implement Hadamard and SearchSorted, fix narrow index dtypes #1856: SearchSorted and Hadamard implemented, narrow Gather/Scatter index dtypes widened to int32 (sort.hip, new hadamard.hip, indexing.hip, primitives.cpp, CMakeLists.txt; items 15 to 17; closed fix(rocm): Gather reads narrow index dtypes as int64 and faults the GPU queue #1853).
• 4641dc7a PR feat(rocm): implement FFT through hipFFT #1861: FFT implemented through hipFFT with an 8-entry plan cache (new fft.hip, primitives.cpp, CMakeLists.txt; item 18). The cap works around a hipFFT hang whose root cause is unknown, tracked in fix(rocm): hipFFT blocks when too many plans are alive, root cause unknown #1876; say so in the upstream proposal.

Items 1 to 7 of LOCAL_FIXES.md are retarget-only and follow the fork when it merges newer MLX.

Scope

Tooling and written procedure so that both kinds of update are routine, plus sending the local fixes upstream.

Implementation plan

1. scripts/mlxcelverse/sync_from_fork.sh <fork-commit>: fetch the fork, copy mlx/backend/rocm/ into patches-rocm/, re-apply the local fix list, and for each of the 15 core files produce the fork's diff against its own merge base so it can be 3-way merged (git merge-file) into the current overlay. Update patches-rocm/UPSTREAM. Not started. The local fix list now covers LOCAL_FIXES.md items 2 to 18, including two files the fork does not have (hadamard.hip, fft.hip), which the script must carry rather than delete.
2. scripts/mlxcelverse/check_api_drift.sh: build the ROCm tree at a candidate pin and list undefined mlx::core symbols and compile errors, so a bump PR sees the ROCm breakage immediately. Not started.
3. Extend the MLX pin-bump procedure in CONTRIBUTING.md (lines 127-153 now CONTRIBUTING.md:127-150) and the technical report template referenced there with a ROCm section: 3-way merge each core overlay, compare each overlay line by line with upstream (not only line counts), re-run the ROCm op checks and the smoke from ci(rocm): self-hosted gfx1151 runner that builds, links and smoke-tests ROCm changes #1811, and record the drift fixes in the PR body. Partly done: CONTRIBUTING.md:150 (PR docs(rocm): document the experimental AMD ROCm build and mlxcelverse #1819) and patches-rocm/README.md:18-22 (PR feat(rocm): vendor the ROCm backend into mlxcelverse, add rocm feature #1818) cover the 3-way merge, the line-by-line comparison, the ROCm build and link, NO_GPU stubs and recording fixes in LOCAL_FIXES.md and UPSTREAM. Still missing: running make verify-rocm (which includes the ci(rocm): self-hosted gfx1151 runner that builds, links and smoke-tests ROCm changes #1811 smoke, Makefile:804), recording drift fixes in the PR body, and the report section. The pin-bump section on origin/main references no report template; the precedent to extend is TECHNICAL_REPORTS/1772-mlx-pin-mxfp8-round-up-20260911.en.md section 3 (lines 61-82).
4. Keep a patches-rocm/LOCAL_FIXES.md listing every change relative to the fork commit, with the reason and an upstream link once sent. Done for the listing (PR feat(rocm): vendor the ROCm backend into mlxcelverse, add rocm feature #1818, 18 items as of PR feat(rocm): implement FFT through hipFFT #1861); no upstream links yet because nothing has been sent.
5. Open PRs against NripeshN/mlx (rocm-support) for the fixes that apply to the fork itself: the E8M0 uint8_t scale dispatch for mxfp4/mxfp8 qmv, and any kernel fixes from fix(rocm): surface ROCm GPU failures through Event::error instead of NaN or hangs #1804 and fix(rocm/quant): mxfp4 qmm hang and GPU quantize launch failure, with an affine 4-bit fallback #1808. Retarget-only fixes (the isnan qualification, API drift) go upstream when the fork itself merges newer MLX. Not started: no PR from this project exists on NripeshN/mlx. The candidate set is now LOCAL_FIXES.md items 8 and 10 to 18 (item 9 only after a root cause), per the inventory above.

Acceptance criteria

• Running the sync script against the current fork commit reproduces the committed patches-rocm/ byte for byte.
• The drift check lists the six known breaks when run against the fork's original merge base plus current upstream.
• CONTRIBUTING.md documents the ROCm part of a pin bump. (CONTRIBUTING.md:150, PR docs(rocm): document the experimental AMD ROCm build and mlxcelverse #1819, pointing to patches-rocm/README.md:18-22; the make verify-rocm step and PR-body record from plan item 3 are still to add.)
• LOCAL_FIXES.md exists; the scale-dispatch fix is proposed upstream with a link recorded. (The file exists, PR feat(rocm): vendor the ROCm backend into mlxcelverse, add rocm feature #1818; the upstream proposal and link do not.)

References

• CONTRIBUTING.md:127-150, TECHNICAL_REPORTS/1772-mlx-pin-mxfp8-round-up-20260911.en.md (lines 61-82)
• src/lib/mlx-cpp/patches-rocm/UPSTREAM, src/lib/mlx-cpp/patches-rocm/LOCAL_FIXES.md, src/lib/mlx-cpp/patches-rocm/README.md
• Fork: https://github.com/NripeshN/mlx/tree/rocm-support

Refresh log

2026-09-29

• Recorded what exists: UPSTREAM (fork commit 75915908, pin 81ba1c6a), LOCAL_FIXES.md and the overlay README procedure from PR feat(rocm): vendor the ROCm backend into mlxcelverse, add rocm feature #1818, and the CONTRIBUTING.md:150 paragraph from PR docs(rocm): document the experimental AMD ROCm build and mlxcelverse #1819; checked off the CONTRIBUTING.md criterion and marked plan items 3 and 4 partly or fully done.
• Recorded what does not: no scripts/mlxcelverse/ sync or drift script, no upstream PR, no upstream link.
• Added the inventory of the five overlay commits since PR feat(rocm): vendor the ROCm backend into mlxcelverse, add rocm feature #1818 (PRs perf(rocm): route f16 gather_qmm to the warp-shared kernel #1822, fix(rocm): stop the tiled qmv path from skipping f32 activations #1823, fix(rocm): make scatter apply every update, check kernel arg sizes #1824, feat(rocm): implement Hadamard and SearchSorted, fix narrow index dtypes #1856, feat(rocm): implement FFT through hipFFT #1861), all backend-only, which enlarges both the sync script's fix list and the upstreaming set.
• Corrected CONTRIBUTING.md:127-153 to :127-150, the report section to lines 61-82, and the backend file count (108, not 106); noted that the pin-bump section references no report template.