[jonathan308]

Progress 2026-10-22 09:00 PDT: sealed reduce-scatter and all-gather

reduce_scatter_sealed and all_gather_sealed for two ranks seal and authenticate every piece through the same codec as all_sum_sealed. Every sealed collective now draws its counter from one process-wide sequence; a graph of all-sums only gets the same values as before.

On the two-Mac pair over JACCL:

• Results: reduce-scatter equals this rank's rows of the sealed all-sum, and all-gather equals the padded all-sum, bit for bit. Checked at five shapes (f32, bf16, f16, up to 8192 x 4096).
• Tamper: a flipped byte in a peer's piece is refused on both ranks for every kind: all-gather at counters 0 and 200, reduce-scatter at 2 and 202, all-sum at 1, 3, 201 and 203.

The first user, GLM sequence-parallel prefill, is bitwise but slower on the pair, so it stays off; the collectives are kept for it and later uses.

Progress 2026-10-21: rebased onto the latest main/master, candidate 24

Head 79d2ebe8.

• Merged the single-Mac core (Single-Mac GLM-5.3-Flash native fixes without JACCL #37, now on main as ad4745a2). Its general commits are cherry-picks of this branch's, so the merge keeps this branch's tree, adds main's [Metal] thread-scoped command-buffer commit limits #36 (thread-scoped command-buffer limits), and records main's squash.
• Reverted 4d913204 (page-backed footprints) explicitly, as the single-Mac core did. It charged GLM MTP requests about 11.3 GB instead of 4.1 GB.
• fork.yaml now describes the host cuts and the streamed sealed sum; the fork-diff gate passes with 233 of 233 changed files described.
• The merged core's metallib is byte-identical to candidate 20's (79567e78).

Verification (two M5 Ultra 256 GB Macs).

• Pair test tool: tokens and top logits identical to the previous verified build, candidate 20, on the 4K plain, 4K MTP-depth-2 and 16K reference runs.

• Cluster check suites: 30 of 37 pass. Six failures were already present on candidate 20 (stale check-runner file lists, the GPT-OSS stage check, a 32K qualification test). PlacementChecks needs its tests updated for the memory change.

• Endpoint (candidate 24, tensor parallel, chosen by the planner):

  Measure	Candidate 24	Candidate 20
  Prefill, 2.7K / 10.5K / 31.5K tokens (tok/s)	2,966 / 3,498 / 3,526	2,951 / 3,511 / 3,512
  Decode, 256 / 1,024 tokens (tok/s)	87.4 / 88.1 (92.1 / 86.2 over 5 reps)	87.9 / 85.5

  Prefix reuse returns the first token in 0.87–1.25 s. Smoke checks pass (391, stop string, length).

Known gap, being fixed next: the cluster serves one request at a time. Aggregate throughput stays at about 80 tok/s from 1 to 16 concurrent streams, and the other streams queue. Concurrent serving with real batching, for every model in both TP and pipeline, is the next change on this stack.

---

Single-Mac split, 2026-10-20

The general (non-JACCL) commits of this branch are also on Layr-Labs/mlx#37 (work/glm53-flash-release → main), cherry-picked with their (cherry picked from …) trailers. Single-Mac GLM-5.3-Flash (d-inference #1516) now pins that chain instead of this one, so it can merge first. This branch is unchanged. Once the single-Mac PR merges, this stack merges main (fork.yaml needs a manual resolution) and the two-Mac pair is re-verified bitwise.

---

Summary

Adds sealed two-rank all-sum and repairs quantized Metal kernels used by the cluster engine. Each partial result crosses the link in a frame sealed by the caller's codec. Sorted NAX gathers bound partial activation tiles, and legacy/NAX affine/FP loaders have distinct symbols and bounded weight reads. Scalar CPU quantized matmul now accumulates in float instead of rounding every partial sum to BF16/FP16. An optional install-once Metal backing cap reserves pending bytes before allocation and returns credit only after Metal releases and unmaps the backing; residency-registration failure also cleans up ordinary allocations.

Capped no-copy backing is prefaulted before Metal receives it. JACCL adds a one-shot bounded startup wait, with refusal and failure cleanup preserved. Custom Metal kernels validate the declared launch geometry and may retry pipeline compilation with an explicit 1024-thread contract; tensor math and dispatch dimensions are unchanged.

The detached Metal footprint policy includes a VM-page minimum, so a capped 384-byte receive with 16,384 bytes of backing fits its conservative 32,767-byte bound. Actual backing allocation, cap accounting and ownership checks are unchanged.

Stacked on #33, with its current target integrated. The Swift and provider consumers must repin this revision; their release and real-model RDMA qualification remain separate.

Before

flowchart LR
  T["Two-rank partial results"] --> P["Plain collective payload"]
  G["Sorted quantized gather"] --> K["NAX tail reads beyond remaining K"] --> B["Wrong values or NaNs"]
  A["Offline Metal link"] --> L["Different loaders share a symbol"] --> B
  Q["CPU quantized product"] --> R["Low-precision partial sums"] --> B
  M["Soft memory limit"] --> U["Backing can exceed an experiment allowance"]
  F["384-byte capped receive"] --> B0["Detached bound: 767 bytes"] --> X["Valid page-backed receive rejected"]
  J["Startup collectives"] --> W["Ordinary idle timeout during load"]
  H["Required 1024-thread kernel"] --> R0["Compiler-selected pipeline capacity too small"]
  classDef default fill:#fff1f2,stroke:#be123c,color:#881337

Loading

After

flowchart LR
  T["Two-rank partial results"] --> C["Caller codec"] --> S["Sealed all-sum frames"]
  G["Sorted quantized gather"] --> K["Bound rows, packed columns and remaining K"] --> V["Exact valid outputs"]
  A["Offline Metal link"] --> L["Distinct loader and helper identities"] --> V
  Q["CPU quantized product"] --> R["Float accumulation; one output cast"] --> V
  M["Opt-in cap"] --> P["Reserve backing before allocation"] --> D["Refund after actual deallocation"]
  F["384-byte capped receive"] --> B0["Detached bound: 32767 bytes"] --> X["Backing covered; ownership checks retained"]
  J["Startup collectives"] --> W["Fixed one-shot deadline then ordinary idle policy"]
  H["Validated 1024-thread kernel"] --> R0["Bounded pipeline retry; same geometry and math"]
  classDef default fill:#ecfdf5,stroke:#047857,color:#064e3b

Loading

Evidence

Current successor qualification

Current signed head is 54975cd; C #15 and Swift #56 now select this revision. The actual target base remains 56a2043.

• Before: the actual custom-kernel evaluator regression rejects required full-size launches when the initial Metal pipeline exposes insufficient capacity. After: 33 evaluator checks pass under UBSan; invalid geometry still refuses. On the M3 Ultra, 17 actual Metal bounds/dispatch checks and all 21 GLM dtype/geometry cases pass with the required threadgroup size unchanged.
• Sixteen separate fresh-process allocator/cap modes pass, including physical footprint growth, retained backing, concurrency, foreign buffers, GPU evaluation and registration failure. The footprint assertion now rejects a decreasing counter before unsigned subtraction; five arithmetic controls cover that regression.
• Startup policy: 23 ASan/UBSan controls and seven real mesh/ring dispatch controls with simulated verbs pass. The actual C bridge passes 17 refusal cases plus valid forwarding. Simulated verbs are not real RDMA proof.
• The source-matched SDK composition passes 16 focused stage/native/MTP-fixture tests, including the new prefill readout and cache/failed-ack behavior. This is component evidence, not clustered MTP support or whole-model performance qualification.
• Configured hooks pass; actual-HEAD fork coverage is 225/225. Independent source/spec/refactor review found no remaining scoped defect; public core production bytes match the tested canonical nested source.
• Current hosted lint, fork, ASan, UBSan and both Linux Fedora jobs pass; Metal CI is still running at the latest readback. Swift consumer CI has a build failure under investigation. Full current consumer suites, final installed-pair qualification and applicable modalities remain open. No release-readiness claim.

Historical page-backed subpage evidence

• Before: the real worker receive regression fails four assertions at 256/384 bytes: actual capped backing is 16,384 bytes, exceeding bounds 511/767; the production owned-receive validator refuses it. The standalone real capped allocator independently reproduces the 256-byte mismatch. A missing-metallib setup failure is recorded separately from those functional failures.
• After: the same worker regression passes (one test, zero failures/skips), including real shared/offset-view refusal. Fourteen fresh-process native allocation/cap controls pass: ordinary/capped footprint, install, retained, churn, foreign, GPU, late, nil-foreign, ordinary/capped registration, foreign-registration failure and concurrent reservations. The formatted public standalone test was freshly compiled and is byte-identical to that tested executable, with the same native object and AOT library; no duplicate GPU run is claimed.
• Both installed ranks subsequently complete the authenticated encrypted GLM data-plane exchange over JACCL. Loading still refuses under file-cache/compression pressure. This establishes the receive-bound correction, not full-model inference, multimodal serving, endpoint readiness or a serving grant.
• All five configured full-file pre-commit hooks pass; fork coverage is 214/214 at the actual upstream merge-base. Independent SOURCE/SPEC/REFACTOR review found no actionable findings and justified no refactor. Swift consumers must repin the public successor.
• Historical CI observation: preceding head 9e22e1998a3b6f7f6f4ad58dc2038ecf9f4d0cb9 failed CPU test_double.TestDouble.test_gemm at its float32/float64 allclose assertion. The hosted log is now available; its cause remains unresolved. See the current successor CI disposition above; no tolerance relaxation or skipped check is included.

The following larger suites describe earlier compositions and unchanged component coverage; they are not a fresh full-suite run of this successor.

• Earlier combined cap/CPU composition, M3: 12 fresh-process Metal cap cases and 1 Swift cap case pass; full native Swift/Cmlx 931 passed, 2 expected skips, 0 failures (74.75 s). Source, native binary, embedded JIT and AOT identities were unchanged during qualification. The skipped cap case passes separately with its required fresh-process opt-in; the other skip is the opt-in performance test.
• CPU numerical regression: BF16 produced 592 instead of the exact 576. Exact-value regressions reproduce six affine and fifteen adjacent mode/orientation subcase failures, then pass with float accumulators. Full local Debug/coverage CPU workflow: 292 C++ passed; 815 Python passed, 62 declared skips, 0 failures across 33 modules. Python 3.14 and the local compiler differ from hosted CI; this is a confirmed local defect, separate from the now-accessible hosted CPU GEMM failure above.
• Cap failure and concurrency controls: retained buffers stay charged, pending real-Metal reservations cannot exceed the cap, cache transfers preserve credit, failed registration rolls back, and failed foreign wrappers cannot silently copy. M5 cap production passed 12 cases plus its full native suite before the CPU-only delta; this is distinct earlier-composition evidence.
• At that earlier composition, pinned hooks and core fork coverage passed at 213/213. Independent reviews cover allocator lifetime, CPU arithmetic, failure cleanup and concurrency. Core, C and Swift consumer pins must move together.

Earlier kernel/execution evidence below remains applicable to unchanged kernel sources; it is not an exact current M5 combined-binary claim.

• Tail regression: the old Swift composition failed 48 of 144 geometry cases. The repaired regression passes 272 cases across affine 2/3/4/5/6/8-bit, MXFP4/MXFP8/NVFP4, packing groups, FP32/FP16/BF16, partial tiles and adjacent NaNs.
• AOT regression: the baseline fails 32 subcases; the tail-only repair still fails 16 MXFP4 subcases. Renaming only the FP NAX loader changes NaNs into the exact expected outputs with an unchanged Python module. The final separation of all incompatible loaders passes the 128-case Python regression and all 39 quantization tests, using the real precompiled Metal library on M5 Max.
• Legacy M3 regression: published core 1fb86d3 fails 26 exact NVFP4 partial-K assertions. This successor fixes legacy row/column bounds; the same focused tests pass without assertion changes. Final M3 native suite: 931 passed, one opt-in performance skip, zero failures.
• Swift integration: the final source-matched M5 native suite passes 930 tests, with 2 explicit skips (opt-in performance and non-NAX-only retract coverage), zero failures. The final AOT library passes all 39 Python quantization tests on both M3 and M5; the targeted 128-cell regression also passes on M3. Expert-tile kernels are enabled and the resource-count limit is 50,000. All four generated kernel strings are verified present byte-for-byte in the tested executable. Broader SDK qualification continues in the consumer PR.
• Configured pre-commit checks pass. Fork-diff coverage: 209/209 changed files described at that earlier head against the actual upstream merge-base. Independent review confirmed tail bounds and identifier-only loader separation; no dispatch disablement, precision change or tolerance relaxation.
• Existing sealed all-sum hardware evidence remains the two-Mac runs linked from d-inference #1407. These new component results do not establish GLM RDMA or installed-endpoint readiness.

Merge Danger

Door: two-way. No storage migration or deployment.

Blast Radius: native. Prefaulting changes when capped pages become resident, and the new startup deadline and bounded pipeline retry affect native execution. The conservative detached minimum changes quotations for subpage buffers; it does not change physical allocations or cap enforcement. Quantized Metal loader identities and partial-tile reads change; consumers must refresh both their core pin and embedded JIT sources. CPU scalar accumulation precision also changes. The cap limits owned Metal backing and conservatively charged foreign wrappers, not whole-process/host/driver memory; host budget and OS headroom remain separately required. Cap installation is opt-in, once-only, and must precede positive buffer attempts. Formatting-only updates to existing sealed sources satisfy the configured formatter.

Implementation and regression investigation used Codex, with an independent agent review. The original sealed collective was prepared with Claude Code.

512-thread compiler-capacity follow-up

A GLM fused KDA test requested 512 threads on a pipeline compiled with a 384-thread limit. The existing retry covered only 1024-thread groups. The core now retries complete 512- or 1024-wide groups with an explicit compiler-capacity declaration, preserving shader math and dispatch geometry. Device bounds and the selected pipeline’s actual capacity still reject unsupported launches.

• Before: SDK native CI failed in GLM5NextLinearDecodeKernelTests/decodeStepMatchesComposedStepOnGPU with the 512-versus-384 capacity error.
• After: the source-matched native consumer suite passed 32 tests across 8 suites, zero failures or skips, including the actual GLM linear-decode kernel suite. The CPU regression using the evaluator’s actual retry block changed from failing to passing across five retry/rejection cases.
• Normal formatting and fork-description checks pass. Python launch geometry coverage includes 512; that added Python case was not separately executed. These are native correctness results, not an isolated performance benchmark or a claim that all SDK CI is green.

Update 2026-10-20: fast-synch GPU wait ordered after pending passes (f0db8cf), 1 MiB JACCL frames (61fd8a8)

• Hang fix. A GPU pass that spins on a CPU-stream fence (MLX_METAL_FAST_SYNCH=1) had no dependency on earlier passes still in flight on its queue. Without hazard tracking Metal could start it first, and when an earlier pass was what the awaited CPU work needed (GPU → CPU → GPU, the sealed all-sum between two layers) the GPU spun forever at 100 %. Traced on two M5 Ultra with a fence/command-buffer ring: a committed buffer of plain elementwise ops stayed scheduled, not started while the later spin ran. Fence::wait now marks the spin's pass CommandEncoder::wait_for_all_pending(), which waits on every pending pass's fence of that queue; the CPU work it waits for already follows them.
• Sealed sum add split across cores (dispatch_apply, 256K-element spans), addends still in rank order (bit-identical). 32 MiB fp32 sealed sum 20–25 ms → 5.9 ms.
• JACCL largest frame 512 KiB → 1 MiB (BUFFER_SIZES 9), mesh pipeline from MESH_PIPELINE (2, static_assert). 16 MiB bf16 all-sum 3.02 → 2.69 ms. 2 MiB frames and a pipeline of 4 fail in the driver.

Result on GLM-5.3-Flash TP2 (sealed, two M5 Ultra): prefill 1,913 tok/s, decode 50.7 tok/s (MLX_METAL_FAST_SYNCH=0 instead: 13 tok/s decode). Native unsealed TP2 unchanged (2,400 / 56.3).

🤖 Generated with Claude Code

Update 2026-10-20 (evening): host-encode ports (872fa01, 5711fa5), streamed sealed all-sum (8c6f9e7)

All three commits are Verified; nothing is merged.

Host time (ports of the owner's mlx-m5u work):

• 872fa01 (mlx-m5u 8c4bf63fe): each command buffer retains its primitives' buffers in one completion handler, not one per primitive.
• 5711fa5 (mlx-m5u 42f2dcefb): a custom kernel's generated source and library name are built once per kernel and template instance.
• GLM-5.3-Flash TP2 decode on two M5 Ultra: 16.75 -> 15.92 ms a step.

Streamed sealed all-sum (8c6f9e7). On a two-rank JACCL mesh, SealedAllSum now runs one CPU-stream task over a new MeshImpl::exchange:

• Each 256 KiB sealed piece carries its own AES-256-GCM tag.
• A piece is sent as soon as it is sealed, and opened and added as soon as it lands.
• Only the first and last pieces wait for crypto.
• Ring, TCP and larger groups keep the v1 path unchanged.

flowchart LR
  subgraph Before["v1: seal, gather, open, add"]
    S1[seal whole frame] --> G1[all_gather 2n] --> O1[open whole frame] --> A1[add]
  end
  subgraph After["v2: per piece, pipelined"]
    S2[seal piece k] --> X2[exchange frame] --> O2[open piece k] --> A2[add piece k in cache]
    S2 -. "piece k+1 seals while k crosses" .-> S2
  end

Loading

Property	How it holds
Every byte authenticated	One AES-GCM box per piece under its direction and generation key. The receiver derives the piece count and lengths from n and the piece size, never from the wire. The sum throws unless every piece opened.
Unique nonces	(piece, sender, counter); each piece is sealed once, never resealed.
v1 and v2 kept apart	v2 authenticated data = v1 authenticated data plus the piece size, so neither layout's piece verifies as the other's.
Fail closed, no hang	A piece that does not open latches the codec. A refused seal sends zeros, so both ranks refuse in lockstep.
Bit-identical results	Same per-element expression and rank-0-first operand order as v1.

v1 fix in the same commit. A refused seal now zeroes its frame before it throws. Before, the following gather could send a recycled buffer's stale bytes in the clear.

Measured on two M5 Ultra over Thunderbolt 5 (GLM TP2, sealed bf16 partials, 15.7K-token prompt, 8,192-token chunks):

• Prefill rose from 3,554 to 3,812 tok/s, with output tokens identical.
• A tampered piece at sealed counter 0, and again at 200, was refused on both ranks. Both ranks ended without hanging.

tests/jaccl/exchange_tests.cpp uses fake verbs with the real MeshImpl and streamed-sum code. It passes under ASan/UBSan and TSan, covering:

• sizes from 1 B to 3 MiB;
• bitwise equality with v1 for fp32, bf16 and fp16 over 1–65 pieces;
• tamper, seal failure and a closed group.

For platform review. Callers still name the reduction sealed_all_gather_sum_v1 (Qwen pair qualification records, ClusterTensorParallelTerms). The results are bit-identical, so existing qualification records stay valid. The wire layout is bound by the authenticated data and by the build SHA in each load agreement.

🤖 Generated with Claude Code

Update 2026-10-21: host-time cuts and opt-in sum plumbing (6034a78 … 7620956)

Six Verified commits; nothing merged. Every change keeps model outputs bit for bit (GLM-5.3-Flash TP2 on two M5 Ultra: tokens and top logits identical).

Commit	Change	Default	Measured
6034a78	Fused coherent fence update for CPU-stream inputs (MLX_METAL_FUSED_FENCE_UPDATE)	off	≤ 0.1 ms/step
a7d4079	Fenced inputs for custom kernels (MLX_METAL_FENCED_INPUTS): the consumer waits on the fence itself, after all pending passes (the fast-synch deadlock ordering is kept)	off	≤ 0.1 ms/step
7ebf8e1	Small flat sets for per-dispatch encoder hazard tracking (same barrier/commit/fence decisions, replayed on 100 traces)	on	~0.2 ms host/step
9ec3dba	Eager commit after a cross-device fence update (MLX_EAGER_FENCE_COMMIT)	off	neutral
a94584b	Separate CPU-stream task budget (MLX_MAX_ACTIVE_CPU_TASKS)	off	neutral
7620956	Shared generated sources, pipeline resolved once, repeat launches skip name/template/key rebuilds, flat map in eval_impl, cheaper graph teardown	on	with mlx-c/mlx-swift: one-rank host CPU per decode step 6.5–6.9 → 4.7–4.8 ms

Not compiled here: the CUDA side of the shared-source change (no CUDA toolchain on these Macs).

🤖 Generated with Claude Code

Progress 2026-10-26: main merged, stack re-pinned, validated on two M5 Ultra 256 GB

Head 2b5253ad: #33 and main merged; Check Lint fixed (95938c4b); the streamed-sum exchange test compiles again (2b5253ad).

• Main merged with signed non-force merges, parent to child; each new head contains the previous head, so the push is a fast-forward. Conflict resolutions, checks and evidence: MLX-MERGE-2026-10-26.md (core), MLX-C-MERGE-2026-10-26.md, MLX-SWIFT-MERGE-2026-10-26.md, MLX-SWIFT-LM-MERGE-2026-10-26.md (kept with the overnight records).
• Core (JACCL: zeroed send-frame tails, a progress guard, the advertised GID and the upstream JACCL fixes #33/Cluster collectives, quantized numerical fixes and optional Metal allocation cap #35/Fix Metal fence ordering and sealed collective portability #41): JACCL keeps the send-frame zeroing, progress guard and advertised-GID fix on every cable of main's multi-cable mesh; the ring's moved wire dispatcher now reports errors from extra cables (187b6c67, test fails without the fix). Cluster collectives, quantized numerical fixes and optional Metal allocation cap #35 carries Fix Metal fence ordering and sealed collective portability #41's barrier line so it does not undo main's [BUG] Metal fence deadlock ml-explore/mlx#4689. Main's value-based fence waits are taken; the cluster keeps its single spin ordering. Upstream JIT bug fixed (ce21a37d): head-dim-256 NAX attention failed to build its run-time template on M5. Fix Metal fence ordering and sealed collective portability #41 adds MLX_QMV_BATCH_INVARIANT (caa9ed53, off by default): quantized products of 2 to the vector limit - 1 rows stay on qmv_wide, so a row's bits do not depend on how many rows share its launch (main's [Metal] Add qmv_mma/qmv_nax kernels for 6-16 rows with large N/K ml-explore/mlx#4712/[Metal] Add staged affine_qmm_t_nax kernels with split-k ml-explore/mlx#4682 routes otherwise change them). Cluster processes turn it on; single-Mac keeps main's kernels.
• mlx-c (Expose coherent allocator accounting and correct quantized bias sums #15): main merged; builds against the merged core with its test.
• mlx-swift (Docker: xcrun: not found ml-explore/mlx#54/Simple test to compare speed against PyTorch on MPS ml-explore/mlx#56/random.uniform with dtype float16 returns all 0s ml-explore/mlx#61): main merged; nested pins restitched (Source/Cmlx/mlx = the matching core layer, Source/Cmlx/mlx-c = mlx-c Expose coherent allocator accounting and correct quantized bias sums #15, Docker: xcrun: not found ml-explore/mlx#54 on main's C); generated sources regenerated with tools/update-mlx.sh and compared; random.uniform with dtype float16 returns all 0s ml-explore/mlx#61 pins core caa9ed53 (only include-framework/mlx-utils.h changes). Fork Tests (all three shards with an AOT metallib): XCTest 849/853 + 2 permitted skips, Swift Testing 749/749.
• SDK (Add Makefile for python related development ml-explore/mlx#308/simple numpy helper for tests ml-explore/mlx#352): main merged (swift-format 603 layout, fork.yaml, Metal-library and tensor-unit test gating, admission-test repair); Add Makefile for python related development ml-explore/mlx#308 pins mlx-swift Simple test to compare speed against PyTorch on MPS ml-explore/mlx#56, simple numpy helper for tests ml-explore/mlx#352 pins mlx-swift random.uniform with dtype float16 returns all 0s ml-explore/mlx#61. simple numpy helper for tests ml-explore/mlx#352 also carries the cluster beta's SDK changes (GLM DSA/NextN history readers for the GLM TP SSD tier, unserved content parts refused, checkpoint row codec for SSD records, Qwen TP on-demand page growth, Bonsai FP32 residual stream fix, Flint precision fix for master How to make an mlx array containing every number from 0-100 and skipping every Nth number ml-explore/mlx#1554) and the two single-Mac GLM speedups (64-head KDA prefill cut, early MTP drafts; bitwise on the real model).
• Numerics: main's precise sigmoid (Use precise::exp in Sigmoid so compiled and eager sigmoid agree ml-explore/mlx#4461), split-K and SDPA changes move GLM-5.3-Flash's bits versus the previous cluster build; on four natural documents the pair's perplexity is lower on every one (−0.16 … −1.16 %). c1 = cN byte equality holds on the pair at c1-c16 with the switch on.
• Pair validation: candidate c48 (d-inference Fix OOB access in qmv ml-explore/mlx#1577) on two Mac Studio M5 Ultra 256 GB; see Fix OOB access in qmv ml-explore/mlx#1577 for the gate table.

Nothing merged or released; the owners sign and release.

Update 2026-10-26 17:15 PDT: main/master 0.10.3 merged through the stack

Head 424562b9: the refreshed #33 merged (brings main 6bc9de5b). Every PR in the cluster stack (mlx #33 → #35 → #41, mlx-c #15, mlx-swift ml-explore#54 → ml-explore#56 → ml-explore#61, SDK ml-explore#308 → ml-explore#352, d-inference ml-explore#1407 → ml-explore#1426 / ml-explore#1577, platform ml-explore#54) is again mergeable in that order, as drafts.

Review fixes and known limit (2026-10-27)

Head 169802f4: review fixes in be4b55b1 (fenced custom-kernel inputs stay in the device address space, fenced waits are taken before any launch check can throw, sealed frames over INT_MAX bytes are refused, codec contract documented in sealed.h), #33's fixes merged in, and the clang-format fix 169802f4.

Known limit (for David to rule): if one rank fails before it takes its sealed counter and the other rank does not, the counters desync and every later sealed operation fails authentication. This fails closed, but nothing recovers the group; the group must be rebuilt.

Known limit (for David to rule): MeshGroup::end_startup_wait_budget ends the budget wire by wire. Each wire's deadline is set a moment after the previous one, so at the very edge of the deadline wire 0 can end while wire 1 throws "Cannot end an expired startup budget". The group is then partly ended: wire 1 still enforces the expired deadline on its next call. This existed before the shared failure flag; it was not changed here.

CI fix 94baa025 (tests only, no library change): progress_guard_checks runs with a 2000 ms limit instead of 250 ms (every case scales with the limit, ratio kept), and the startup expiry cases spin their polls. The abort in startup_wait_budget_groups was the fake poll's 8 s safety deadline, not a crash in the library.

[davidtai]

Review fixes in be4b55b:

• Fenced custom-kernel inputs are always in the device address space. A 0-d or small input was constant, and the GPU may load a fixed-size constant buffer before MLX_FENCED_INPUT_WAIT runs.
• CustomKernel::eval_gpu takes its fenced waits before any launch check can throw. Before, a throw left the wait in the thread-local list for a later kernel at the same address.
• Sealed frame lengths go through detail::frame_extent, which refuses frames over INT_MAX bytes instead of wrapping the int shape. Unit test added.
• sealed.h documents the codec contract: the counter restarts at 0 in each process, so keys must not be reused across processes, and the codec ctx must outlive the graphs built with it.

Not changed (design question): if one rank fails before it takes its counter and the other rank does not, the counters desync and every later sealed op fails authentication. This fails closed, but nothing recovers it.