[jonathan308]

An ordinary fast Metal fence wait now follows earlier GPU kernels in the same concurrent pass. Its existing waits for previous command passes remain. This closes a missing dependency before the spinning wait without changing arithmetic, collective operations, or command-buffer limits.

This draft builds on core #35, branch darkbloom/sealed-all-sum, head 360d11b90dcf26fe1b0c9be1789d5cb9d2c49f80. Consumers are coordinated separately through mlx-swift #61, mlx-swift-lm #352, and d-inference #1577. No C API or shader changes are required.

Before

flowchart TD
  A["Earlier GPU producer in the current pass"] --> B["CPU stream work"]
  B --> C["GPU spinning wait"]
  D["Previous-pass dependencies only"] --> C
  C -. "No explicit current-pass barrier" .-> A
  classDef default fill:#e8f0fe,stroke:#174ea6,color:#102a43,stroke-width:2px;
  classDef warning fill:#fff4ce,stroke:#8a6100,color:#332500,stroke-width:2px;
  class C,D warning;

Loading

After

flowchart TD
  A["Earlier GPU producer"] --> B["Current-pass buffer barrier"]
  D["Previous-pass dependencies"] --> C["GPU spinning wait"]
  B --> C
  A --> E["CPU stream work completes"]
  E --> C
  C --> F["GPU consumer may proceed"]
  classDef default fill:#e8f0fe,stroke:#174ea6,color:#102a43,stroke-width:2px;
  classDef guard fill:#fff4ce,stroke:#8a6100,color:#332500,stroke-width:2px;
  classDef success fill:#d7f0df,stroke:#176b39,color:#103b23,stroke-width:2px;
  class B,D guard;
  class F success;

Loading

The standalone metal_fence_chain test builds 96 GPU→CPU→GPU layers with shared side roots, widths 1/2/3/4/8 and three repetitions. It checks every result and zero remaining request-array bytes after drain. CTest registers three separate processes with command-buffer limits 8/50/1024 and a 60-second timeout. The Metal Fork Tests job runs these registrations explicitly and rejects missing tests.

Validation and limits

• Old and candidate CPU executions each passed 15 cases and 9,123,840 exact comparisons.
• On M5 Max, all six old/candidate Metal processes passed: 15 cases per process at each of the three buffer limits. The old binary also passes; this synthetic test did not reproduce the installed Qwen27 stall.
• Both versions used the same 193 copied c11 Cmlx objects, identical regression object and matching metallib; only identically compiled fence.o differed.
• Five actual-child runner tests pass, including timeout, SIGHUP and forced termination with definitive reaping. Independent source/refactor review found no outstanding issue.
• Shared-code merge gate: mlx/backend/metal/fence.cpp also serves GLM. The GLM owner must verify its paired execution before merge; this draft does not establish GLM parity.
• The matching c12 signed app is installed on both Macs. Its normal Darkbloom setup automatically selects TP and completes singleton requests, but two automatic batching attempts hit LOAD memory admission refusals before wide execution. The prior native width3 stall has therefore not been retested. No causal fix for that stall or full-model/performance pass is established here.

Reproduction from a configured Metal build:

cmake --build build --target metal_fence_chain
ctest --test-dir build -R '^metal_fence_chain_' --output-on-failure
./build/tests/metal_fence_chain --cpu
uvx --from clang-format==21.1.8 clang-format --dry-run --Werror \
  mlx/backend/metal/fence.cpp tests/metal_fence_chain.cpp
uvx --from cmakelang==0.6.13 cmake-format --check tests/CMakeLists.txt
python3 scripts/check_forkdiff.py

Evidence: reviewed implementation patch e1c98427cedc685fd6d5b445623e9f4fe48a167aef497ecb10579c508770a7bf; compiled probe inventory 76b3c03f2a2773187a388871bb0a018711263910dc0a06e29eada89a22cc5705; M5 six-process summary ca7f846f14454c7948aa74c233e0dca79dc9b4a2ffc59e5b3cda24c187785747. Publication adds fork metadata, a narrow CTest CI invocation and formatter-only changes. Follow-up66edc771 formats six inherited paths; source bodies are unchanged, with one existing system-include reorder required by clang-format. Lint and fork coverage now pass. Four Linux jobs exposed an inherited unconditional Apple dispatch/QoS dependency at compile time. Commit9ec023f9 guards that platform code and adds a synchronous portable path; hosted Linux and Metal checks for this head remain pending.

Sealed collective portability

Apple dispatch/QoS/Blocks statements retain their existing order and behavior. Other platforms seal the same authenticated pieces synchronously before exchange, then open only complete received frames and apply the same rank-ordered addition. This introduces no worker pool, plaintext transport or arithmetic change.

flowchart LR
  subgraph Before
    L1["Linux sealed collective build"] --> L2["Unconditional Apple headers"] --> L3["Compilation fails"]
  end
  subgraph After
    N1["Platform selection"] --> A1["Apple: existing parallel callbacks"]
    N1 --> P1["Other platforms: synchronous callbacks"]
    A1 --> F1["Same authenticated frames and reduction"]
    P1 --> F1
  end
  classDef input fill:#dbeafe,stroke:#2563eb,color:#172554
  classDef fail fill:#fee2e2,stroke:#dc2626,color:#450a0a
  classDef fixed fill:#dcfce7,stroke:#16a34a,color:#052e16
  class L1,N1 input
  class L2,L3 fail
  class A1,P1,F1 fixed

Loading

Three CPU cases cover callback identity, complete-piece boundaries, exact output, failed-seal zeroing, rejected output and callback drainage after exchange failure. The deterministic fixture codec is not a cryptography test. On macOS, C++20/Werror execution passes 5,251 Apple-branch and 5,263 portable-selected assertions; the latter compiles without Blocks and has no Apple dispatch/QoS/Blocks symbols. Actual Linux headers/ABI and sanitizer execution remain hosted-CI gates. Fork coverage236/236, pinned formatting and independent source/refactor review pass. Patch SHA440b275ec0ea12903da1ca665b4b10135b0777e7e68518f07f659b83739e1e01. Downstream candidate c13 still uses the earlier reviewed4e2455b0 core; portability/default pin updates are separate from its Qwen test.

AI disclosure: implementation assistance, focused validation and independent agent review used OpenAI Codex. This is a maintainer-review draft prepared at the user's explicit request.

🤖 Generated with Claude Code

Current integrated hardware gate — 2026-10-23

Normal signed c16 TP2 completed through the installed Darkbloom command/services/API on M3 Ultra256GB + M5 Max128GB (recursive source SHA d6f6adcedca1aeacfc5265ec427e3ef913633f5b77629b8c61cbdaf4ba278000). Stable endpoint37PASS/0FAIL/1intentional skip, context6/6 through32K, media/structured-output/mixed30/30 and cancellation/survivor/recovery3/3 pass. All61 matching native scopes retire with zero in-use KV;13 reach actual width2 on both ranks. Initial before/after-plan exact outputs remain29PASS/8FAIL/1SKIP. Aggregate throughput at offered concurrency1/2/4/8 is13.26/13.56/13.52/13.43tokens/s at native capacity2. Cleanup and restored idle services pass, but final workers exit1 after input closes; graceful shutdown remains a separate open gate. No full model or beta release is qualified.

Provider follow-up #1577 at signed/Verified b845e27771cb7c355ce1baa35731f343a16fec2c retains typed eager-fence diagnostic forwarding and completed-decode width counters. c17 is installed on both Macs. Its bounded FAST1+EAGER1 diagnostic completed FAIL:11PASS/7FAIL/6intentional skips. Both settings are verified in both fresh native pairs; a real width-two scope then stalls with an M3 JACCL timeout at60001ms and M5 lost input. Original service-job bytes and healthy idle services are restored on both hosts. Eager commits alone do not fix the fast-path stall. The same c17 build is now running short default-policy1/2/4 checks to measure actual batch utilization. Defaults remain the c16 event-synchronization policy for batched Qwen27.

Installed TP MTP remains disabled. The prepared foundation compiles and passes selected CPU checks, but only speculates singleton; true multirow target verification, whole-cohort memory accounting and full-artifact acceptance remain unfinished. The normal c14 fast-path stall and c15 exact-output mismatch remain preserved. No tolerance, memory guard or capability gate is relaxed.

Core#41 at9ec passes7 hosted checks; inherited recoverable UBSAN diagnostics remain recorded. Installed/downstream core still uses4e's Apple fence change;9ec portability changes are not propagated. Swift#61 has3 passing checks. SDK#352 at4f still fails:3613tests/622suites/847issue records plus one separate unit issue, including842 NemotronSuper assertions and two Prism error-contract regressions; four GLM issue records match foundation limitations. Serial CI uses event synchronization with matching reported runner image/OS/toolchain. A common cause with the cluster fast-path stall is not established. Final reviews, exact pins, SDK isolation and full-model checks remain gates.

Progress 2026-10-26: main merged, stack re-pinned, validated on two M5 Ultra 256 GB

Head caa9ed53: fast-forward of d11238f1 plus #35/main merges, the head-dim-256 NAX attention JIT fix (ce21a37d) and MLX_QMV_BATCH_INVARIANT (caa9ed53).

• Main merged with signed non-force merges, parent to child; each new head contains the previous head, so the push is a fast-forward. Conflict resolutions, checks and evidence: MLX-MERGE-2026-10-26.md (core), MLX-C-MERGE-2026-10-26.md, MLX-SWIFT-MERGE-2026-10-26.md, MLX-SWIFT-LM-MERGE-2026-10-26.md (kept with the overnight records).
• Core (JACCL: zeroed send-frame tails, a progress guard, the advertised GID and the upstream JACCL fixes #33/Cluster collectives, quantized numerical fixes and optional Metal allocation cap #35/Fix Metal fence ordering and sealed collective portability #41): JACCL keeps the send-frame zeroing, progress guard and advertised-GID fix on every cable of main's multi-cable mesh; the ring's moved wire dispatcher now reports errors from extra cables (187b6c67, test fails without the fix). Cluster collectives, quantized numerical fixes and optional Metal allocation cap #35 carries Fix Metal fence ordering and sealed collective portability #41's barrier line so it does not undo main's [BUG] Metal fence deadlock ml-explore/mlx#4689. Main's value-based fence waits are taken; the cluster keeps its single spin ordering. Upstream JIT bug fixed (ce21a37d): head-dim-256 NAX attention failed to build its run-time template on M5. Fix Metal fence ordering and sealed collective portability #41 adds MLX_QMV_BATCH_INVARIANT (caa9ed53, off by default): quantized products of 2 to the vector limit - 1 rows stay on qmv_wide, so a row's bits do not depend on how many rows share its launch (main's [Metal] Add qmv_mma/qmv_nax kernels for 6-16 rows with large N/K ml-explore/mlx#4712/[Metal] Add staged affine_qmm_t_nax kernels with split-k ml-explore/mlx#4682 routes otherwise change them). Cluster processes turn it on; single-Mac keeps main's kernels.
• mlx-c (Expose coherent allocator accounting and correct quantized bias sums #15): main merged; builds against the merged core with its test.
• mlx-swift (Docker: xcrun: not found ml-explore/mlx#54/Simple test to compare speed against PyTorch on MPS ml-explore/mlx#56/random.uniform with dtype float16 returns all 0s ml-explore/mlx#61): main merged; nested pins restitched (Source/Cmlx/mlx = the matching core layer, Source/Cmlx/mlx-c = mlx-c Expose coherent allocator accounting and correct quantized bias sums #15, Docker: xcrun: not found ml-explore/mlx#54 on main's C); generated sources regenerated with tools/update-mlx.sh and compared; random.uniform with dtype float16 returns all 0s ml-explore/mlx#61 pins core caa9ed53 (only include-framework/mlx-utils.h changes). Fork Tests (all three shards with an AOT metallib): XCTest 849/853 + 2 permitted skips, Swift Testing 749/749.
• SDK (Add Makefile for python related development ml-explore/mlx#308/simple numpy helper for tests ml-explore/mlx#352): main merged (swift-format 603 layout, fork.yaml, Metal-library and tensor-unit test gating, admission-test repair); Add Makefile for python related development ml-explore/mlx#308 pins mlx-swift Simple test to compare speed against PyTorch on MPS ml-explore/mlx#56, simple numpy helper for tests ml-explore/mlx#352 pins mlx-swift random.uniform with dtype float16 returns all 0s ml-explore/mlx#61. simple numpy helper for tests ml-explore/mlx#352 also carries the cluster beta's SDK changes (GLM DSA/NextN history readers for the GLM TP SSD tier, unserved content parts refused, checkpoint row codec for SSD records, Qwen TP on-demand page growth, Bonsai FP32 residual stream fix, Flint precision fix for master How to make an mlx array containing every number from 0-100 and skipping every Nth number ml-explore/mlx#1554) and the two single-Mac GLM speedups (64-head KDA prefill cut, early MTP drafts; bitwise on the real model).
• Numerics: main's precise sigmoid (Use precise::exp in Sigmoid so compiled and eager sigmoid agree ml-explore/mlx#4461), split-K and SDPA changes move GLM-5.3-Flash's bits versus the previous cluster build; on four natural documents the pair's perplexity is lower on every one (−0.16 … −1.16 %). c1 = cN byte equality holds on the pair at c1-c16 with the switch on.
• Pair validation: candidate c48 (d-inference Fix OOB access in qmv ml-explore/mlx#1577) on two Mac Studio M5 Ultra 256 GB; see Fix OOB access in qmv ml-explore/mlx#1577 for the gate table.

Nothing merged or released; the owners sign and release.

Update 2026-10-26 17:15 PDT: main/master 0.10.3 merged through the stack

Head ea318ee0: the refreshed #35 merged; this branch fixed the same JIT bug as main (ce21a37d), and get_steel_attention_nax_kernel now reads exactly as main's. Every PR in the cluster stack (mlx #33 → #35 → #41, mlx-c #15, mlx-swift ml-explore#54 → ml-explore#56 → ml-explore#61, SDK ml-explore#308 → ml-explore#352, d-inference ml-explore#1407 → ml-explore#1426 / ml-explore#1577, platform ml-explore#54) is again mergeable in that order, as drafts.

Review fixes and CI follow-up (2026-10-27)

Head e637499f: review fix b2d92e5b (eval_impl updates the fast fence before the event fence, MLX_EAGER_FENCE_COMMIT commits after both; metal_fence_chain covers two GPU streams that feed the CPU stream) and #35's fixes merged in.

CI follow-up: build-cpp builds without MLX_METAL_JIT, so kernels_jit_compiled() is false and quantized_tile_tests compares the 32 x 32 kernel with itself. The 64-row, 64-column and BK=64 tiles are not exercised in CI. Covering them needs a JIT C++ lane.

[davidtai]

Review fixes in b2d92e5:

• eval_impl updates the fast fence before the event fence. On CUDA the cross-device fast update moves the array to unified memory (an async copy and a pointer swap). With the event recorded first, another GPU stream could read the new buffer before the copy finished. MLX_EAGER_FENCE_COMMIT now commits after both updates.
• metal_fence_chain: new case where both GPU streams feed the CPU stream. This covers the event wait on CPU work and the event fence on a CPU stream.

Not changed (CI design question): build-cpp builds without MLX_METAL_JIT, so kernels_jit_compiled() is false and quantized_tile_tests compares the 32 x 32 kernel with itself. The 64-row, 64-column and BK=64 tiles are not exercised in CI. Covering them needs a JIT C++ lane.