[inureyes]

Summary

The vendor FP8 releases of the Qwen3.5 family store every converted projection as raw E4M3 bytes in <name>.weight plus a bf16 inverse scale per 128x128 block in <name>.weight_scale_inv. MLX's safetensors reader maps F8_E4M3 to uint8, so those bytes already arrived intact and nothing errored: on main this checkpoint loads, generates, and is wrong by a per-block factor on every projection. That is why this is a correctness fix and not a missing-capability one.

models::fp8_block reconstructs each pair at load and requantizes it to MLX-native mxfp8 (group size 32, 8 bits), which the existing quantized-matmul path executes with no per-token dequantization.

What changed

• src/models/fp8_block.rs (new): qwen_fp8_block_quantization reads quantization_config from the top level or text_config and accepts quant_method: "fp8" with fmt absent or "e4m3" and weight_block_size: [128, 128]. Any other geometry, a non-E4M3 fmt, and a missing weight_block_size (per-tensor FP8, a different sidecar layout) are named errors rather than an assumed 128, because the wrong pairing mis-scales a tensor without failing.
• requantize_block_fp8_weights decodes the E4M3 bytes through a 256-entry device LUT (take), pads to whole blocks, applies the block scale by broadcast over [rb, 128, cb, 128], slices back, and quantizes. Rank other than 2, a sidecar whose shape does not tile its weight, an orphan sidecar, a width that is not a multiple of 32, and a weight already decoded to a float dtype are each refused by name.
• Peak memory is bounded by construction: one tensor at a time, with each packed pair evaluated before the next tensor starts, so the f32 reconstruction of the largest projection is the only transient.
• Wired into all three Qwen3.5 entry points (text loader, VLM loader before the weight map is split, adapter-side special loader). Each also merges the effective quantization block into the text config, which these releases do not carry.
• docs/supported-models.md: the Qwen3.5 entry no longer calls the FP8 builds out of scope; it records the layout, the served footprint, which tensors stay dense, and which declarations are refused.
• TECHNICAL_REPORTS/1742-qwen35-fp8-block-requantize-20260910.{en,ko}.md.

Validation

Every command below ran on Linux / aarch64 / GB10 (sm_121) with --features cuda and MLX_ENABLE_TF32=0.

• cargo test --profile test-fast --features cuda --lib models::fp8_block: 10/10. The central test compares the device path bitwise, on both the packed uint32 plane and the uint8 scale plane, against quantize_weights_with_mode(decode(bytes) * expanded_scales, 32, 8, "mxfp8") built independently on the host, over a 130x160 tensor that is block-aligned on neither axis. A second case covers the block-aligned no-padding branch, and a third checks all 256 E4M3 encodings through the device LUT.
• cargo test --profile test-fast --features cuda --lib models::qwen3_5 (28), --lib models::sanitize (71), --lib loading::special (6): all pass.
• Real checkpoint Qwen/Qwen3.8-27B-FP8 (29 GB, 1606 tensors, 407 sidecars): mlxcel generate -n 220 --show-reasoning produced a coherent reasoning trace and a well-formed limerick, stopping naturally at 207 tokens. Load took 32.1 s at 23.36 GB resident and 24.37 GB peak (MLX allocator), a ratio of 1.04, against the 1.5x bound the issue asks for. Process RSS peaked at 6.9 GB, because MLX's CUDA allocations on this host do not appear in RSS, so the allocator figures are the meaningful ones.
• Checkpoint structure verified directly rather than taken from the issue: all 407 F8_E4M3 tensors have a bf16 sidecar in the same shard, every sidecar shape is exactly ceil(R/128) x ceil(C/128), every width is a multiple of 32, no F8_E4M3 tensor lacks a sidecar, and every tensor is block-aligned (so the real checkpoint never exercises the padding branch, which is why the unit test uses 130x160).
• Scale direction checked independently in numpy on three real tensors: decode * scale_inv gives max 0.32 to 0.44 and std 0.011 to 0.017, a normal projection distribution, while decode / scale_inv gives about 4e6. The E4M3 decode alone tops out at exactly 448 in every tensor, confirming the vendor scaled each block to fill the E4M3 range.
• cargo fmt --all -- --check and cargo clippy --workspace --all-targets --features cuda -- -D warnings pass.

Not verified

• The issue's "logits agree with the bf16 original within the mxfp8 tolerance" is unverified and left unticked: the bf16 reference Qwen/Qwen3.8-27B (55.6 GB) was deliberately not fetched on this host, and no weaker oracle is offered in its place.
• The issue's cargo test --workspace --profile test-fast --features metal,accelerate is not runnable on this host (Linux / CUDA, no Metal and no Accelerate) and was not run. The CUDA equivalents above were run instead.
• A qwen3_5_moe FP8 release was not available to test. A fused 3-D experts.gate_up_proj carrying block scales would be refused by name (rank 2 is required) rather than mis-converted.
• The MTP drafter keeps its own loader in mlxcel-core and does not take this pre-pass; MTP speculative decoding is already unsupported for this family.

Closes #1361