Skip to content

[Metal] NVFP4 tensor-scale (global_scale) not implemented — throws runtime error on Metal backend #3911

Closed

Description

@dahai80

Summary

PR #3022 added per-tensor scale (global_scale) support for NVFP4 quantization on CUDA and CPU backends. However, the Metal backend explicitly rejects it:

https://github.com/ml-explore/mlx/blob/main/mlx/backend/metal/quantized.cpp#L1725-L1730

if (mode_ == QuantizationMode::Nvfp4 &&
    static_cast<int>(inputs.size()) > base_size) {
  throw std::runtime_error(
      "[QQMatmul] Global scale (tensor-scale nvfp4) is not supported "
      "on the Metal backend.");
}

Impact

Without tensor-scale support, NVFP4 on Metal has ~137x less dynamic range than NVIDIA Blackwell (unsigned UE4M3 vs signed E4M3 scales), which was the original concern in #2962. The fix (PR #3022) landed for CUDA/CPU but Metal users still hit the error.

This blocks NVFP4 quantization for Apple Silicon users running MoE models (DeepSeek-V3/V4, GLM-5.1, etc.) where tensor-scale is critical for expert routing accuracy.

Request

Please add Metal kernel support for consuming the global_scale inputs in the qqmm / fp_quantized Metal kernel paths, matching the CUDA/CPU behavior from PR #3022.

Environment

  • MLX commit: 973e27f
  • Platform: Apple Silicon (M-series)
  • Backend: Metal

Activity

  1. zcbenz commented on Aug 5, 2026

    @zcbenz
    Member

    #3757 has added some basic support for global scale in QQMatmul, but at the moment not all fast kernels get the support and we are still working on improving things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions