Repository navigation
[quantization] per-layer quantization plan: mixed NVFP4 + FP8 checkpoints - #50245
ArthurZucker wants to merge 11 commits into
Conversation
…on: pin its output
…ad op, uncompressed checkpoints no longer force it
… the checkpoint format; drop no-op after-load hook
…hts load decompressed, compressed-tensors only sets up activation / kv cache quantization
…ix and name boundaries to #50245
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. Pages changed in this PR: |
| quantizer.preprocess_model(model, **kwargs) | ||
|
|
||
| def is_serializable(self): | ||
| return all(quantizer.is_serializable() for quantizer in self.quantizers.values()) |
There was a problem hiding this comment.
Could the serialization interface be made consistent with NVFP4? NVFP4HfQuantizer.is_serializable is a property returning a boolean, so this call raises TypeError: bool object is not callable for the advertised mixed plan. A save attempt should produce the intended support decision instead.
| self.quantization_config = { | ||
| pattern: quantizer.quantization_config for pattern, quantizer in quantizers.items() | ||
| } | ||
| self.pre_quantized = all(quantizer.pre_quantized for quantizer in quantizers.values()) |
There was a problem hiding this comment.
Could fully dequantized plans be cleaned up during postprocessing? The inherited postprocessor checks quantization_config.dequantize, but this config is a dictionary, so an all-dequantized model retains its pre-quantized recipe and quantizer. Please add a dense save/reload round trip, including removal of dequantized plan entries.
| max_memory = quantizer.adjust_max_memory(max_memory) | ||
| return max_memory | ||
|
|
||
| def preprocess_model(self, model, **kwargs): |
There was a problem hiding this comment.
Could the plan also delegate each child's post-load hook? It delegates preprocessing but inherits the base postprocessor, so FP8's _disable_deepgemm_on_multi_device() never runs for a mixed plan. A multi-device FP8 plan can therefore keep a backend that the standalone quantizer explicitly disables as unsafe.
bc6733c to
7cf9e8a
Compare
…tized_module` (#50343) * [quantization] swap modules in one pass through a per-quantizer quantized_module * [quantization] build_fp8_linear / build_fp8_experts
…-load' into HEAD # Conflicts: # src/transformers/quantizers/quantizer_compressed_tensors.py # tests/quantization/compressed_tensors_integration/test_compressed_models.py
…ch transform's ops; fp8 declares fused experts
|
[For maintainers] Suggested jobs to run (before merge) run-slow: ministral3, mistral4, aqlm_integration, bitnet_integration, eetq_integration, finegrained_fp8, higgs, metal, nvfp4, sinq |
A quantization plan,
quantization_config={glob: config}with the dtype plan's matching, loads each layer through its own quantizer; pre-quantized NVFP4 loads through adeserializeop. Tested onhf-internal-testing/tiny-random-Mistral4-fp8-nvfp4(fp8 attention + experts, nvfp4 shared experts).Supersedes #49818 and #49658 (mixing quantization methods per module); overlaps #49779 and #49851 for pre-quantized NVFP4.
Related issues: #38253, #48371, #44292, #46772. Needs huggingface/kernels-community#1291 for fp8 experts.