convert : handle pre-quantized models

[compilade]

Should fix #14762, and also address #3353.

This roughly implements the idea in #14762 (comment) to allow converting from pre-quantized models, by splitting ModelBase.get_tensors into an intermediate ModelBase.index_tensors which returns a dict[str, Callable[[], Tensor]], which can be modified before get_tensors is called. get_tensors still keeps the same signature (it still returns an Iterator[tuple[str, Tensor]]).

For now, support for these pre-quantizations has been implemented:

• bitnet
  • https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens
    • works
  • https://huggingface.co/codys12/Qwen3-8B-BitNet
    • not really supported (yet) because of the extra RMS norms; will be possible in a separate PR
  • https://huggingface.co/microsoft/bitnet-b1.58-2B-4T
    • should be possible to support once the architecture is also added
• fp8
  • https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-1.2B-FP8
    • has relatively high perplexity, not sure why
  • https://huggingface.co/Qwen/Qwen3-4B-FP8
    • has good relatively low perplexity, so I assume this works
• gptq (only 2, 4 and 8 bits, not 3)
  • https://huggingface.co/AlignQuant/Meta-Llama-3-8B-Instruct-GPTQ-2bit
    • high perplexity (in the hundreads), but I guess that's normal for 2-bit GPTQ? (maybe not...)
  • https://huggingface.co/eewer/Qwen3-0.6B-4bit-GPTQ
    • works
  • https://huggingface.co/reinattwijaya/Qwen3-0.6B-final-noreason-gptq-8bit-c4
    • works

The 3-bit variant of GPTQ is more complicated, and so was omitted for now.

Notes

• This removes ModelBase.tensor_names in favor of self.model_tensors which also allows getting the tensor data (because it's a dict[str, Callable[[], Tensor]].
• CodeShell had a workaround in llama : add support for GPT2, Bloom and CodeShell tied word embeddings #12456 (by @CISC) which removed the lm_head.weight tensor from self.tensor_names, but I don't see why it's necessary. I've removed it because self.tensor_names was also removed.
  • I've converted https://huggingface.co/WisdomShell/CodeShell-7B-Chat without problem
  • I've also ran a --dry-run of a --remote conversion of https://huggingface.co/WisdomShell/CodeShell-7B, which does run the tensor index completeness check. No problem either.
• Lambda functions in Python don't quite capture their environment, so I've used the default-parameter trick from https://stackoverflow.com/a/2295372 a lot.

TODO

• Test if this causes memory usage regressions
  • Lazy or not, safetensors or not
  • So far it seems good.
• Test remote conversion (with --remote)

---

Make sure to read the contributing guidelines before submitting a PR

[ggerganov]

Perfect!

In case you feel like it, add support for MXFP4 as well. I will be upstreaming a ggml implementation soon and it would be nice to have HF conversion support. You can use some of the smaller models here https://huggingface.co/models?sort=created&search=mxfp4 (anyone without hadamard matrices should work).

[CISC]

In case you were wondering, the workaround was about this line:
https://huggingface.co/WisdomShell/CodeShell-7B-Chat/blob/main/pytorch_model.bin.index.json#L555

Conversion would fail because that tensor doesn't exist.

Edit: I might have used the safetensors version, could be it actually works with pytorch bins.

[CISC]

MLX support could be useful, they have .scales and .biases tensors:
https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.dequantize.html

Edit: group_size and bits are stored in config.json (not sure why there are two entries):

    "quantization": {
        "group_size": 32,
        "bits": 5
    },
    "quantization_config": {
        "group_size": 32,
        "bits": 5
    },

[MaoJianwei]

Is it ready to be merged? We need this :)

#15173

[compilade]

Is it ready to be merged? We need this :)

It kind of is, but I wanted to make it more general, and also repack instead of going through F32 and requantize. I've described this in #15111 (comment), but the approach here isn't compatible with how MXFP4 is handled for gpt-oss. I mean, whenever the mxfp4 quant method is handled, it will break the repacking which the gpt-oss conversion relies on.

I guess I can change this in a follow-up PR, since mxfp4 isn't handled here yet.

I have started working on a deferred repacking system to transparently handle transformations of pre-quantized tensors. (it's a bit more involved than I thought, so I guess it makes sense to leave it for a follow-up pull-request)
It's going to be a bit like LoraTorchTensor from convert_lora_to_gguf.py, but for pre-quantized tensors, and with a repacking API for types gguf-py/gguf/quants.py can currently quantize. It should allow avoiding to go through F32 when possible, which means it will be possible to handle mxfp4 and also ternary models more satisfyingly (pre-quantized tensors will also keep a compatible type when using --outtype auto in what I'm planning).

So this can be considered ready to be merged (assuming this doesn't break remote conversion), and a follow-up pull-request will handle repacking (instead of requantizing).

[MaoJianwei]

So this can be considered ready to be merged (assuming this doesn't break remote conversion), and a follow-up pull-request will handle repacking (instead of requantizing).

It sounds good :)

[BugBusterMax]

How can I convert llama3.2_1b after GPTQ to gguf?

[CISC]

@compilade Preferably #14737 should be merged first, then you can rebase this.

[CISC]

@compilade Merge now?

[MaoJianwei]

when to merge?

[Green-Sky]

I think this is ready, but it has been stuck in unmerged limbo for a while now. @ggerganov please merge/rereview.

[CISC]

It is approved, but should be merged by @compilade esp. as there are some interdependent PRs.

[ggerganov]

@compilade Checking in to see if you plan to bring this to completion. If you don't have time - no worries. We'll try to take over and finish it. Just give us some indication about your plans.

[compilade]

@compilade Checking in to see if you plan to bring this to completion. If you don't have time - no worries. We'll try to take over and finish it. Just give us some indication about your plans.

I do plan to finish this. Sorry for the delay (@ggerganov, @CISC, @MaoJianwei, @Green-Sky).

My plans are still somewhat as before in #14810 (comment), but this doesn't necessarily block merging this. After merging this, I plan to finalize #15667 and #15727 next.

I will merge this today at 20:30UTC (in around 1h30 after this message is sent), so that I can test for regressions in the meantime (in case some were introduced in recent merges), although this PR apparently has been already used a bunch for recent-ish FP8 models.

The most noticeable difference between this PR and master (apart from dequantizing pre-quantized models) is when the source tensor files are enumerated. Before this PR, they were interleaved with tensor transformations, but now they are all indexed up-front. This shouldn't affect conversion speed, I think.

Again, sorry for not merging this earlier; I have a tendency to avoid merging right before going to sleep, and in recent months my availability times didn't really align with that approach (but today it's earlier).

I'll try to be clearer in the future when I don't merge for similar reasons, so that merge limbo doesn't last as long.

[julmb]

Is there an issue or a PR that I can track to see when mxfp4 will be available with the new conversion method?

[MaoJianwei]

Congratulations!