[danbev]

This commit adds conversion support for NVIDIA Nemotron 3.5 Lightning.
It does not include NVFP4 models, including dflash and dspark.

The motivation for not including these conversion is that those
checkpoint was designed as W4A16, but the converted GGUF only says that
Its weights are NVFP4. On Blackwell, llama.cpp sees NVFP4 weights and
will automatically selects its native FP4 path, dynamically converting
activations to FP4. That makes the executed operation W4A4 which can
affect the accuracy of the draft model and therefor in turn effect the
acceptance rate of the target model. The references PR below has more
details on a solution to address this issue. Once it has been merged we
should be able to convert and publish these models, but for now we are
just commenting them out.

Refs: ggml-org/llama.cpp#26675

---

I've build this and uploaded to my HF account:
https://huggingface.co/danbev/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF/tree/main

I'm creating this as a draft as I want to run some manual test to try out the mtp, dflash, and dspark speculative drafting.

[danbev]

Just a note, I think it might be nice to have --keep flag for the conversion script to not remove the downloaded models , or the converted models even if the conversion script succeeds.

[ggerganov]

Yes, a --keep flag would be useful.

[danbev]

I've tried out MT and DFlash which work, but I was not able to get DSpark to work. Trying to load the draft model results in the following error:

0.02.977.124 E llama_model_load: error loading model: check_tensor_dims: tensor 'conf_proj.weight' not found
0.02.977.147 E llama_model_load_from_file_impl: failed to load model               

I suspect that support for it might not be in llama.cpp yet and the current implementation assumes that there is a confidence projection tensor but this model does not seem to include one and perhaps does not use confidence verification at all. I'll look into this.

[ggerganov]

Ok, I am also not sure about the status of DSpark for this model.