Eval bug: Ternary-Bonsai-27B fails to load when using its Dspark draft model

[binkleym]

Name and Version

$ llama-cli --version
version: 10197 (958d9c0)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

I have tested this bug on two servers and gotten similar results.

• Server 1: AMD Strix Halo AI MAX 395+ with 128 GB DDR5
• Server 2: AMD 5700G with 64 GB of DDR4

Models

Ternary-Bonsai-27B with the DSpark draft model. In particular, I'm using the Q2_g64 quant of the main model and the Q4_1 quant of the draft model at the URL below:

https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf/tree/main

Problem description & steps to reproduce

I attempt to load the model using llama-server with the following config:

[Ternary-Bonsai-27B]
model = Ternary-Bonsai-27B-Q2_g64.gguf
mmproj = Ternary-Bonsai-27B-mmproj-Q8_0.gguf
model-draft = Ternary-Bonsai-27B-dspark-Q4_1.gguf
ctx-size = 32768
temp = 0.7
top-p = 0.95
top-k = 20
cache-type-v = q8_0
cache-type-k = q8_0
spec-type = draft-dspark
spec-draft-n-max = 2
flash-attn = 1
batch-size = 512
ubatch-size = 512
cache-prompt = 1
parallel = 1
models-max = 1
threads = 6
threads-batch = 6
n-gpu-layers = all
load-mode = mmap

Loading the model fails with the following message:

[39829] 0.00.032.539 I srv load_model: loading model 'Ternary-Bonsai-27B-Q2_g64.gguf'
[39829] 0.00.101.674 E gguf_init_from_reader: tensor 'dspark.fc.weight' has offset 337718592, expected 357584192
[39829] 0.00.101.682 E gguf_init_from_reader: failed to read tensor data
[39829] 0.00.101.777 E llama_model_load: error loading model: llama_model_loader: failed to load model from Ternary-Bonsai-27B-dspark-Q4_1.gguf
[39829] 0.00.101.784 E llama_model_load_from_file_impl: failed to load model
[39829] 0.00.101.798 W srv load_model: [spec] failed to measure draft model memory: failed to load model

First Bad Commit

No response

Relevant log output

Logs

[GTaf]

I was able to reproduce the problem on my setup with the latest master branch.

[eeemapo]

I can reproduce the exact error using the Q4_1 quant of the Dspark model and these settings:

"Bonsai-27B":
cmd: >
llama-server
--port ${PORT}
--model /models/Bonsai-27B/Ternary-Bonsai-27B-Q2_g64.gguf
--ctx-size 32768
--temp 0.6
--top-p 0.95
--top-k 20
--reasoning on
--model-draft /models/Bonsai-27B/Ternary-Bonsai-27B-dspark-Q4_1.gguf
--spec-type draft-dspark
--spec-draft-n-max 2

0.05.617.656 I common_speculative_init_result: loading draft model '/models/Bonsai-27B/Ternary-Bonsai-27B-dspark-Q4_1.gguf'
0.05.617.908 E gguf_init_from_reader: tensor 'dspark.fc.weight' has offset 337718592, expected 357584192
0.05.617.912 E gguf_init_from_reader: failed to read tensor data

Oddly, swapping out the Q4_1 model for the bf16 version leads to a different error:

0.03.329.487 I common_speculative_init_result: loading draft model '/models/Bonsai-27B/Ternary-Bonsai-27B-dspark-bf16.gguf'
0.03.329.972 E llama_model_load: error loading model: unknown model architecture: 'dspark'
0.03.329.975 E llama_model_load_from_file_impl: failed to load model

From this observation, I am inclined to believe that there is an issue with the way the model was packed vs the way llama.cpp expects it to be packed.

[GTaf]

After looking at the code, the issue seems to come from the a tensor of the new type introduced for this model (Q2_0).
The failing layer on load is dspark.fc.weight, which is not at the right offset. This leads to inspecting the former layer which might be faulty.

The former layer token_embd.weight has type Q2_0, which was introduced for this model in a recent PR.

The offset announced by the gguf format, and the expected offset computed by llama.cpp from the tensor shape and type don't match, hence the error.
According to https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf/blob/main/Ternary-Bonsai-27B-dspark-Q4_1.gguf, Q2_0 is supposed to store 64 elements of size 2 bits in a total of 18 bytes and the layer is of shape [5 120, 248 320]. So one element of the first dim is 1440 bits and the total tensor size is 357,580,800. It is impossible to fit with an offset of 337718592 as stated in the previous comment.

However, if we guess a type_size of 17 and not 18 for Q2, numbers add up. It would mean a delta for scaling quantized weights of size 8 bits and not 16 as in other cases (Q1_0 for example).
Edit : the mismatch comes from the fact that Prism own Q2 stores 128 elements with still 2 bytes of delta, hence the relative weight of delta divided by two.

I don't know where the truth lies in this case, if the gguf file is wrong or if the code misses this special uint_8 delta type.
Maybe @khosravipasha can help ?

[GTaf]

Found the right thread after digging the code, the problem is known and a new gguf is incoming : #24448 (comment)

[khosravipasha]

Yeah the dspark version only work with our fork at the moment, it was highly experimental since the the paper came out ~2 weeks before we released our models+drafters). Now seems llama.cpp has a stable implementation of dspark but its slightly different than ours.

In future releases will make sure it work with main llama.cpp, and trying to migrate/refactor our fork to get recent changes from llama.cpp. Might take some time before we finalize this.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.