[Feature] Support for Q1_0_g128 quantization format (PrismML Bonsai models)

[guilamu]

Summary

Attempting to load any model from the prism-ml/Bonsai-*-gguf family
(1-bit LLMs by PrismML) fails with the following error:

Failed to load model: Failed to load model even at minimum context (2048).
This may indicate insufficient memory, a corrupted model file, or an
unsupported model format. (Failed to load model)

Device

• Samsung Galaxy S23 (Snapdragon 8 Gen 2, 8GB RAM)
• Android 14
• Off Grid (latest Play Store version)

Steps to Reproduce

1. Models → Search "Bonsai"
2. Tap Bonsai-4B-gguf or Bonsai-8B-gguf by prism-ml
3. Download and attempt to load
4. Error appears immediately on load

Root Cause

The Bonsai models use a Q1_0_g128 quantization format — a native 1-bit format (1 bit per weight + 1 FP16 scale per 128-weight group) introduced by PrismML in their own llama.cpp fork: https://github.com/PrismML-Eng/llama.cpp

This format is not yet merged into upstream llama.cpp, which is what llama.rn currently bundles.

Why This Matters

Bonsai 4B is only ~1 GB and benchmarks comparably to standard 8B models. It would be an ideal fit for mobile — but requires the Q1_0_g128 kernels to load.

Request

Could you update llama.rn / the bundled llama.cpp to include PrismML's Q1_0_g128 support? Their fork is MIT-licensed and the relevant changes appear to be isolated to the new quant type kernels.

Reference:

• Model: https://huggingface.co/prism-ml/Bonsai-4B-gguf
• PrismML's llama.cpp fork: https://github.com/PrismML-Eng/llama.cpp
• Announcement: https://prismml.com/news/bonsai-8b

[fu8765]

Support is now added to llama.cpp

ggml-org/llama.cpp#21273

[guilamu]

@fu8765 : I still have the an error message loading Bonsai-8B.gguf on my S23 using 0.0.83. The error message is now:

Failed to load model: Failed to load model even at minimum context (2048).
This may indicate insufficient memory, a corrupted model file, or an
unsupported model format.
> Error chain: min-ctx: Failed to load model

[khosravipasha]

Currently only generic CPU backend is merged. There is few more PRs coming for x86, Metal, CUDA.
On phone were you running CPU backend or GPU (something like Vulkan or OpenCL)? There is Vulkan backend too in the works but not fully ready.

[guilamu]

@khosravipasha tried both CPU and GPU with the same error message.

[khosravipasha]

Yes, the app needs to be rebuilt most likely, my comment was for Q1_0 kernels in llama.cpp.
Best is to wait until x86 kernels and/or Vulkan are merged, not sure which CPU backend is used for phones.