Add the package localai_decisions_qwen35: the decision models based on Qwen3.5, with llama.cpp in Rust #721

Goal

Add a Rust package that runs the open decision models based on Qwen3.5 on the computer, with llama.cpp.

• Package name: localai_decisions_qwen35
• Folder: packages/experiments/localai_decisions_qwen35
• It gives a Rust library that Heidrun can link, and a command line tool.
• It runs on the processor by default. A build option adds the GPU (Metal, CUDA, or Vulkan).
• It supports every decision model based on Qwen3.5 that llama.cpp supports, not only one model. The model is a parameter.

A decision model reads a text (the "state") and typed questions with their options. It returns a probability for each option, in one pass, and writes no text. The API is the System One API of Jev (TypeSafe).

Why llama.cpp

• Candle cannot run Qwen3.5 yet: Qwen3.5 Support huggingface/candle#3393
• llama.cpp runs these models, with the API /v1/systemone of its server:
  • llama, server: add /v1/systemone API (models: laya, julia-1, lev, openjev, kev) ggml-org/llama.cpp#29818 (merged on 2 October 2026: the API, and the models Kev, lev, OpenJev, Laya, and Julia-1)
  • model: add support for clef decision model (text-only) ggml-org/llama.cpp#29831 (merged on 3 October 2026: the model Clef, text only)
• The Rust binding llama-cpp-2 (version 0.1.160, 15 October 2026) builds llama.cpp inside the program. It runs on the processor by default, and has the features metal, cuda, and vulkan for the GPU.

The main difficulty

The model (Qwen3.5, and the Clef layers) is in the library of llama.cpp (src/), so the Rust binding can load it. The decision part is only in the server of llama.cpp: tools/server/server-decision.cpp (about 800 lines). It builds the prompt from the state and the questions, and reads the probability of each option. The library does not export this part.

Choices:

1. Write this part again in Rust, from server-decision.cpp.
2. Compile server-decision.cpp with the library, and call it from Rust.
3. Start the server of llama.cpp as a separate program, and send HTTP requests to /v1/systemone. This is the simplest, but it is not a linked library.

First test, before the package

One assumption can make this work impossible: the Rust binding llama-cpp-2 cannot give the values that the decision part needs (for example, the binding uses an older llama.cpp than the two pull requests above, or it hides the functions that the decision part calls). The first step proves or kills this assumption:

1. Run llama serve -hf ggml-org/Kev-0.8B-GGUF on the processor, send the example request of the README of the server, and save the probabilities.
2. Run the same request through the Rust binding, on the processor.
3. Compare the two results, and measure the time on the processor.

Build the package only when the two results match.

Decision models based on Qwen3.5 with a GGUF file

All these files are on Hugging Face. The size is the size of the file Q8_0.

Proposed default model: ggml-org/Kev-0.8B-GGUF, file Q8_0. It is the smallest, and its license is Apache 2.0.

Laya and Julia-1 also work with /v1/systemone, but their architecture is ModernBERT, not Qwen3.5, so they are not in this package.

Done when

• The library and the command line tool run every model of the table on the processor, with a model given as a parameter.
• A build option runs the same models on the GPU.
• The package has a CONTEXT.md file and a README.md file.
• The tests compare the results of the package with the results of the server of llama.cpp on the same requests, with Kev-0.8B.
• The section "Tests" of AGENTS.md lists the command pnpm --filter localai_decisions_qwen35 test.

Not known yet

• The speed of each model on the processor.
• If the Rust binding llama-cpp-2 already has the two pull requests of llama.cpp above.
• The image input of the models that have a file mmproj. This issue covers text only.

Related

• Add the package localai_decisions: a local decision model with GLiNER2.5-Decide in Rust #717