enhancement(backends): run Granite Switch adapter functions on llama-server (llama.cpp) #1735

Today a Granite Switch checkpoint has two supported runtimes: vLLM through OpenAIBackend, and local HuggingFace through LocalHFBackend. On a laptop or a GPU-less box there is no supported route at all. OllamaModelBackend has no embedded-adapter path (the binding table in docs/docs/advanced/intrinsics.md:406-410 has a dash in that column), and pointing OpenAIBackend at a llama.cpp server almost works, except the document-based adapter functions quietly return wrong answers.

After this change, scripts/start_llamacpp_switch.sh serves a Switch GGUF locally and OpenAIBackend(base_url=...) runs adapter functions against it, same API as the vLLM path.

What already works

llama.cpp merged the granite-switch architecture in August 2026 (ggml-org/llama.cpp#25107, in releases since v0.1.0), and convert_hf_to_gguf.py handles GraniteSwitchForCausalLM. I converted ibm-granite/granite-switch-4.1-3b-preview to an f16 GGUF and served it with llama-server --jinja (v0.6.0, Homebrew). Through OpenAIBackend(load_embedded_adapters=True):

• guardian_check on "Group X people are all lazy." scored 0.97 for social_bias (the granite-switch README gets 0.964 on vLLM); a benign prompt scored 1.6e-07.
• core.check_certainty scored 0.94 on a correct answer and 0.59 on a wrong one.

So activation is already portable: llama-server accepts chat_template_kwargs.adapter_name, renders the Switch chat template server side, and the control token lands in the prompt exactly as under vLLM.

What breaks

rag.check_answerability answers "unanswerable" for every input, including when the documents contain the answer. citations, hallucination_detection, query_clarification and context-attribution have the same failure.

The intrinsic path in OpenAIBackend puts documents in a top-level documents field of the request body (mellea/backends/openai.py:1130). vLLM declares that field on its chat completion request and folds it into the template context. llama-server ignores unknown top-level fields, so the Switch template's documents variable stays unset, the documents system message never renders, and the model correctly reports that it cannot answer.

The same request with documents inside chat_template_kwargs discriminates correctly under llama-server: "answerable" with documents that contain the answer, "unanswerable" with documents that don't.

Proposed change

1. Send documents in both places. In _generate_from_intrinsic, after the extra_body merge and before apply_activation, copy documents into chat_template_kwargs unless the caller set their own value there. The top-level field stays, so vLLM behaviour is unchanged; llama-server reads the copy. vLLM folds both shapes into the same template context (build_chat_params in vllm/entrypoints/openai/chat_completion/protocol.py), so the duplicated placement is harmless there.
2. scripts/start_llamacpp_switch.sh, modelled on scripts/start_llamacpp.sh: fetch the checkpoint, convert to GGUF, serve with --jinja, wait for readiness, print the connect snippet. One workaround is needed: current transformers refuses to load the checkpoint's tokenizer_config.json because extra_special_tokens is a list. The twelve control tokens are already in tokenizer.json, so the script strips that key in a shadow copy before converting. This quirk is worth an upstream note on generative-computing/granite-switch.
3. Example and docs. docs/examples/granite-switch/answerability_llamacpp.py with the existing server-detection skip pattern, a llama-server section in the granite-switch examples README, and a note in docs/docs/advanced/intrinsics.md that the OpenAI backend also serves Switch checkpoints from llama-server.

Testing

• Unit test in test/backends/test_openai_intrinsics_unit.py: the request carries documents both top-level and inside chat_template_kwargs, and a caller-supplied chat_template_kwargs.documents is not overwritten.
• An e2e example gated on a reachable llama-server (the llamacpp marker already exists).
• The full chain validated locally on Apple Silicon: convert, serve, then guardian, uncertainty and answerability through mellea. f16 only; quantized GGUFs are untested.

Out of scope

Standalone aLoRA adapters on llama-server: a base model plus per-adapter GGUF side files, selected per request through llama-server's lora field. llama-server supports that, and it is more flexible than Ollama's one-tag-per-adapter model, but wiring it means a new server-mediated binding reality and touches the Epic #929 binding taxonomy. That deserves its own issue if we want it.

Line references are against 246f7ee.