model : add LiquidAI/d1-3B decision model by tdakhran · Pull Request #30110 · ggml-org/llama.cpp

tdakhran · 2026-10-07T18:05:36Z

Overview

Add support for the LiquidAI/d1-3B vision decision model.

Huge thanks to @ngxson for the help with development, and @kbenkhaled for testing!

GGUF repository with examples for testing LiquidAI/d1-3B-GGUF.

Additional information

Add a new GGUF key for the image interpolation algorithm.

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES, prototyping and correctness testing, reviewed by a human.

mtmd : read LFM2 image resize algo from GGUF Assisted-by: Claude Opus 5.5

Assisted-by: Claude Opus 5.5

ServeurpersoCom · 2026-10-07T20:01:43Z

Hi, LGTM, matches prompt.py. Note the converter now writes the resize algo for all LFM2-VL, so reconverting LFM2-VL-1.6B (resample 3) switches it from bilinear to bicubic: intended?

Checked the reference code: it's a fidelity improvement!

tdakhran · 2026-10-07T20:21:09Z

Hi, LGTM, matches prompt.py. Note the converter now writes the resize algo for all LFM2-VL, so reconverting LFM2-VL-1.6B (resample 3) switches it from bilinear to bicubic: intended?

Checked the reference code: it's a fidelity improvement!

Thanks for the quick review! I kept it to default to what was there before, only newly converted ggufs will pick up the resizing algorithm.
In the end, the resizing algorithm doesn't drastically change the quality of the output.

ServeurpersoCom · 2026-10-07T20:26:47Z

Makes sense, thanks!

Brings LiquidAI d1-3B / d1-omni-600M decision models (ggml-org#30110, ggml-org#30114), CLEF vision, pplx-decider, LFM2.5 encoder, split router child stdout/stderr (ggml-org#29895), MoE expert-copy callback (ggml-org#29943) and general upstream fixes. 27 conflicted files (69 hunks) resolved keeping both sides; fork-line audit on every conflicted file and on the 57 auto-merged co-touched files: no fork lines lost. server-node.cpp ported to the split subproc API. Server keeps the fork's llama_batch decode path and routes decision/joint-head batches through llama_process (common_batch). Fork's non-fatal spec/checkpoint restore kept. Notes: ~/.cache/llama-sync-20261008/ (reports, review, rulings). Backup: backup/pre-upstream-sync-2026-10-08 @ 49e0f85. Not yet compiled. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

…built (#416) Part of #336. ## What The owner asked on 2026-10-09: 「採用予定のd1がローカル運用可能になったから判断モデルを導入する時にローカル運用も候補に入れて」 (d1 now runs locally, so make running it locally a candidate when the judgement model is introduced). - `docs/spec.md`, under decision 38: a dated paragraph, "A local d1 (owner, 2026-10-09)". Decision 38 had left a local endpoint to a setting "once Ollama or llama.cpp ship it"; llama.cpp now ships it. - 3.5 and 4 (B): d1 may run on Liquid's hosted API or on a local llama.cpp server. Both remain candidates; nothing becomes a default. - 4 and A109: a local server the user runs (Ollama for curation, a llama.cpp decision server for #336) is a provider endpoint, not a model oboete loads or serves. A109 is a Claude decision; this states its scope rather than changing it. - `docs/spec-webui.md`, "Additional admitted AI roles": d1's choices include the local server (a loopback endpoint with a connection test and no key). ## Facts checked (2026-10-09, primary sources) - Liquid AI, [Open d1](https://www.liquid.ai/blog/d1-open) (2026-10-07): open weights for d1-3B (3.12B, text and images, 32K context) and the experimental d1-omni-600M, LFM Open License v1.0. d1-3B's [model card](https://huggingface.co/LiquidAI/d1-3B) lists Japanese among its languages. - llama.cpp: `/v1/systemone` in [`tools/server/README.md`](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) ("Follows the TypeSafe API"), d1-3B in ggml-org/llama.cpp#30110 (2026-10-07), d1-omni-600M in ggml-org/llama.cpp#30114 (2026-10-08). [d1-3B-GGUF](https://huggingface.co/LiquidAI/d1-3B-GGUF): Q8_0 2.87 GB, BF16 5.4 GB. - Hosted [d1](https://docs.liquid.ai/lfm/models/d1): `POST https://api.liquid.ai/decisions/v1/systemone` with `model`, `state`, `questions`; "Each question is charged for its text and all supplied images"; `d1:free` is text-only. - Cautions carried into the single evaluation: Japanese decision quality is unmeasured; a quantized decision head has collapsed toward uniform probabilities before (ggml-org/llama.cpp#30064, Clef); llama.cpp states the probabilities are not calibrated for the user's data. Nothing was downloaded or run: no model, no inference, no evaluation (decision 34). This is a design-doc PR: after two consecutive review rounds whose findings are mechanism only, the rest goes to #336 as acceptance tests (CLAUDE.md). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Vs2me5nWqSZLFn65f7iASY  <a href="https://cubic.dev/pr/ojungo69/oboete/pull/416?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="View guided diff" src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a> <a href="https://www.cubic.dev/action/auto-fix/pr/ojungo69/oboete/416?returnTo=https%3A%2F%2Fgithub.com%2Fojungo69%2Foboete%2Fpull%2F416&source=description" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img alt="Turn on auto-fix" src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>  ## Summary by Sourcery Document local llama.cpp hosting as an additional, non-default d1 provider option for #336’s judgement model. Enhancements: - Document Liquid AI d1 as a candidate for both hosted use and user-managed local llama.cpp servers when #336’s judgement and reranking roles are implemented. - Clarify that user-run model servers are provider endpoints and preserve existing fallback, privacy, cost, and evaluation requirements. Documentation: - Update the WebUI and specification documents to cover local llama.cpp configuration for d1, including loopback connectivity testing and keyless operation. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

tdakhran · 2026-10-09T18:48:36Z

Would it be possible to add this model to https://huggingface.co/collections/ggml-org/decision-models?

ngxson · 2026-10-09T19:29:05Z

yes I just added d1 and d1-omni

tdakhran · 2026-10-09T19:33:48Z

Thank you, @ngxson!

**Problem** Liquid AI released [d1-3B](https://huggingface.co/LiquidAI/d1-3B), a decision model (#2743). You give it a state and named questions (yes/no `noul`, `choice`, `score`), and it answers each question with probabilities from one forward pass. It writes no text. d1-3B is a stock `Lfm2VlForConditionalGeneration`, so the model builder already exports it and GenAI already runs it. What is missing is the d1 part: the prompt, reading the answer from the logits, and the answer format. Today every user has to rebuild these from d1's Python code. The tokenizer also needs the same regex fix as LFM2.5-VL-3B. **Solution** An example, a tutorial and tests. No runtime, builder or API change. - `examples/python/d1-decisions.py`: - builds d1's prompt for each question and runs one batch-1 prefill, with `max_length` set to the prompt length; - reads the logits at the last position; each option takes the best log-probability among its single-token forms (for example `A` and ` A`), then a softmax over the options; - returns d1's `{"answers": ..., "usage": ...}`. Token ids are built as d1 builds them: one question is one string; with several questions the shared prefix is encoded once. Each question's generator is freed before the next one starts. `--image` runs the `lfm2_vl` processor and vision model. - Before running the model, it refuses requests it would answer from less than the caller sent, or differently from d1: unknown request or question fields (the error names them), a missing `state`, missing or non-string `instructions`, an unknown `type`, bad `criteria` (a choice needs 2 or more options, a score 2 to 10 levels, `noul` keys are `true` and `false`), a NUL character, a prompt longer than the context, images on a text-only model, and a tokenizer without single-token yes/no, digits or option codes. Some of these are stricter than d1's own code, which for example treats an unknown `type` as a choice; the tutorial lists them. On the command line a refusal prints one `error: ...` line on stderr and exits with code 2. - The prompt and readout use the same format as llama.cpp's d1 support (ggml-org/llama.cpp#30110) and were checked against d1's reference code, which is not in this repo. Caller text is not escaped, as in d1 and llama.cpp. - `examples/python/d1-tutorial.md`: install, build (int8 CPU and fp16 CUDA, both with `prune_lm_head=true`), the tokenizer fix, a run, the request format, images (reuse the LFM2.5-VL-3B vision encoder at a pinned revision, build the embedding model from d1's own table), limitations, and measured accuracy and speed. - The d1-3B model is released under the LFM Open License v1.0. The tutorial links it and says that the script does not change its terms. Things to know: - **One prefill per question.** LFM2's state cannot be forked today, so several questions process the state again for each question. `usage.input_tokens` counts the shared part once, as d1 does. - **No image tiling.** GenAI resizes each image once; d1 tiles images above about 0.5 megapixels, so answers on large images differ. - **Batch 1 only**, so the padded-batch bugs fixed in #2745 and #2746 do not affect it. **Testing** - `test_d1_decisions.py`: 40 passed on macOS arm64 (onnxruntime-genai 0.18.0, ORT 1.31.0) on the tiny LFM2-VL model: golden prompts, readout against numpy, answers from crafted logits (including a tie), ids and usage for one and several questions, one image with one and with two questions, generator lifetime, refusals, and the command line (a request file with a byte order mark; setup messages go to stderr, so stdout is only the JSON). The same 40 pass on Linux with onnxruntime-genai 0.18.0 and onnxruntime-genai-cuda 0.18.0. `test_lfm2_vl_models.py` still passes. `test_lfm2_vl.py` has one new case: a config with d1's `auto_map` loads through the builder without remote code. - CI on an earlier revision: the Python tests passed on every job that runs them; the one red check (Windows CUDA, C# step, NuGet 401) is an infrastructure problem. Real model: d1-3B against transformers fp32 (`D1Model.system_one`), through `d1-decisions.py`. For text, the token ids and `usage` match transformers on every request below. 32 hand-written decisions (all three types; JSON, text, log and code states): | Machine, export | same top answer | max \|Δp\| | latency, 39 / 101 / 467 tokens | | --- | --- | --- | --- | | Linux, CUDA (H100), fp16 | 32/32 | 0.0033 | 5.5 / 6.0 / 9.0 ms | | Linux, CPU (Xeon 8480+), int8 | 32/32 | 0.0088 | 200 / 458 / 2039 ms | | Linux, CPU, fp32 | 32/32 | 2.5e-6 | 844 / 2037 / 9219 ms | | macOS, WebGPU (M3 Ultra), fp32 | 32/32 | 6.8e-6 | 88 / 110 / 296 ms | Public sets, first 1000 items, same top answer as transformers: - BoolQ: 998 (fp16, CUDA), 995 (int8, CPU) - MNLI matched: 997 (fp16, CUDA), 995 (int8, CPU) - POPE (images, the tutorial's int8 CPU pipeline): 998 Every changed answer was a close call (top two options within 0.09 in transformers). On CUDA, all 5 fp16 changes were exact ties from fp16 logits; bf16 was worse (BoolQ 994, MNLI 988), so the tutorial recommends fp16. - **Images:** the model card's two-cats picture gives P(two) = 0.98563 (transformers 0.98532), with the same 271 tokens. - **Tokenizer:** with the regex fix, GenAI's ids equal transformers' on 3,200 prompts in the 16 languages the model card lists. - **Trim parity:** the old and the trimmed script give identical prompts, ids, `usage` and probabilities on 31 valid requests on the test model, with its own logits and with random logits, and on d1-3B on the H100 box (int8 CPU and fp16 CUDA exports): all 32 decisions give the same ids, `usage` and probabilities as before (difference 0.0). Refs #2743

tdakhran added 2 commits October 7, 2026 09:54

model : add LiquidAI/d1-3b decision model

d145e9f

mtmd : read LFM2 image resize algo from GGUF Assisted-by: Claude Opus 5.5

common : rename decision type d1 to lfm2-d1

0786504

Assisted-by: Claude Opus 5.5

tdakhran requested review from a team and CISC as code owners October 7, 2026 18:05

github-actions Bot added documentation Improvements or additions to documentation server mtmd Related to multimodal functionality (video/image/audio) conversion labels Oct 7, 2026

tdakhran mentioned this pull request Oct 7, 2026

model : add LiquidAI/d1-omni-600M decision model #30114

Merged

CISC approved these changes Oct 7, 2026

View reviewed changes

ServeurpersoCom approved these changes Oct 7, 2026

View reviewed changes

ServeurpersoCom merged commit 88dcc46 into ggml-org:master Oct 7, 2026
14 checks passed

tdakhran deleted the tarek/feat/d1-3b branch October 7, 2026 20:21

ykhrustalev mentioned this pull request Oct 8, 2026

feat: add LiquidAI d1-3B and d1-omni-600M decision models Blaizzy/mlx-vlm#2467

Merged

CISC added the highlight Changes that will be highlighted in the next release notes label Oct 8, 2026

This was referenced Oct 9, 2026

docs(spec): d1 may also run on a local llama.cpp server when #336 is built ojungo69/oboete#416

Merged

judge/search: 要約前・検索後の選別を採用する（rerankerとの役割分担） ojungo69/oboete#336

Open

ggerganov mentioned this pull request Oct 15, 2026

llama.cpp : bump version to 0.7.0 #30603

Merged

ykhrustalev mentioned this pull request Oct 16, 2026

feat: add a d1-3B decision example and tutorial microsoft/onnxruntime-genai#2744

Merged

ykhrustalev mentioned this pull request Oct 21, 2026

mtmd : lfm2 resize matches torchvision #30943

Merged

christopherthompson81 mentioned this pull request Oct 22, 2026

LFM2: support LFM2-Audio, the bidirectional LFM2 encoders and embedders, and the d1 decision models christopherthompson81/inference.rs#582

Open

Navigation Menu

Repository navigation

model : add LiquidAI/d1-3B decision model - #30110

model : add LiquidAI/d1-3B decision model#30110
ServeurpersoCom merged 2 commits into
ggml-org:masterfrom
tdakhran:tarek/feat/d1-3b

tdakhran commented Oct 7, 2026 •

edited

Loading

Uh oh!

ServeurpersoCom commented Oct 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

tdakhran commented Oct 7, 2026

Uh oh!

ServeurpersoCom commented Oct 7, 2026

Uh oh!

tdakhran commented Oct 9, 2026

Uh oh!

ngxson commented Oct 9, 2026

Uh oh!

tdakhran commented Oct 9, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

tdakhran commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Additional information

Requirements

Uh oh!

ServeurpersoCom commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

tdakhran commented Oct 7, 2026

Uh oh!

ServeurpersoCom commented Oct 7, 2026

Uh oh!

tdakhran commented Oct 9, 2026

Uh oh!

ngxson commented Oct 9, 2026

Uh oh!

tdakhran commented Oct 9, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tdakhran commented Oct 7, 2026 •

edited

Loading

ServeurpersoCom commented Oct 7, 2026 •

edited

Loading