Skip to content

model : add LiquidAI/d1-omni-600M decision model - #30114

Merged
ngxson merged 4 commits into
ggml-org:masterfrom
tdakhran:tarek/feat/d1-omni-600M
Oct 8, 2026
Merged

model : add LiquidAI/d1-omni-600M decision model#30114
ngxson merged 4 commits into
ggml-org:masterfrom
tdakhran:tarek/feat/d1-omni-600M

Conversation

@tdakhran

@tdakhran tdakhran commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overview

Add support for the LiquidAI/d1-omni-600M omni decision model.
Supports audio, images, and text as inputs.

Huge thanks to @ngxson for the help with development!

GGUFs for testing with examples LiquidAI/d1-omni-600M-GGUF

Additional information

Depends on #30110.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES, prototyping and correctness testing, reviewed by a human.

@github-actions github-actions Bot added documentation Improvements or additions to documentation model Model specific server mtmd Related to multimodal functionality (video/image/audio) conversion labels Oct 7, 2026
@tdakhran
model : add LiquidAI/d1-omni-600M decision model
a5b036c 
Assisted-by: Claude Opus 5.5
@tdakhran
mtmd : keep conformer GLU sigmoid on CUDA
bc271ba 
Assisted-by: Claude Opus 5.5
@tdakhran
server : take d1omni audio through images and input_audio, scope memo…
56f9519 
…ry-less lfm2 to non-causal

Assisted-by: Claude Opus 5.5
@tdakhran
common : rename decision type d1omni to lfm2-d1-omni, server : make i…
59f8fe7 
…mages an alias of files

Assisted-by: Claude Opus 5.5
@tdakhran
tdakhran force-pushed the tarek/feat/d1-omni-600M branch from 25f6c3f to 59f8fe7 Compare October 7, 2026 20:26
@tdakhran
tdakhran marked this pull request as ready for review October 7, 2026 20:26
@tdakhran
tdakhran requested review from a team and CISC as code owners October 7, 2026 20:26
CISC
CISC approved these changes Oct 7, 2026
View reviewed changes
Comment thread conversion/lfm2.py
Comment on lines +433 to +435
for old, new in (("adapter.norm", 0), ("adapter.linear_1", 1), ("adapter.linear_2", 3),
("residual.ln", 4), ("residual.down", 5), ("residual.up", 6)):
name = name.replace(f"audio.{old}.", f"audio_adapter.model.{new}.")

@CISC CISC Oct 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one was mildly confusing. :)

@tdakhran tdakhran Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A small sacrifice to reuse the existing LFM2-Audio encoder implementation.

@tdakhran

tdakhran commented Oct 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

CI error looks unrelated, passes on my laptop

❯ ./build/bin/test-llama-archs -s 1
0.01.744.847 I ggml_cuda_init: found 1 CUDA devices (Total VRAM: 8187 MiB):
0.01.744.869 I   Device 0: NVIDIA GeForce RTX 4070 Laptop GPU, compute capability 8.9, VMM: yes, VRAM: 8187 MiB
0.01.746.192 I main: using seed 1, stdev 0.100000
|     Model arch.|                            Device|Config|   NMSE vs. CPU|Roundtrip|    Mixed batch|
|----------------|----------------------------------|------|---------------|---------|---------------|
|            clip|NVIDIA GeForce RTX 4070 Laptop GPU| Dense|SKIP           |     SKIP|SKIP           |
|            clip|    Intel(R) Core(TM) Ultra 9 185H| Dense|SKIP           |     SKIP|SKIP           |
|            clip|                              Meta| Dense|SKIP           |     SKIP|SKIP           |
|           llama|NVIDIA GeForce RTX 4070 Laptop GPU| Dense|  OK (1.22e-07)|       OK|  OK (1.56e-08)|
...
|      k2-horizon|NVIDIA GeForce RTX 4070 Laptop GPU|   MoE|  OK (2.51e-07)|       OK|  OK (2.09e-08)|
|      k2-horizon|    Intel(R) Core(TM) Ultra 9 185H|   MoE|  OK (0.00e+00)|       OK|  OK (2.86e-10)|
|      k2-horizon|                              Meta|   MoE|SKIP           |     SKIP|SKIP           |
Summary: all 375 test(s) passed

@ngxson
ngxson merged commit a657f7e into ggml-org:master Oct 8, 2026
19 of 20 checks passed
@ykhrustalev ykhrustalev mentioned this pull request Oct 8, 2026
Merged
@tdakhran
tdakhran deleted the tarek/feat/d1-omni-600M branch October 8, 2026 06:08
@CISC CISC added the highlight Changes that will be highlighted in the next release notes label Oct 8, 2026
@anis-marrouchi anis-marrouchi mentioned this pull request Oct 8, 2026
Open
4 tasks
kmbandy added a commit to kmbandy/llama.cpp that referenced this pull request Oct 8, 2026
@kmbandy @claude
Merge upstream/master (140 commits, bed0a85 -> c35b667) into fork
523e2fd 
Brings LiquidAI d1-3B / d1-omni-600M decision models (ggml-org#30110, ggml-org#30114),
CLEF vision, pplx-decider, LFM2.5 encoder, split router child stdout/stderr
(ggml-org#29895), MoE expert-copy callback (ggml-org#29943) and general upstream fixes.

27 conflicted files (69 hunks) resolved keeping both sides; fork-line audit
on every conflicted file and on the 57 auto-merged co-touched files: no fork
lines lost. server-node.cpp ported to the split subproc API. Server keeps the
fork's llama_batch decode path and routes decision/joint-head batches through
llama_process (common_batch). Fork's non-fatal spec/checkpoint restore kept.
Notes: ~/.cache/llama-sync-20261008/ (reports, review, rulings).
Backup: backup/pre-upstream-sync-2026-10-08 @ 49e0f85. Not yet compiled.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This was referenced Oct 9, 2026
Merged
Open
ojungo69 added a commit to ojungo69/oboete that referenced this pull request Oct 9, 2026
@ojungo69 @claude
docs(spec): d1 may also run on a local llama.cpp server when #336 is …
58a82df 
…built (#416)

Part of #336.

## What

The owner asked on 2026-10-09:
「採用予定のd1がローカル運用可能になったから判断モデルを導入する時にローカル運用も候補に入れて」 (d1 now runs locally,
so make running it locally a candidate when the judgement model is
introduced).

- `docs/spec.md`, under decision 38: a dated paragraph, "A local d1
(owner, 2026-10-09)". Decision 38 had left a local endpoint to a setting
"once Ollama or llama.cpp ship it"; llama.cpp now ships it.
- 3.5 and 4 (B): d1 may run on Liquid's hosted API or on a local
llama.cpp server. Both remain candidates; nothing becomes a default.
- 4 and A109: a local server the user runs (Ollama for curation, a
llama.cpp decision server for #336) is a provider endpoint, not a model
oboete loads or serves. A109 is a Claude decision; this states its scope
rather than changing it.
- `docs/spec-webui.md`, "Additional admitted AI roles": d1's choices
include the local server (a loopback endpoint with a connection test and
no key).

## Facts checked (2026-10-09, primary sources)

- Liquid AI, [Open d1](https://www.liquid.ai/blog/d1-open) (2026-10-07):
open weights for d1-3B (3.12B, text and images, 32K context) and the
experimental d1-omni-600M, LFM Open License v1.0. d1-3B's [model
card](https://huggingface.co/LiquidAI/d1-3B) lists Japanese among its
languages.
- llama.cpp: `/v1/systemone` in
[`tools/server/README.md`](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
("Follows the TypeSafe API"), d1-3B in ggml-org/llama.cpp#30110
(2026-10-07), d1-omni-600M in ggml-org/llama.cpp#30114 (2026-10-08).
[d1-3B-GGUF](https://huggingface.co/LiquidAI/d1-3B-GGUF): Q8_0 2.87 GB,
BF16 5.4 GB.
- Hosted [d1](https://docs.liquid.ai/lfm/models/d1): `POST
https://api.liquid.ai/decisions/v1/systemone` with `model`, `state`,
`questions`; "Each question is charged for its text and all supplied
images"; `d1:free` is text-only.
- Cautions carried into the single evaluation: Japanese decision quality
is unmeasured; a quantized decision head has collapsed toward uniform
probabilities before (ggml-org/llama.cpp#30064, Clef); llama.cpp states
the probabilities are not calibrated for the user's data.

Nothing was downloaded or run: no model, no inference, no evaluation
(decision 34).

This is a design-doc PR: after two consecutive review rounds whose
findings are mechanism only, the rest goes to #336 as acceptance tests
(CLAUDE.md).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01Vs2me5nWqSZLFn65f7iASY


<!-- This is an auto-generated description by cubic. -->
<a
href="https://cubic.dev/pr/ojungo69/oboete/pull/416?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="View guided diff"
src="https://www.cubic.dev/buttons/review-in-cubic-light.svg"></picture></a>
<a
href="https://www.cubic.dev/action/auto-fix/pr/ojungo69/oboete/416?returnTo=https%3A%2F%2Fgithub.com%2Fojungo69%2Foboete%2Fpull%2F416&source=description"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"><img
alt="Turn on auto-fix"
src="https://www.cubic.dev/buttons/turn-on-auto-fix-light.svg"></picture></a>
<!-- End of auto-generated description by cubic. -->

## Summary by Sourcery

Document local llama.cpp hosting as an additional, non-default d1
provider option for #336’s judgement model.

Enhancements:
- Document Liquid AI d1 as a candidate for both hosted use and
user-managed local llama.cpp servers when #336’s judgement and reranking
roles are implemented.
- Clarify that user-run model servers are provider endpoints and
preserve existing fallback, privacy, cost, and evaluation requirements.

Documentation:
- Update the WebUI and specification documents to cover local llama.cpp
configuration for d1, including loopback connectivity testing and
keyless operation.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@tdakhran

tdakhran commented Oct 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Would it be possible to add this model to https://huggingface.co/collections/ggml-org/decision-models?

Update: already added! Thanks!

@ykhrustalev ykhrustalev mentioned this pull request Oct 10, 2026
Merged
@ggerganov ggerganov mentioned this pull request Oct 15, 2026
Merged
@christopherthompson81 christopherthompson81 mentioned this pull request Oct 22, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@CISC CISC CISC approved these changes
@ngxson ngxson ngxson approved these changes

Assignees

No one assigned

Labels

conversion documentation Improvements or additions to documentation highlight Changes that will be highlighted in the next release notes model Model specific mtmd Related to multimodal functionality (video/image/audio) server

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tdakhran @CISC @ngxson