Skip to content

exp/decide: add Open for GGUF models that describe their readout - #441

Merged
deadprogram merged 2 commits into
exp-decide-escape-specialfrom
exp-decide-typed
Oct 6, 2026
Merged

deadprogram merged 2 commits into
exp-decide-escape-specialfrom
exp-decide-typed

Conversation

@deadprogram

Copy link
Copy Markdown
Member

Stacked on #438. Merge #437 and #438 first.

ggml-org/llama.cpp#29818 stores the decision type, temperatures and a systemone prompt template in the GGUF. decide.Open reads them, so the ggml-org OpenJev and Lev GGUFs need no config file.

  • The prompt is the GGUF's systemone template, rendered with ardanlabs/jinja. JSON values are written first as the llama.cpp tojson filter writes them, with the same separators, key order and float format.
  • OpenJev reads the letters A to Z and a to z at the last token.
  • Lev reads A to Z and then AA to ZZ, reads choice questions in both orders and averages them, sorts JSON keys, and reads noul as a rating from 0 to 8.
  • Temperatures use the per-type buckets from the GGUF.
  • ParseRequest now writes non-string instructions and descriptions in the same tojson format.
  • The decide example gets -readout gguf, which needs no -config.

Tested with

  • go test ./exp/decide, and the wasm unit tests
  • TestLevModel with lev-Q8_0, exact and batched. Decide matches DecideMany exactly.
  • llama-server built at the merge commit of #29818, on the request from that PR. Every Lev probability matches within 6e-9, and the input tokens are the same (1194). It also runs on b11146.
  • TinyGo build of examples/wasm/decide, which grows by 38 KB.

OpenJev is covered by unit tests against its template only. The model is 19 GB at Q4_K_M and was not run.

llama.cpp PR 29818 stores the decision type, temperatures and a
systemone prompt template in the GGUF. Open reads them, so OpenJev
and Lev need no config file. The template is rendered with JSON
written as the llama.cpp tojson filter does. Lev matches llama-server
within 6e-9 on the PR test request. The decide example gets -readout gguf.

Signed-off-by: deadprogram <ron@hybridgroup.com>
The Windows runner checks files out with CRLF, which broke
TestTypedPrompt.

Signed-off-by: deadprogram <ron@hybridgroup.com>
@deadprogram
deadprogram merged commit 4c58768 into main Oct 6, 2026
6 checks passed
@deadprogram
deadprogram deleted the exp-decide-typed branch October 6, 2026 12:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant