[leehack]

Summary

Closes #716
Closes #720

Detects template capabilities the way llama.cpp 7fe450e19 does, gives each template the content form llama.cpp gives it, and routes LFM2.5 tool lists to the LFM2 format.

• #720: JinjaAnalyzer inferred string/typed content caps from the template's structure, and got SmolVLM, TranslateGemma 2B (typed-only), Ministral 3 and LFM2.5-8B (string-only) wrong. llama.cpp instead renders fixed probe conversations and reads which input values the template used (jinja::caps_get).
  • New JinjaUsageProbe rewrites the parsed template so each identifier, member access, filter, test and for iterable passes through a recording function, then renders it with dinja. As in llama.cpp, a for loop over a string throws.
  • JinjaAnalyzer runs llama.cpp's probes on it for system role, tools, tool calls, parallel calls, string/typed content and object arguments. supportsThinking still comes from thinking markers.
  • The rewrite writes back every construct dinja parses, including floats Dart prints with an exponent and *args parameters, so the written source parses to the same AST. A test writes, re-parses and compares every fixture template plus generated literal and signature forms.
  • If a template still cannot be prepared for the probes, it gets base's analysis (LegacyJinjaAnalyzer, base's JinjaAnalyzer renamed), never default caps.
  • A probe that throws is one of llama.cpp's outcomes: logged at debug level and cached. Only a template that does not parse is left uncached.
• Content form, as llama.cpp's messages_inp_normalizer and to_json_oaicompat build it:
  • A typed-only template gets every string content as one text part, with media as the <__media__> marker inside the text. Only templates with both caps get media as typed parts.
  • Tool results stay strings on the typed path. This fixes Kimi-K2 printing [{'type': 'text', 'text': 'sunny'}] after an image.
  • TranslateGemma builds typed parts only for user turns (they carry the language codes); other turns keep strings.
  • Ministral turns empty string content into an empty text block, as llama.cpp does. Without it the Ministral 3 14B template throws on an image followed by an assistant turn without text.
• #716: LFM2 detection now matches llama.cpp: <|tool_list_start|> and <|tool_list_end|> (LFM2), or List of tools: [ without <|tool_list_start|> (LFM2.5). llamadart also required <|tool_call_start|>, so LFM2.5-1.2B-Instruct and -Thinking fell to the generic format and got a stray "Respond in JSON format" instruction.
  • Lfm2Handler keeps passing each tool flat (name, description, parameters), as the LFM2 and LFM2.5 model cards show. This deliberately differs from llama-server, which passes the OpenAI shape (type, function). The LFM2.5 parity case and the differential ignore only that shape and compare the rest exactly.

Production-readiness scope

• User-facing scope: SmolVLM prompts keep the message text; Ministral 3 and TranslateGemma 2B render an image followed by a reasoning-only turn; Kimi-K2 tool results after an image are plain text; LFM2.5-1.2B tool prompts use the LFM2 format.

• Supported platforms/paths: All. The change is in the shared Dart template path (native, WebGPU, LiteRT-LM templates).

• Unsupported or intentionally unavailable paths: None added. A template dinja cannot parse still falls back to the regex caps, uncached and logged.

• Out of scope / follow-ups: Gaps that exist at base and are unchanged here:

  • llamadart routes tool requests on templates without tool support to the generic JSON tool format; llama-server adds nothing. SmolVLM now shows this, since its text renders (24 of the 25 differential renders counted as away, below).
  • The TranslateGemma handler defaults language codes to en-GB; llama.cpp removed its TranslateGemma handler (ggml-org/llama.cpp#18675) and prints empty codes.
  • Templates with both caps get media as typed parts, which llama.cpp never passes to a template.
  • Lfm2Handler builds a grammar only for required or the force-JSON marker; llama.cpp's lfm2.cpp builds a lazy Python-style tool-call grammar whenever tools are present. LFM2.5 now inherits this.

  Follow-ups from this PR's audit (non-blocking):

  • JinjaAnalyzer and LegacyJinjaAnalyzer duplicate about 70 lines (_supportsThinking, _findAll). The legacy analyzer could be deleted once the probe is trusted, since it only runs for a template the probe cannot prepare.
  • FireFunction v2 now logs the warning "Template appears to support tool-call output but does not advertise tool definitions", because its caps now match /props (tools false, tool calls true). The warning is misleading for this template: its prompt and grammar are unchanged.

Completeness checklist

• Declared scope is fully implemented, or explicitly reduced above.
• Unsupported platform/option combinations fail loudly with actionable diagnostics or are clearly documented as unavailable.
• Public API docs, README/website docs, examples, support matrices, and changelog entries are updated where relevant. No public API change. Changelog in both files; the model-family table lists LFM2.5 Instruct under LFM2; the template-engine internals page describes the probes.
• Regression coverage covers the original issue plus key negative/version-skew paths where relevant.
• Security/privacy review completed: no secrets, bearer tokens, signed URLs, or raw secret-bearing paths leak through logs, cache keys, metadata, errors, or snapshots.
• Follow-up work that is useful but not required for this PR is tracked in GitHub Issues, or explicitly marked "None" above.

PR type guidance

• Bugfix PR: Root causes are above. Regression tests:
  • template_message_json_parity_test.dart: 5 new llama-server /apply-template cases, each failing at base db7b45881: SmolVLM text (typed-only), Ministral 3 and TranslateGemma 2B image plus reasoning-only turn, Ministral 3 14B image plus empty assistant turn, LFM2.5 with tools. Kimi-K2's image case now compares everything after the user text, including the tool result (fails at base). A new test checks detected caps against the chat_template_caps /props reported for these templates.
  • template_caps_cache_fixtures_test.dart pins caps for all 32 fixture templates; each bit but supports_thinking equals llama-server /props.
  • jinja_usage_probe_test.dart covers the probe; jinja_analyzer_test.dart covers probe outcomes.
  • specialized_tool_grammar_validation_e2e_test.dart: the LFM2.5 required grammar compiles and accepts or rejects calls (fails at base: generic format).
  • Platforms: all.
• Feature PR: N/A, bugfix.
• Docs-only PR: N/A.

Test Plan

At head 3ab42978e, run serially on a loaded shared Mac.

• dart run tool/prepare_workspace.dart: no tracked changes, at head and base.
• dart format --output=none --set-exit-if-changed .
• dart analyze
• dart run tool/testing/check_platform_boundaries.dart
• dart test -p vm -j 1 --exclude-tags local-only: 3,561 pass, 4 skipped; base db7b45881: 3,502 test names. 22 base names are gone, all renamed: 16 render-context variants now also named by typedContentOnly; LFM2.5 detection now expects LFM2; the probe-failure cache test now expects caching; 4 jinja_analyzer_test.dart tests now name the llama.cpp probe outcome they assert. None removed against the previous head b37c64868.
• dart test -p chrome --exclude-tags local-only: 1,703 pass; base: 1,655 test names. 21 base names are gone, the same renames less the VM-only detection test. None removed against b37c64868.
• Other targeted/local-only validation:
  • ./tool/testing/run_template_parity_suites.sh passes: detection 74, diagnostic 96, unit template 800, upstream chat tests 4, compiled grammar 18.
  • (cd example/chat_app && flutter test): 491 pass.
  • flutter test --platform chrome test/chat_generation_service_test.dart passes.

Caps against llama-server 7fe450e19 /props (the 7 fields both report): 97 of 97 templates llama-server loads (base: 90), and 173 of 173 distinct templates the VM unit tests passed to caps detection. One more, the empty template, is not comparable: llama-server replaces it with the model's default template.

Differential against llama-server 7fe450e19 /apply-template, base db7b45881 vs head:

• Templates: 98 (test/fixtures, tool/litert_lm_templates, llama.cpp models/templates, and templates from local GGUF files).
• Conversations: 27 text (tool-call-only, reasoning-only, empty assistant, parallel calls, string/Map/List/number/null results, spaced/nested/unicode/malformed arguments, system turns) and 10 with an image (image alone, text first, system, reasoning-only, empty assistant, tool calls, Map/List/parallel typed results), with and without tools as each case defines. 3,626 renders per side; 3,192 have a server prompt.
• Server: text on stories15M; image conversations on LFM2-VL-450M with its mmproj. Random <__media_…__> markers, the leading BOS and dates are normalized; LFM2 tool lists are compared in the flat shape.
• Identical to the server: 2,063 at base, 2,127 at head.

Result	Text	Image
Unchanged	2,598	897
Changed, now identical to the server	21	35
Changed, closer to the server	1	34
Newly render, identical to the server	6	2
Newly render, server fails to load the template	1	5
Changed, no server prompt	0	1
Changed away from the server (edit distance)	19	6
Newly throw	0	0

The 25 away:

• 24 SmolVLM tool conversations. At base, message text rendered empty; now it renders, and with tools it carries the pre-existing generic tool instruction and tool-call JSON. With those two removed, 23 equal the server; the last (res_str) differs only in llama-server continuing a final assistant turn, which llamadart does not do (pre-existing).
• TranslateGemma 2B image plus reasoning turn: base dropped the user text, head prints <__media__>Describe; the rest differs by the pre-existing en-GB codes.

Grammar, triggers, stops, preserved tokens, parser and thinking-forced-open change only for LFM2.5-1.2B (format generic to LFM2, intended) and for 38 image renders, each of which now equals its text analogue.

Mutations, each run against the tests named and each failing:

Mutation	Fails
Caps probe keeps string content for a template that reads it as a list and does not print it	TranslateGemma parity case; /props caps test
No text-part wrapping for typed-only templates	SmolVLM parity case
Media parts to typed-only templates	typed-only templates get the audio marker in a text part
LFM2.5 detection requires `<	tool_call_start
TranslateGemma wraps every turn in text parts	TranslateGemma parity case
Ministral drops empty text blocks	Ministral 3 14B parity case
Tool results as text parts on the typed path	Kimi-K2 parity case
Probe does not mark tojson input deeply	marks every nested value used when tojson reads a value
Probe iterates strings	throws on a for loop over a string, as llama.cpp does
Floats written with Dart toString	both float repros; round-trips literals in every expression position
Negative numbers not parenthesized	round-trips signatures, calls, filters, tests and statements
*args parameters rejected	macro repro; round-trips signatures, calls, filters, tests and statements
Object literals not parenthesized	round-trips signatures, calls, filters, tests and statements
Default caps instead of the legacy fallback	falls back to the legacy analysis when the probe cannot be prepared

Matrix Evidence

Matrix row	Scope covered	Platform / model / backend	Result	Evidence / notes
static-format-analyze	format, analyzer, import boundaries	macOS arm64, Flutter 3.47.1	PASS	Test Plan
root-vm	VM unit/integration	macOS arm64	PASS	3,561 pass
root-chrome	browser unit/integration	Chrome	PASS	1,703 pass
template-parity	llama.cpp template detection, render, parse parity and compiled GBNF	macOS arm64, llama.cpp b10549 templates	PASS	run_template_parity_suites.sh
structured-output-adversarial	rendering/history, compiled grammar, upstream parity	llama-server 7fe450e19	PASS	parity fixtures, differential, mutations above
examples-tests	chat app	Flutter	PASS	491 pass; Chrome chat_generation_service_test.dart 4 pass
high-risk-exact-head-independent-qa	independent review of exact head	3ab42978e vs db7b45881	PASS	Accepted by claude-opus-5-5-fresh-subagent-audit721: 0 PR-caused P1s, 0 unresolved threads
Other rows	N/A	N/A	N/A	Template-path change only; no runtime, native, web bridge, release or device path changes.

Review Notes

• Independent review status: a fresh auditor accepted b37c64868 against db7b45881 with 0 P1s. It asked for one PR-caused, non-blocking fix before merge: the probe rewriter threw on floats Dart prints with an exponent and on *args macros, leaving default caps (tools, tool calls and parallel calls false; base and llama-server report true). 3ab42978e fixes it: the three repros failed at b37c64868 and pass now. At 3ab42978e, caps and prompts equal b37c64868's on all 3,626 differential renders. A delta audit then accepted 3ab42978e against db7b45881 with 0 P1s and 0 unresolved threads (record below).
• CI status / head SHA: green at 3ab42978e (23 pass, 5 skipped).

High-risk regression review

• Classification: high-risk (structuredOutput), from classify_high_risk_changes.dart.
• Implementation task: this PR's author session.
• Independent blocking QA task: accepted, two rounds (b37c64868, then the 3ab42978e delta). tool/testing/high_risk_readiness.dart on the evidence with this record exits 2 (unverifiedPrerequisites). Record:

{"auditor_identity":"claude-opus-5-5-fresh-subagent-audit721","audit_kind":"operator-owned","audit_head_sha":"3ab42978eecd6d64495edbf7815cce3a99849eb3","audit_base_sha":"db7b45881d8a92cc6d9d1e0aabfe7e368e9856a1","decision":"accepted","unresolved_review_threads":0,"known_pr_caused_p1_regressions":0,"summary":"2026-09-25, two rounds. R1 (b37c6486): own harness vs own llama-server 7fe450e19 build, 95 templates x 36 conversations in 4 variants: 0 new throws, caps 94/94 vs /props (base 87), 2334 vs 2282 of 3268 identical to the server, 20 away checked hunk by hunk (19 SmolVLM, 1 TranslateGemma, all old behaviour); FunctionGemma real run byte-identical to base; FireFunction prompt and grammar unchanged plus a warn log; LFM2.5 gains native parallel calls; 9/9 mutations red; gates green. Flagged a non-blocking rewriter fallback to default caps on exponent floats and *args. R2 (3ab42978, delta): repros fixed; caps and renders identical to b37c6486 on all 3420x4 renders and 87 synthetic templates; synthetic caps equal /props on all 81 server-loadable templates; 183-template corpus round-trips; 25k fuzzed templates (8222 parse): 0 failures outside the dinja `not`-as-identifier quirk on invalid Jinja (163 cases fall back to base or probe unary not); legacy fallback is base's analyzer verbatim, so never worse than base, and fixture round-trip tests keep writer bugs visible; about 70 duplicated lines, non-blocking; 5/5 delta mutations red; VM 3485+78 skipped, Chrome 1702, parity, chat_app green; CI 23 pass and 5 skipped; 0 threads; merges cleanly with main (#724 docs only)."}

• Exact head / current base: 3ab42978e / db7b45881.
• Production-branch deletion, bypass, or miswire proof: mutation table above.
• Affected-family real-model/artifact evidence: no local weights for these families; primary upstream emissions (llama-server 7fe450e19 /apply-template and /props) pinned as durable fixtures.
• Explicit unavailable-family or other N/A evidence: SmolVLM, Ministral 3, TranslateGemma 2B, LFM2.5, Kimi-K2: unavailable weights, fixtures above.
• Known PR-caused P1 regressions: 0
• Unresolved review threads: 0

[github-actions]

Chat app preview removed for leehack/llamadart-chat-pr-721.