Skip to content

LFM2.5 plain-tool-list templates get the generic JSON tool prompt; llama.cpp routes them to LFM2.5 #716

Closed

Description

@leehack

Tool prompts for LFM2 models differ from llama-server in two ways.

1. LFM2.5 templates with a plain tool list get the generic JSON tool prompt

detectChatFormat sends a template with List of tools: [ to ChatFormat.lfm2 only if it also contains <|tool_call_start|> and <|tool_call_end|> (lib/src/core/template/chat_format.dart:332 at fa9561838). llama.cpp only requires List of tools: [ without <|tool_list_start|> (common/chat.cpp:1158-1163 at 7fe450e19, and the same at b10549).

Templates affected:

  • LFM2.5-1.2B-Instruct: llama.cpp's models/templates/LFM2.5-Instruct.jinja, byte-identical to the template in LFM2.5-1.2B-Instruct-Q4_K_M.gguf.
  • LFM2.5-1.2B-Thinking: test/fixtures/templates/LFM2_5-1_2B-Thinking.jinja.

Without tools they render as content-only. With tools they take the generic handler.

Cause: llamadart. llama_cpp_template_detection_integration_test.dart:68 and chat_format_test.dart pin contentOnly for these templates.

Repro at fa9561838: ChatTemplateEngine.render(templateSource: <LFM2_5-1_2B-Thinking.jinja>, messages: [user 'Weather in Paris?'], metadata: {}, tools: [get_weather(city)]). The result's format is generic and its prompt is:

<s><|im_start|>system
Respond in JSON format, either with `tool_call` (a request to call tools) or with `response` reply to the user's request
List of tools: [{"type": "function", "function": {"name": "get_weather", ...}}]<|im_end|>
<|im_start|>user
Weather in Paris?<|im_end|>
<|im_start|>assistant

llama-server 7fe450e19 POST /apply-template for the same input:

<|im_start|>system
List of tools: [{"type": "function", "function": {"name": "get_weather", ...}}]<|im_end|>
<|im_start|>user
Weather in Paris?<|im_end|>
<|im_start|>assistant

2. Lfm2Handler flattens the tool list

Lfm2Handler._serializeToolsForTemplate (lib/src/core/template/handlers/lfm2_handler.dart:151) passes each tool as {"name", "description", "parameters"}. llama-server passes the OpenAI shape.

For LFM2-8B-A1B.jinja with tools:

  • llamadart: List of tools: <|tool_list_start|>[{"name": "get_weather", ...
  • llama-server: List of tools: <|tool_list_start|>[{"type": "function", "function": {"name": "get_weather", ...

LiquidAI's LFM2 model card shows the flat form. This one is a choice between the model card and llama-server, not a clear bug. Settle it when fixing part 1, because routing LFM2.5 through Lfm2Handler would apply the flat form to those templates too.

Found in the prompt differential for #714. This behavior is pre-existing, not caused by dinja.

Expected

  • Route plain-list LFM2.5 templates as llama.cpp does, and pin the result with /apply-template fixtures.
  • Decide whether to follow the model card or llama-server for the tool list shape, and record the decision in the handler test.

Activity

  1. added
    bugSomething isn't working
    structured-outputJSON schema, grammar, and constrained output work
    priority:P2Planned next: useful unblocked work or validation after P1 items
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority:P2Planned next: useful unblocked work or validation after P1 itemsstructured-outputJSON schema, grammar, and constrained output work

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions