Repository navigation
LFM2.5 plain-tool-list templates get the generic JSON tool prompt; llama.cpp routes them to LFM2.5 #716
Description
Tool prompts for LFM2 models differ from llama-server in two ways.
1. LFM2.5 templates with a plain tool list get the generic JSON tool prompt
detectChatFormat sends a template with List of tools: [ to ChatFormat.lfm2 only if it also contains <|tool_call_start|> and <|tool_call_end|> (lib/src/core/template/chat_format.dart:332 at fa9561838). llama.cpp only requires List of tools: [ without <|tool_list_start|> (common/chat.cpp:1158-1163 at 7fe450e19, and the same at b10549).
Templates affected:
- LFM2.5-1.2B-Instruct: llama.cpp's
models/templates/LFM2.5-Instruct.jinja, byte-identical to the template inLFM2.5-1.2B-Instruct-Q4_K_M.gguf. - LFM2.5-1.2B-Thinking:
test/fixtures/templates/LFM2_5-1_2B-Thinking.jinja.
Without tools they render as content-only. With tools they take the generic handler.
Cause: llamadart. llama_cpp_template_detection_integration_test.dart:68 and chat_format_test.dart pin contentOnly for these templates.
Repro at fa9561838: ChatTemplateEngine.render(templateSource: <LFM2_5-1_2B-Thinking.jinja>, messages: [user 'Weather in Paris?'], metadata: {}, tools: [get_weather(city)]). The result's format is generic and its prompt is:
<s><|im_start|>system
Respond in JSON format, either with `tool_call` (a request to call tools) or with `response` reply to the user's request
List of tools: [{"type": "function", "function": {"name": "get_weather", ...}}]<|im_end|>
<|im_start|>user
Weather in Paris?<|im_end|>
<|im_start|>assistant
llama-server 7fe450e19 POST /apply-template for the same input:
<|im_start|>system
List of tools: [{"type": "function", "function": {"name": "get_weather", ...}}]<|im_end|>
<|im_start|>user
Weather in Paris?<|im_end|>
<|im_start|>assistant
2. Lfm2Handler flattens the tool list
Lfm2Handler._serializeToolsForTemplate (lib/src/core/template/handlers/lfm2_handler.dart:151) passes each tool as {"name", "description", "parameters"}. llama-server passes the OpenAI shape.
For LFM2-8B-A1B.jinja with tools:
- llamadart:
List of tools: <|tool_list_start|>[{"name": "get_weather", ... - llama-server:
List of tools: <|tool_list_start|>[{"type": "function", "function": {"name": "get_weather", ...
LiquidAI's LFM2 model card shows the flat form. This one is a choice between the model card and llama-server, not a clear bug. Settle it when fixing part 1, because routing LFM2.5 through Lfm2Handler would apply the flat form to those templates too.
Found in the prompt differential for #714. This behavior is pre-existing, not caused by dinja.
Expected
- Route plain-list LFM2.5 templates as llama.cpp does, and pin the result with
/apply-templatefixtures. - Decide whether to follow the model card or llama-server for the tool list shape, and record the decision in the handler test.
Activity