chat: add new template for DeepSeek V4 Flash 0731

[tarruda]

Overview

Fix the existing dsv4 preview template to match the reference encoding and add new template for the 0731 which uses different prompts for reasoning levels.

Additional information

• Fix the existing dsv4 preview template to include handling for max effort reasoning, structured output and the default value of drop_thinking to True (which is the default behavior in deepseek encoding).
• Add new template for 0731, which only changes the prompts used for triggering max effort reasoning and now has an explicit "high" setting.
• Also added tests

References for these changes:

• https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/blob/main/encoding/encoding_dsv4.py
• https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes I used codex to assist, but I reviewed the changes and the official encoding files. Also have been testing these changes for weeks in the preview dsv4 and started testing the new 0731.

cc @pwilkin @aldehir @am17an

[tarruda]

this makes sense to me, but I can revert if not desired

[tarruda]

It seems like the "max effort" reasoning in DSv4 0731 is much more verbose than in the preview version. I also noticed an interesting pattern: DSv4 now has "caveman reasoning" similar to Nex N2:

[ionizing-plasma]

It seems like the "max effort" reasoning in DSv4 0731 is much more verbose than in the preview version. I also noticed an interesting pattern: DSv4 now has "caveman reasoning" similar to Nex N2:

I thought maybe it was how you worded the input and tested the same prompt, and did not get back caveman reasoning traces (this is with the unsloth IQ4_XS):

[tarruda]

I thought maybe it was how you worded the input and tested the same prompt, and did not get back caveman reasoning traces (this is with the unsloth IQ4_XS):

Try my quants: https://huggingface.co/tarruda/DeepSeek-V4-Flash-0731-GGUF

And compare with the official deepseek API

[tarruda]

@ionizing-plasma TBH this thinking style doesn't seem to be 100% consistent, a few times I got standard short reasoning even though I've set the reasoning_effort = max (but this also happens in the official API)

[tarruda]

@ionizing-plasma it seems to be possible to steer the model into that way of thinking by customizing the template. I've created a version that adds We need answer right after <think> in high/max reasoning: https://gist.github.com/tarruda/d60797a81277d781f5e70209e007cc34

Clearly that is not part of official encoding so won't be adding to this PR, but I saw great performance out of Nex N2 when using that thinking style, so I will be playing with it locally.

[ngxson]

the conversion script automatically pick up the original template, not the new 0731, would you mind having a look to see the conversion need to be updated too? (i.e. switch the file based on input model name)

[tarruda]

Since the model architecture is the same, should I just use "0731" being present in the model name?

[tarruda]

f841b4e

[coder543]

Has anyone else noticed that DSv4 Flash has to reprocess its output for every subsequent agentic turn? If it thinks for 5000 tokens then calls a tool, the next turn will also have 5000 additional tokens it needs to process on prefill.

I found a way to fix this by modifying the chat template. This diff is against the chat template that Unsloth includes with their GGUFs:

215c215,220
<       {{- '\n\n<' + dsml_token + 'tool_calls>\n' -}}
---
>       {#- llama.cpp preserves the DSML separator in parsed content; do not add it twice. -#}
>       {%- set content_has_separator = message['content'] is defined and message['content'] is string and message['content'].endswith('\n\n') -%}
>       {%- if not content_has_separator -%}
>         {{- '\n\n' -}}
>       {%- endif -%}
>       {{- '<' + dsml_token + 'tool_calls>\n' -}}

The issue was an extra \n\n that would get rendered during agentic turns, causing large amounts of wasted additional prefill in my testing. This modification to the template avoids duplicating the \n\n separator.

I believe the template in this PR will suffer the same reprocessing issue unless the fix is applied. (Someone could also fix the parser instead of the template, perhaps? I'm not sure what the best approach is here.)

[tarruda]

Has anyone else noticed that DSv4 Flash has to reprocess its output for every subsequent agentic turn?

That does not happen with this template

[coder543]

Ok... so now I've spent 30 minutes to reproduce it with this PR and show that it does happen with this PR and this exact template.

repro.sh
deepseek-v4-flash-0731-chat-template.txt

(I had to rename it to a .txt extension to attach it to this comment)

Running against a llama-server binary that is built from this PR using this PR's template:

$ export SERVER_BIN=./llama-server
$ export MODEL=./deepseek-v4-flash-0731-ud-q2_k_xl.gguf
$ export TEMPLATE=./deepseek-ai-DeepSeek-V4-Flash-0731.jinja
$ ./repro.sh
Starting llama-server...
Turn 1: generating a tool call...
{
  "finish_reason": "tool_calls",
  "assistant_content": "\n\n",
  "prompt_tokens": 343,
  "completion_tokens": 241
}
Turn 2: returning the tool result...
{
  "reproduced": true,
  "first_completion_tokens": 241,
  "second_prompt_tokens": 606,
  "expected_cached_history_tokens": 583,
  "actual_cached_tokens": 339,
  "avoidably_reprocessed_history_tokens": 244,
  "genuinely_new_turn_tokens": 23,
  "total_reprocessed_tokens": 267,
  "prompt_ms": 972.062,
  "response": "OK"
}

So, it wasted the processing time to reprocess 244 tokens on the second turn.

Running with the template I provided above:

$ export TEMPLATE=./deepseek-v4-flash-0731-chat-template.jinja
$ ./repro.sh
Starting llama-server...
Turn 1: generating a tool call...
{
  "finish_reason": "tool_calls",
  "assistant_content": "\n\n",
  "prompt_tokens": 343,
  "completion_tokens": 241
}
Turn 2: returning the tool result...
{
  "reproduced": false,
  "first_completion_tokens": 241,
  "second_prompt_tokens": 606,
  "expected_cached_history_tokens": 583,
  "actual_cached_tokens": 583,
  "avoidably_reprocessed_history_tokens": 0,
  "genuinely_new_turn_tokens": 23,
  "total_reprocessed_tokens": 23,
  "prompt_ms": 288.51,
  "response": "OK"
}

No wasted tokens.

There is an issue with the template in this PR.

[coder543]

Fundamentally, the issue is that the model seems to be outputting two newlines which the parser isn't catching, which is what we see in assistant_content in the repro, then the template reinserts those when the turn is rendered the next time around. This causes four newlines, and invalidates the cache from that point on. Someone could fix the parser, or someone make the template more robust. I don't know if DSv4 is using the autoparser or a custom parser. I fixed the template on my machine to avoid the issue, but I'm not sure what the preferred fix is here.

[tarruda]

@coder543 dsv4 is using a manual parser, I'm gonna try to fix that. Thanks for catching.

[tarruda]

@coder543 a8aa9d4

[aldehir]

Parser edits are fine, but it seems this could also be handled with | trim, which is common across templates.

[aldehir]

@coder543 can you verify the parser updates address the cache issue.

[coder543]

@aldehir yes, the round trip rendering error is fixed now!

[tarruda]

@coder543 thanks for validating it!

[ggerganov]

When we merge this, how is the new chat template going to be propagated to the GGUFs?

[tarruda]

@ggerganov it will have to be re-embedded into existing GGUFs (gguf-new-metadata). New GGUFs should pick it up automatically.

Note that this is not a critical template fix, main things it enables is supporting reasoning levels and adding the prompt DSv4 was trained to emit structured output.

But the later bugfix caught by @coder543 will actually benefit existing GGUFs.

[ggerganov]

New GGUFs should pick it up automatically.

I'm not sure how that works. I thought when converting the GGUFs with convert_hf_to_gguf.py it picks up the template from the source transformers repo?

[tarruda]

it picks up the template from the source transformers repo?

Deepseek v4 doesn't have a jinja template. They use a specialized parsing/encoding python module: https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731/blob/main/encoding/encoding_dsv4.py.

Note very familiar with convert_hf_to_gguf.py, but I imagine it picks up a template from llama.cpp repo if the official repo doesn't have a template.

[tarruda]

I already tested this by converting safetensors with this branch and confirmed the new template is picked up.

[ggerganov]

Ok, I see it now. Thanks

[tarruda]

@coder543 it seems the prompt reprocessing issue is still there, I saw this running locally:

That is: the model generated close to 60k tokens of thinking, then it did a tool call. When processing the tool call, it had to reprocess the entire prompt.

Going to see if I can reproduce consistently and come up with a fix.

[ggerganov]

@tarruda Which client do you used?

Btw, you should set these envs for some extra logs:

export LLAMA_TRACE=1
export LLAMA_SERVER_SLOTS_DEBUG=1

[tarruda]

I'm actually using my own client, so it could be a bug on it too: https://github.com/tarruda/neoagent

I will turn those llama traces on, but will also enable request logging on my client so I can compare individual requests to see if it changes something in the payload prefix.

[coder543]

Interesting. I had run my repro.sh script on the latest version of this branch, and it reported no issue anymore. On my DGX Spark, I get better performance using a fork of ds4, so that’s what I’ve been playing with the past few days, so I haven’t tested in practice whether the issue is still recurring on llama-server.

[tarruda]

The issue was pretty consistent across all clients I tested (mine, codex and pi) and it always happened in streamed thinking blocks. I believe the main reason it was not noticed before is that most times it doesn't think long enough before a tool call for the extra time to be noticeable. This time it reasoned for 60k tokens, and given the slow promt processing on my mac, it was easy to spot.

Apparently it was a tokenization bug. I've pushed a fix to my branch: 7018277

Still not sure because I used deepseek v4 flash itself to assist in debugging this and it provided the fix plus this summary:

## Summary

With the DeepSeek V4 Flash 0731 template, a coding harness can observe the
following behavior:

1. The model generates a large `<think>` block and then a tool call.
2. The harness executes the tool and sends the result back.
3. llama-server reprocesses the thinking block on the second turn instead of
   reusing the KV cache prefix.

The root cause is a tokenization mismatch, not a template or parser bug:

- During turn 1, the model samples token `6883` (`Ġ~`, the BPE encoding of
  ` ~`) inside the thinking block, and that token ID is appended to the slot
  context.
- On turn 2, llama.cpp re-tokenizes the same text and produces `223` (`Ġ`) +
  `96` (`~`) instead of `6883`.
- KV cache reuse compares token IDs, so the common prefix stops at the first
  divergent token and everything after it (the rest of the thinking block) is
  reprocessed.

The patch looks like it fixed for me. Will use for a week to see if there are any side effects and can create a PR if that looks right for @ggerganov.