fix qwen tool parser anyOf object args

[lnigam]

Overview

Fixing parser bug where llama-server returns object-typed values of a tool parameter whose schema uses anyOf as a stringified JSON object instead of a nested JSON object. The tool is selected correctly, and the parameter names are correct — only the serialization of the value is wrong, so a caller that expects a structured object receives a string it must parse a second time.

Additional information

record 0 -- "anyOf object variant", tool set_location, param location
GOT {"location": "{\"city\": \"Paris\", \"country\": \"France\"}"} <- value is a STRING
EXPECTED {"location": {"city": "Paris", "country": "France"}}

record 1 -- "anyOf date variant", tool record_event, param when
GOT {"when": "{\"year\": 2026, \"month\": 3, \"day\": 15}", "title": "Dentist appointment"}
EXPECTED {"when": {"year": 2026, "month": 3, "day": 15}, "title": "Dentist appointment"}

record 2 -- "anyOf numeric / string", tool set_filter, param value
GOT {"field": "price", "value": "250"} <- scalar alternative, PASSES

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: Yes, code review

[ggml-gh-bot]

Hi @lnigam, thanks for your contribution!

Per our contribution guidelines, the automated PR checker found the following issue(s) that need your attention:

• Multiple open PRs from a new contributor: We limit new contributors (those without a previously merged PR) to 1 open PR at a time. You currently have 3 open PRs.

• AI-generated content: While code is allowed to be generated by AI, please write the PR description and commit messages on your own without the help of AI.

---

Please note that maintainers reserve the right to make final decisions on PRs. If you believe there is a mistake, please comment below.

[aldehir]

The choice doesn't work, because the model can freely generate an invalid object after we already sent the tool call argument deltas, causing it to fallback to a string and produce a completely different argument value that invalidates all prior deltas. This means you cannot do this and simultaneously support streaming arguments.

Provide a stricter schema in your client, as Qwen 3 syntax has no way to disambiguate. Otherwise, string is valid per the schema and what you get thanks to the model's limitation.

[aldehir]

I apologize for closing this prematurely, but without a solution for the ambiguity it is simply too risky to accept this.

It has already been tried in the past and it breaks when the model begins to generate what appears to be a valid JSON type but deviates and fails the entire request. Other models like Gemma 4 (unique string delimiters) and DeepSeek (type defined in the parameter tag) can handle this gracefully, Qwen cannot.

[ORippler]

This means you cannot do this and simultaneously support streaming arguments.
It has already been tried in the past and it breaks when the model begins to generate what appears to be a valid JSON type but deviates and fails the entire request.

Wouldn't a solution then be to adjust llama-server's buffering behavior when streaming is set? I.e. to buffer beginning JSON types and return only after validation has completed?

[pwilkin]

Wouldn't a solution then be to adjust llama-server's buffering behavior when streaming is set? I.e. to buffer beginning JSON types and return only after validation has completed?

The PEG parser already has a solution for that and it's called atomic(). However, wrapping the entire tool call in atomic() means that in practice there's no streaming of parameters since the entire parameter has to be parsed whole.

[aldehir]

I have a solution, I just haven't opened up a PR. Fundamentally it is as @pwilkin said, rooted in atomic() to suppress streaming deltas until there is enough information.

Any arguments that use a union type string | object or string | array will suppress streaming until the most constraining type is accepted/rejected. This means if it generates an object/array, the entire argument is never streamed until complete. But others, like string | null, resolve much faster.

I'll open the PR soon.

[aldehir]

This issue should be resolved in #28742.