fix: Loop detection truncates schema-valid output when a json_schema array holds 12 or more identical values #977

Follow-up to #967 / PR #969.

Problem / Background

PR #969 raised the Gemma 4 loop-detection threshold from min_count=4 to min_count=12 and gated the family default-on so that only requests carrying an amplifier arm the detector. That shrank the false-positive class but did not eliminate it on the one surface that is still armed by default: grammar-constrained decoding.

A json_schema request whose schema legitimately permits a long run of identical values is still cut mid-document, and the client sees an ordinary stop.

Verified reproduction

Model models/gemma-4-12b-it-4bit (model_type: gemma4_unified), served by a CUDA build of mlxcel-server, no MLXCEL_LOOP_DETECTION set, plain POST /v1/chat/completions.

Request:

• response_format: {"type":"json_schema","json_schema":{"name":"vals","schema":{"type":"object","properties":{"values":{"type":"array","items":{"type":"integer"}}},"required":["values"],"additionalProperties":false}}}
• temperature: 0, seed: 42, max_tokens: 400
• Prompt: Return the values array containing the number 0 repeated 30 times.

Response content, in full:

{
  "values": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

Cut after exactly 12 values. completion_tokens: 43, finish_reason: "stop". Server log:

loop detection: ending generation early (repetition loop) generated=43

Why it matters

The returned document is truncated mid-array. It is not valid JSON, and it violates the very schema the client constrained the request to, while the client sees an ordinary stop with no error. A structured-output caller that trusts finish_reason will hand malformed JSON to its parser.

Why PR #969 did not fix it

After #969 the json_schema grammar constraint is one of the amplifiers that arms the detector, so structured-output requests are exactly the traffic that still runs with detection on. That surface is also the one most likely to emit legitimately repetitive token sequences: a uniform array, repeated structural separators, enum values repeated across records. Any fixed min_count is exceeded by a longer legitimate run, so raising the constant again only moves the boundary.

This is not a regression from #969. At the old min_count=4 the same request was cut after 4 values, so #969 improved the behavior; it just did not remove the class.

Current recourse for a client

Send max_pattern_size: 0 on the request to disable detection for that call. The per-request override sits above the gate in resolve_loop_detection, so it is not subject to the family default-on. This is a workaround, not a fix, since it requires the client to already know about the problem.

Proposed Solution

This needs a design decision, so the options are listed neutrally rather than pre-selected.

1. Drop the grammar signal as an amplifier. Leave tool-shaped prompts as the only condition that arms the detector. This contradicts fix(server): Gemma 4 default-on loop detection falsely truncates markdown tables; raise min_count and gate activation to tool/grammar-constrained requests #967's Part 2 as filed and would need agreement before implementing.
2. Suppress detection while the structured-output matcher is actively masking, on the reasoning that the grammar already bounds the output shape.
3. Make the detector grammar-aware, so a repetition that the schema forces or permits does not count as a loop.
4. Accept and document. This is today's state.

Acceptance Criteria

• A direction is chosen from the options above (or a fifth), with the rationale recorded on this issue.
• The reproduction request above returns a complete, schema-conformant values array of 30 zeros, or the truncation is surfaced to the client as something other than a plain finish_reason: "stop".
• Whatever is chosen preserves the fix(server): Gemma 4 default-on loop detection falsely truncates markdown tables; raise min_count and gate activation to tool/grammar-constrained requests #967 protection: a genuine Gemma 4 repetition collapse on a tool-shaped request still terminates early.
• A regression test covers the grammar-constrained long-legitimate-run case.
• docs/supported-models.md is updated to match the resulting behavior.

Technical Considerations

Line numbers are on main at 053443d4.

• src/server/request_options.rs:61 defines LOOP_DETECTION_RECOMMENDED. Its doc comment already records this limitation and notes that a retune of the constant cannot fix it.
• src/server/request_options.rs:224 resolve_loop_detection and src/server/request_options.rs:170 chat_carries_loop_amplifier hold the gate.
• src/lib/mlxcel-core/src/loop_detection.rs:145 detect_repetition_loop is the detector itself.
• docs/supported-models.md documents this limitation, including the reproduction above.