feat: Detect refusal-style VLM image generations in bench_mlxlm.py #2134

Follow-up to #1683 (the generation-content check its Scope section puts out of scope). Root cause of #1683 is fixed in PR #2131.

Problem / Background

In #1683, granite-vision-3.2-2b-4bit answered descriptive image prompts with refusals such as "I'm sorry, but I can't provide a description of the image as I can't see it." and "I'm sorry, but I can't see any images. I can only provide text-based responses." while prompt_tokens stayed a healthy 1545 (1485 image tokens inserted). The image tokens were present but mispositioned: <image> was declared only in tokenizer_config.json, the tokenizer split it, and the image block was spliced after the first token (fixed in PR #2131). No length-based gate can see this class, and docs/benchmark_results/model_tests_m1ultra.md:242 already records the gap.

Current Behavior

• scripts/bench_mlxlm.py:489 (inside bench_one, :441) returns FAIL:image_not_applied only when prompt_tokens <= text_only_prompt_tokens, that is, when the image never expanded the prompt.
• The VLM child (SUBPROCESS_VLM_CODE, :302) keeps only the last GenerationResult (last = r, :381-384), so the generated text is never captured, and the RESULT dict (:393-399) carries counts and rates only.
• EOS is suppressed (:346-370), so output past the first answer can be filler; the refusal appears at the start.
• VLM_PROMPT (:28) is the only image prompt: "What is in this image?".

Proposed Solution

1. In SUBPROCESS_VLM_CODE, accumulate r.text across the measured stream_generate loop into one string and add it to the RESULT dict as generated_text. The warmup loop is unchanged.
2. Add a module-level, side-effect-free function vlm_refusal_reason(text: str) -> str | None to scripts/bench_mlxlm.py. It lowercases the text, normalizes the typographic apostrophe ’ to ', and returns the first matching phrase from a module-level tuple VLM_REFUSAL_PHRASES = ("can't see", "cannot see", "unable to view", "don't have the ability to view", "can't provide a description"), or None. It matches anywhere in the text.
3. In bench_one, for vlm runs only and only after the image_not_applied check passed, call vlm_refusal_reason(data.get("generated_text") or ""); on a match return ("FAIL:image_ignored", None) and print the matched phrase to stderr so the sweep log shows why. Text runs are never checked.
4. Document FAIL:image_ignored in the comment block next to the image_not_applied check, and update docs/benchmark_results/model_tests_m1ultra.md:242 to say the sweep now flags this class.

Rejected: a second descriptive prompt alongside VLM_PROMPT. It doubles VLM sweep time and changes the CSV row shape, and the phrase check on the existing prompt already catches the #1683 output. Do not add it.

Scope

In scope: scripts/bench_mlxlm.py, a new tests/test_bench_mlxlm_refusal.py, the docs line above.

Out of scope: the mlxcel-side sweep (scripts/bench_decode.sh driving src/bin/bench_decode.rs). The #1683 refusal came from mlxcel, but bench_decode.rs never prints generated text, so the same check there needs a Rust change and a separate issue, which should reuse VLM_REFUSAL_PHRASES rather than copy the list.

Implementation Notes

• Reuse: keep the classifier next to the existing image_not_applied logic; no new module.
• Edge cases: missing or empty generated_text (zero tokens) means no match, so the status stays OK; matching is case-insensitive.
• Constraints: the test imports bench_mlxlm with scripts/ on sys.path (it already inserts its own directory for vlm_detect); mlx imports live only inside the child code strings, so the import works without mlx installed. Keep it that way.

Acceptance Criteria

• vlm_refusal_reason returns a phrase for both fix(granite_vision): descriptive prompts draw a refusal, colour questions work #1683 strings verbatim, and for a variant using ’.
• It returns None for a normal description, for example "The image shows a solid orange square on a white background."
• A VLM run whose output matches is recorded as FAIL:image_ignored in the CSV row and on stderr, distinct from FAIL:image_not_applied.
• tests/test_bench_mlxlm_refusal.py (stdlib unittest, same style as tests/test_bench_decode_oom_classifier.py) covers the cases above.

Verification

python3 -m unittest tests/test_bench_mlxlm_refusal.py
python3 scripts/bench_mlxlm.py --help

Manual: one bench_mlxlm.py VLM run on a working checkpoint still records OK.