transformers_adapter: VLM image generate fails — missing chat-template / image-placeholder tokens

[AlanKharebov]

Summary

TransformersAdapter.generate() skips the chat-template / image-placeholder injection on the VLM (processor) path. Any caller that sends { prompt, images: [...] } to a Qwen3-VL head receives ValueError: Image features and image tokens do not match and the circuit breaker trips after ~24 retries.

Verified on 02dd56d with qwen3-vl-8b (Qwen/Qwen3-VL-8B-Instruct) at 4-bit on an RTX 4080. Same code path is hit by every VLM head loaded through the transformers adapter, so vision-vlm (32B) is almost certainly affected too.

Reproduction

1. Wake qwen3-vl-8b: POST /heads/qwen3-vl-8b/wake → state becomes active.
2. Call generate with an image:

   POST /heads/qwen3-vl-8b/generate
   { "prompt": "Describe this image.", "images": ["<base64>"] }

3. Every call fails with:

   ValueError: Image features and image tokens do not match: tokens: 0, features 1333
     File ".../transformers/models/qwen3_vl/modeling_qwen3_vl.py", line 1093,
       in get_placeholder_mask
   

4. After ~24 failures the circuit breaker opens; head stays in state: error until a manual re-wake.

Root cause

src/multihead/adapters/transformers_adapter.py already calls apply_chat_template for the LLM (tokenizer) path, but the VLM (processor) branch passes the raw user prompt straight to the processor:

elif self._processor is not None:
    images = kwargs.get("images")
    if images:
        images = self._decode_images(images)
        inputs = self._processor(text=prompt, images=images, return_tensors="pt").to(self._model.device)
    else:
        inputs = self._processor(text=prompt, return_tensors="pt").to(self._model.device)

Qwen3-VL's processor needs <|vision_start|><|image_pad|><|vision_end|> placeholder tokens inside the text so the image features (1333 patches in the failing case) can bind. Without them, the forward pass detects zero placeholder tokens vs N image features and raises.

Proposed fix

Apply the chat template in the if images: branch — same shape the LLM path uses — and fall back to the raw prompt if the processor doesn't implement apply_chat_template:

if images:
    images = self._decode_images(images)
    try:
        messages = [{
            "role": "user",
            "content": [{"type": "image"} for _ in images] + [{"type": "text", "text": prompt}],
        }]
        templated = self._processor.apply_chat_template(
            messages, add_generation_prompt=True, tokenize=False,
        )
    except Exception:
        templated = prompt
    inputs = self._processor(text=templated, images=images, return_tensors="pt").to(self._model.device)

Why this can't be fixed caller-side

The HTTP API contract is { prompt: str, images?: [base64] }. Callers (Cortex UI, H2V.AI preflight, H2V.UI stage1) have no model-family awareness and shouldn't — the adapter is the boundary that owns model-specific input formatting and already does so for LLMs. Pushing the placeholder responsibility upward would break the abstraction for every consumer.

Cross-check needed

The fix is the same shape for vision-vlm (Qwen3-VL-32B-Thinking) — same processor family — but I can't validate the 32B path on a 16 GB 4080. Worth running on a 24 GB+ machine before merge.

Local mitigation

A PR with this fix follows.

[AlanKharebov]

Closing — after checking Axsar/multihead-dev (the active development repo), this code path is already fixed there: transformers_adapter.py on multihead-dev/main builds a structured content_blocks list with {type: image, image: img} entries before calling apply_chat_template on the processor (lines ~200-229 on dev). The bug only exists on Axsar/multihead because that repo is ~592 commits behind multihead-dev. Filed in error against a stale mirror.