input_video rejects data URLs: "Failed to load image or audio file" — data: URLs fall into the raw-base64 branch (accept_base64_uri=false)

[Flo5k5]

What happened

Sending a video as a standard data URL in an input_video content part fails with HTTP 400:

{"error":{"code":400,"message":"Failed to load image or audio file","type":"invalid_request_error"}}

while the same video sent as raw base64 (no data: prefix) works correctly — frames are decoded and temporally understood by the model.

Repro (verified on current master, tools/server/server-common.cpp as of 2026-08-26)

Failing request:

{
  "model": "qwen3.8",
  "max_tokens": 200,
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Describe this video."},
      {"type": "input_video", "input_video": {"data": "data:video/mp4;base64,AAAAIGZ0eXBpc29t..."}}
    ]
  }]
}

Working request (only change is the prefix):

{"type": "input_video", "input_video": {"data": "AAAAIGZ0eXBpc29t..."}}

Both against /v1/chat/completions. Images via image_url with a data:image/...;base64,... URL work fine on the same server, so clients reasonably expect data URLs to work for video too.

Root cause

The input_video (and input_audio) branches call handle_media with accept_base64_uri = false (line ~1246):

} else if (type == "input_video") {
    if (!opt.allow_video) {
        throw std::runtime_error("video input is not supported - hint: ...");
    }

    json input_video = json_value(p, "input_video", json::object());
    std::string url  = json_value(input_video, "data",
                            json_value(input_video, "url", std::string()));
    handle_media(out_files, url, opt.media_path, false);   // <-- false

In handle_media (line ~1102), with accept_base64_uri == false a data: URL skips the data-URL branch entirely and falls into the final else:

} else {
    // try as raw base64 string
    auto decoded_data = base64_decode(url);
    if (decoded_data.empty()) {
        throw std::runtime_error("Invalid base64 value");
    }
    out_files.push_back(decoded_data);
}

so the entire string — "data:video/mp4;base64,AAAA..." — gets base64-decoded as-is. The decoder is lenient enough to return garbage bytes instead of failing, and those bytes reach mtmd_helper_bitmap_init_from_buf(): not audio, not stb_image, not webp → the video probe runs ffprobe on the garbage, which (with -v quiet) prints nothing → width == 0 → probe() returns false → the generic "Failed to load image or audio file" error.

Two side observations from debugging this:

1. strace confirmed ffprobe is spawned and exits silently on the garbage input — the misleading error message sent me looking for PATH/ffmpeg problems before I found the actual decode issue.
2. Even with accept_base64_uri = true, the data-URL branch currently requires string_starts_with(parts[0], "data:image/") (line ~1107), so data:video/... would be rejected there too ("Invalid uri format").

Suggested fix

Either (or both):

1. Parity for input_video/input_audio: pass accept_base64_uri = true for these branches and relax the media-type check in handle_media to accept any data: prefix (or at least data:video/ and data:audio/).
2. Fail loudly: in the final else branch, if the string starts with "data:" (or contains ";base64,"), throw "Invalid uri format" instead of attempting a raw base64 decode of the whole URL. This turns a confusing downstream decode error into an actionable one.

Environment: 2× V100, CUDA 12.6, mmproj for Qwen3.8-27B, ffmpeg 4.4 in PATH (MTMD_VIDEO=ON, LLAMA_SUBPROCESS=ON defaults).

Related: #27587 (long-video probe deadlock) — separate failure mode, found while testing the video path.

[geckguy]

opened #27735 for this