mtmd: one-sample audio input causes process abort

[hera2019]

Name and Version

version: 0.2.0-dev (build 10618, commit eb25b72)
built with AppleClang 21.0.0.21000101 for Darwin arm64

Operating systems

Mac

Which llama.cpp modules do you know to be affected?

libllama (core library), llama-cli, llama-server

Command line

llama-server -hf ggml-org/Voxtral-Mini-3B-2507-GGUF --port 18080 -c 4096
# Then send a request to /v1/chat/completions with a single-sample WAV file provided as input_audio.

Problem description & steps to reproduce

When a WAV file containing is exactly one sample, the process aborts at mtmd.cpp:1553 GGML_ASSERT(buf.size() > sizeof(float)). Because one float occupies 4 bytes, the buffer size is also 4 bytes, the condition evaluates to false (4 > 4 is false).
I verified the crash on Voxtral; the assertion sits on the shared path before the per-model preprocessor is selected, so it should affect all audio models.
The Whisper preprocessor is designed to zero-pad very short audio clips. Therefore, regardless of whether such extremely short input should ultimately be accepted or result in an error, a single request should not trigger an assertion that kills the entire server process. The erroneous assertion causes the entire server process to abort instead of returning an error for the request.

Steps to reproduce:

1. Build any recent master with audio-capable mtmd.
2. Create a 1-sample 16 kHz mono WAV (see attached script).
3. Feed it to llama-mtmd-cli, or POST it to llama-server as input_audio.
4. CLI aborts with exit code 134; the server process dies and /health stops responding.

AI usage disclosure: I used an AI assistant to help troubleshoot this bug and draft this report. I personally reproduced and verified the crash and also understand the relevant code.

First Bad Commit

Not bisected. The assertion is long-standing and was not touched by the recent
short-input fix (#26536), which changed min_samples inside the preprocessor but
not this buffer-size guard at the tokenizer entry.

Relevant log output

Logs

tools/mtmd/mtmd.cpp:1553: GGML_ASSERT(buf.size() > sizeof(float)) failed

0   ggml_abort
1   GGML_ASSERT(buf.size() > sizeof(float))          [mtmd.cpp:1553]
2   handle_completions_impl                          [server]
3   httplib::Server::process_request
4   httplib::ThreadPool::worker

[Sumire-no-kai]

Hi, I'd like to work with this issue. I've checked the code in mtmd.cpp where GGML_ASSERT(buf.size() > sizeof(float)) causes abort() on 1-sample inputs. I am setting up a local repro with a 1-sample WAV and will submit a PR with graceful error handling (nx < 2) and a regression test once verified.
I'll confirm whether that could slove it.

[hera2019]

@Sumire-no-kai Thanks for taking a look at this.

I had a fix prepared locally before filing the issue. I tested changing the condition from > to >=, and the 1-sample input was processed successfully on my machine. I was planning to wait for maintainer feedback on the preferred behavior before opening a PR.

One thing worth considering is whether a 1-sample input should actually be rejected. The current Whisper preprocessor appears capable of handling such extremely short input by padding it to the minimum required length. The preprocessor pads short input up to n_fft/2 + 1 samples (201 for this model). With the >= change, the 1-sample WAV reached the preprocessor and was handled normally in my test. So rejecting nx < 2 may turn an input that can currently be handled into an error.

I think the main question is whether this case should be accepted and padded, or rejected gracefully. It would be good to get maintainer guidance on which behavior is preferred.

I'm happy to open the PR since I already have a tested local fix. But if you'd prefer to work on it, I can share my reproduction script and test results.

[Sumire-no-kai]

@hera2019 Thanks for taking this on.

Happy to let you lead the implementation and PR to avoid duplicated effort.

A quick technical note: static inspection shows that different preprocessors handle ultra-short inputs differently, so these observations are worth further testing. Right now, the only dynamic evidence is your report on Voxtral/Whisper, so whether >= works safely as a global fix across all models remains to be proven by cross-model tests.

I can try to help independently reproduce and verify your fix on Windows. Could you share your script, full commands, model version, and test results or some related information when ready?

[hera2019]

@Sumire-no-kai Thanks for letting me take the lead on the implementation and PR. I'd really appreciate your help verifying it on Windows. I'm also glad to connect, and I'd be happy to collaborate again in the future.

Here are the reproduction details and test results you asked for:

Environment / repro details:

- Model: ggml-org/Voxtral-Mini-3B-2507-GGUF (default Q4_K_M + mmproj Q8_0, fetched via -hf)
- Build: version 0.2.0-dev (build 10618, commit eb25b7263), AppleClang 21.0.0, Darwin arm64, Metal
- Machine: Apple M2 Max, 32 GB, macOS 26.5.2

Repro (CLI):
  # generate a 1-sample 16 kHz mono WAV
  python3 -c "import wave; w=wave.open('1sample.wav','wb'); w.setnchannels(1); w.setsampwidth(2); w.setframerate(16000); w.writeframes(b'\x20\x00'); w.close()"

  llama-mtmd-cli -hf ggml-org/Voxtral-Mini-3B-2507-GGUF --audio 1sample.wav -p "T." -n 1 --no-warmup
  # -> mtmd.cpp:1553 GGML_ASSERT(buf.size() > sizeof(float)) failed, exit code 134

Repro (server):
  llama-server -hf ggml-org/Voxtral-Mini-3B-2507-GGUF --port 18080 -c 4096
  # POST /v1/chat/completions with the same WAV as input_audio
  # -> process aborts; /health stops responding

Results by sample count (my machine, before the fix):
  0 samples -> graceful error ("empty audio data")
  1 sample  -> abort (exit 134)
  2 samples -> OK
  3 samples -> OK

After changing `>` to `>=`: 1, 2, 3 samples all processed normally (exit 0, one encode each).

You're right that my dynamic evidence so far is limited to the Whisper-based path, specifically Voxtral. I tried to find models using other audio preprocessors for testing, but the small audio-capable models I could readily use from ggml-org, including Voxtral, Qwen2.5-Omni, and Ultravox, all appear to use the Whisper preprocessor. So I haven't yet been able to dynamically cover the qwen3a, gemma4a, or conformer paths.

My current understanding is that these two checks have different responsibilities. The guard at mtmd.cpp:1553 checks whether the input buffer is structurally large enough to contain at least one float. Whether a particular model can semantically handle such a short audio input seems to belong to the model-specific preprocess() implementation. The individual preprocessors already have their own validation logic, for example checks such as n_samples == 0 returning false. Whisper does more than simply reject invalid inputs; it features an explicit zero-padding path (min_samples = n_fft/2 + 1), indicating that accepting extremely short audio clips was part of the original design. If preprocess() returns false, the caller reaches the LOG_ERR(...) / return 2 path instead of aborting the process. So changing > to >= would at least remove this unconditional abort at the tokenizer entry and allow the model-specific preprocessor to decide whether the input can be handled. For Voxtral/Whisper, I've verified that the 1-sample input is then padded and processed successfully. For the other preprocessors, I agree that this still needs dynamic verification.

mtmd.cpp:1553  GGML_ASSERT(buf.size() > sizeof(float));     // structural check on the buffer
mtmd.cpp:1559  bool ok = ctx->audio_preproc->preprocess(...); // per-model semantic check
mtmd.cpp:1560  if (!ok) { LOG_ERR("Unable to preprocess audio\n"); return 2; }  // graceful

If you can run a model using a different preprocessor on Windows, that would be especially helpful. For example, something using the qwen3a, gemma4a, or conformer path would cover exactly the gap in my current testing. That would give us much better evidence for whether >= is appropriate as the shared-path fix.