[webgpu] Whisper encoder fp16 precision issues

[jozefchutka]

System Info

transformers.js v4

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

When using the automatic-speech-recognition pipeline with the model onnx-community/whisper-large-v3-turbo_timestamped and return_timestamps: true, version 4.0.0-next.7 (and starting from at least 4.0.0-next.1) outputs a single segment covering the full audio duration (and missing the expected transcripts), while 3.8.1 correctly produces multiple shorter sentence/phrase-level segments with accurate boundaries.

This appears to be a regression in the v4 preview series, as the 3.x behavior better preserves the natural sentence/phrase structure of the audio.

Reproduction

1. Download attached audio file ( ch.zip ) and extract. Use the exact same pipeline setup in both versions:

<script type="module">
import { pipeline } from "https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.8.1";
//import { pipeline } from "https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.0.0-next.7";
const audio = new Float32Array(await (await fetch("ch.pcm")).arrayBuffer());
const pipe = await pipeline("automatic-speech-recognition",
	"onnx-community/whisper-large-v3-turbo_timestamped",
	{dtype:{encoder_model:"fp16", decoder_model_merged:"fp16"},
	device:"webgpu"});

const result = await pipe(audio, {
	return_timestamps: true});

console.log(result.chunks.map(chunk => `${chunk.timestamp[0]} -> ${chunk.timestamp[1]} ${chunk.text}`))
</script>

2. v3.8.1 output in console log:

"12 -> 16  Some stay dry and others feel the pain"
"16 -> 17  Chocolate Rain"
"17 -> 21  A baby born will die before the sin"
"21 -> 22  Chocolate Rain"
"22 -> 26  The school books say it can't be here again"
"26 -> 28  Chocolate Rain"

v4 output:

"0 -> 29.98  Chocolate Rain"

I tried various values for generation parameters like changing chunk_length_s (e.g., to 10, 15, 20), adjusting stride_length_s, and experimenting with other options such as num_beams or temperature, but none of these changes improved the output in v4 - the result remained a single merged segment regardless of the settings.

I expect the output to retain sentence/phrase-level segmentation similar to 3.8.1 (multiple segments with timestamps aligned to lyrical/phrasal breaks), rather than aggressively merging everything into one incomplete chunk. The model itself is capable of predicting distinct segments, and previous versions respected that.

This might be related to:

• whisper-large-v3-turbo_timestamped has broken timestamps #1357
• whisper-base_timestamped broken with chunk_length_s=30 #1358

[xenova]

When I test in python, I get pretty poor results:

from transformers import pipeline
import numpy as np

pipe = pipeline("automatic-speech-recognition", "openai/whisper-large-v3-turbo")

audio = np.fromfile("ch.pcm", dtype=np.float32)
print(audio.shape)
result = pipe(audio)
print(result)

setting return_timestamps='word' does seem to produce better results.

{'text': " Chocolate Rain Some stay dry and others feel the pain Chocolate Rain A baby born will die before the sin Chocolate Rain The school books say it can't be here again Chocolate Rain The prisons", 'chunks': [{'text': ' Chocolate', 'timestamp': (0.0, 10.98)}, {'text': ' Rain', 'timestamp': (10.98, 11.72)}, {'text': ' Some', 'timestamp': (13.06, 13.52)}, {'text': ' stay', 'timestamp': (13.52, 13.9)}, {'text': ' dry', 'timestamp': (13.9, 14.28)}, {'text': ' and', 'timestamp': (14.28, 14.56)}, {'text': ' others', 'timestamp': (14.56, 15.08)}, {'text': ' feel', 'timestamp': (15.08, 15.46)}, {'text': ' the', 'timestamp': (15.46, 15.68)}, {'text': ' pain', 'timestamp': (15.68, 15.96)}, {'text': ' Chocolate', 'timestamp': (16.1, 16.44)}, {'text': ' Rain', 'timestamp': (16.44, 17.38)}, {'text': ' A', 'timestamp': (18.42, 18.64)}, {'text': ' baby', 'timestamp': (18.64, 19.1)}, {'text': ' born', 'timestamp': (19.1, 19.66)}, {'text': ' will', 'timestamp': (19.66, 19.88)}, {'text': ' die', 'timestamp': (19.88, 20.14)}, {'text': ' before', 'timestamp': (20.14, 20.72)}, {'text': ' the', 'timestamp': (20.72, 21.0)}, {'text': ' sin', 'timestamp': (21.0, 21.2)}, {'text': ' Chocolate', 'timestamp': (21.36, 21.72)}, {'text': ' Rain', 'timestamp': (21.72, 22.48)}, {'text': ' The', 'timestamp': (23.68, 24.02)}, {'text': ' school', 'timestamp': (24.02, 24.28)}, {'text': ' books', 'timestamp': (24.28, 24.6)}, {'text': ' say', 'timestamp': (24.6, 24.94)}, {'text': ' it', 'timestamp': (24.94, 25.16)}, {'text': " can't", 'timestamp': (25.16, 25.52)}, {'text': ' be', 'timestamp': (25.52, 25.72)}, {'text': ' here', 'timestamp': (25.72, 26.14)}, {'text': ' again', 'timestamp': (26.14, 26.5)}, {'text': ' Chocolate', 'timestamp': (26.7, 27.06)}, {'text': ' Rain', 'timestamp': (27.06, 27.84)}, {'text': ' The', 'timestamp': (28.0, 29.38)}, {'text': ' prisons', 'timestamp': (29.38, 29.88)}]}

[xenova]

changing model id to "openai/whisper-tiny.en" seems to work:

{'text': ' Chocolate rain. Some stay dry and others feel the paint chocolate rain. A baby born will die before the same chocolate rain. The schoolbooks say it can be here again chocolate rain.'}

[xenova]

Switching to

dtype: { encoder_model: "fp32", decoder_model_merged: "fp16" },

does give much better results. Possibly an issue with fp16.

[jozefchutka]

I switched encoder model to fp32, but got transformers@4.0.0-next.7:25 Uncaught (in promise) TypeError: Failed to fetch when downloaded around ~2.5GB. I have working internet connection but this happened twice now.

fp32 being 2.5GB seems just too much to my taste, I would rather stick to fp16 which is half the size and will perform faster.

I am trying to understand how this is fp16 model issue and not javascript one. Considering tjs 3.8.1 and v4 uses the same model while both provide different results. That seems to me more like implementation difference b/c model is the very same and inputs provides deterministic outputs.

[jozefchutka]

Can you please reopen or revisit considering comments in #1594 (comment)

[xenova]

Oh right -- this was closed as it was linked from #1594. My main guess is some precision loss in the new webgpu EP. will investigate.