Repository navigation
Description
Summary
When skipValidation is set, transcribeAudio() stubs contentLength to 0. requiresChunking(0) is then false, so the code takes the "small file" branch and pulls the whole file into memory with fetchAudio() before a size check bounces it into chunked transcription. For a 90 MB episode against a Worker's 128 MB memory limit that is uncomfortably close, and the full download is wasted work — chunking immediately re-fetches the same bytes via Range requests.
Where
src/services/transcription.ts, in transcribeAudio():
if (opts.skipValidation) {
// Caller has already resolved the URL — skip validation and redirect resolution
validation = { contentLength: 0, contentType: "audio/mpeg" };
}then:
// Step 3: Route based on file size
if (requiresChunking(validation.contentLength)) { // false when contentLength === 0
...
}
// Small file - fetch and transcribe directly
const audioBuffer = await fetchAudio(audioUrl); // <-- downloads all 90 MB
// Double-check actual size (in case Content-Length was inaccurate)
if (audioBuffer.byteLength > MAX_AUDIO_SIZE_BYTES) {
// falls back to chunking, re-fetching the same bytes by range
}How it is reached
skipValidation is !!(ctx.preResolvedAudioUrl || ctx.audioUrlOverride) in src/queue/consumer.ts. Both come from the Audio URL (override) field under Advanced options on /admin/submit — the escape hatch for CDNs whose WAF blocks server-originated HEAD requests (Podbean and similar).
So today this is only reachable from a manual admin submit, which is why it has not bitten. It is a loaded gun though: that override field is exactly what you reach for when an episode is already misbehaving, and the episodes that misbehave are the large ones.
Repro
- Go to
/admin/submit, open Advanced options. - Submit any episode over ~90 MB with a direct CDN URL in the Audio URL (override) field.
- Watch the consumer logs — no
chunked_transcription_starton the first pass, thensize_mismatch_chunkingafter the entire file has already been buffered.
Suggested fix
Prefer a real size over the 0 stub, and only fall back to a blind download when there genuinely is no other option. Roughly, in priority order:
- Issue the HEAD against the resolved URL anyway and use it if it succeeds — the reason for
skipValidationis that HEAD may be blocked, not that it is useless, so a successful HEAD should still win. - Failing that, take the size from a
Range: bytes=0-0probe'sContent-Rangeheader, which many WAF-fronted CDNs will answer even when they reject a bare HEAD. - Failing that, treat unknown size as "assume chunking" rather than "assume small". Chunking already handles a short final chunk, so guessing large is the safe direction — guessing small is the one that can OOM.
Worth keeping skipValidation's original intent intact: skip the content-type gate that was rejecting these CDNs, not skip knowing how big the file is.
Context
Found while fixing the audio-CDN rate-limit failures in aa4935c. Not caused by that change — this predates it and is on a different code path.
Activity