Skip to content

skipValidation downloads the entire audio file into memory before falling back to chunking #51

Open

Description

@rianvdm

Summary

When skipValidation is set, transcribeAudio() stubs contentLength to 0. requiresChunking(0) is then false, so the code takes the "small file" branch and pulls the whole file into memory with fetchAudio() before a size check bounces it into chunked transcription. For a 90 MB episode against a Worker's 128 MB memory limit that is uncomfortably close, and the full download is wasted work — chunking immediately re-fetches the same bytes via Range requests.

Where

src/services/transcription.ts, in transcribeAudio():

if (opts.skipValidation) {
    // Caller has already resolved the URL — skip validation and redirect resolution
    validation = { contentLength: 0, contentType: "audio/mpeg" };
}

then:

// Step 3: Route based on file size
if (requiresChunking(validation.contentLength)) {   // false when contentLength === 0
    ...
}

// Small file - fetch and transcribe directly
const audioBuffer = await fetchAudio(audioUrl);      // <-- downloads all 90 MB

// Double-check actual size (in case Content-Length was inaccurate)
if (audioBuffer.byteLength > MAX_AUDIO_SIZE_BYTES) {
    // falls back to chunking, re-fetching the same bytes by range
}

How it is reached

skipValidation is !!(ctx.preResolvedAudioUrl || ctx.audioUrlOverride) in src/queue/consumer.ts. Both come from the Audio URL (override) field under Advanced options on /admin/submit — the escape hatch for CDNs whose WAF blocks server-originated HEAD requests (Podbean and similar).

So today this is only reachable from a manual admin submit, which is why it has not bitten. It is a loaded gun though: that override field is exactly what you reach for when an episode is already misbehaving, and the episodes that misbehave are the large ones.

Repro

  1. Go to /admin/submit, open Advanced options.
  2. Submit any episode over ~90 MB with a direct CDN URL in the Audio URL (override) field.
  3. Watch the consumer logs — no chunked_transcription_start on the first pass, then size_mismatch_chunking after the entire file has already been buffered.

Suggested fix

Prefer a real size over the 0 stub, and only fall back to a blind download when there genuinely is no other option. Roughly, in priority order:

  1. Issue the HEAD against the resolved URL anyway and use it if it succeeds — the reason for skipValidation is that HEAD may be blocked, not that it is useless, so a successful HEAD should still win.
  2. Failing that, take the size from a Range: bytes=0-0 probe's Content-Range header, which many WAF-fronted CDNs will answer even when they reject a bare HEAD.
  3. Failing that, treat unknown size as "assume chunking" rather than "assume small". Chunking already handles a short final chunk, so guessing large is the safe direction — guessing small is the one that can OOM.

Worth keeping skipValidation's original intent intact: skip the content-type gate that was rejecting these CDNs, not skip knowing how big the file is.

Context

Found while fixing the audio-CDN rate-limit failures in aa4935c. Not caused by that change — this predates it and is on a different code path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions