Skip to content

Cache chunk transcripts in KV so queue retries resume transcription instead of restarting #48

Closed

Description

@rianvdm

Why

On 2026-08-02 a Lenny's Podcast episode ("This CPO regrets that product management exists", episodeId 1627920305_rss_9d799ca4a3, 84 min, 77.8MB Substack-hosted MP3) could not be processed and burned through its retry budget without producing anything. The failure chain, verified live:

  • Both gpt-transcribe and gpt-4o-mini-transcribe reject this file's chunks with HTTP 400 "Audio file might be corrupted or unsupported" — confirmed by posting the actual first 15MB chunk to both models directly (identical error body, ~1.5s each). Not a Migrate transcription from gpt-4o-mini-transcribe + whisper-1 to gpt-transcribe #46 regression; the file looks ordinary (MPEG-1 Layer III, 128kbps, 44.1kHz stereo, 269-byte ID3v2.4 tag) and whisper-1 decodes it fine.
  • So every chunk takes the whisper-1 fallback at ~90–150s per 15MB chunk (measured 92–96s in the Migrate transcription from gpt-4o-mini-transcribe + whisper-1 to gpt-transcribe #46 pre-flight), instead of gpt-transcribe's ~20s.
  • Six chunks at fallback speed plus fetches exceeds the 15-minute wall-clock limit Cloudflare puts on queue consumer invocations. The invocation was killed (exceededWallTime in the observability logs), the message redelivered, and the retry restarted transcription from chunk 1 — so it can never converge. With max_retries = 5 (wrangler.toml:39) that's up to ~75 minutes of API spend with zero output, re-paying OpenAI for the same chunks on every attempt.
  • Meanwhile the 20-minute stale-job sweep marked the still-working job failed ("Job timed out after 21 minutes") and fired a false Discord failure alert.

Contrast, same day: an Ezra Klein episode (58MB → 4 chunks, all on gpt-transcribe at 17–26s each) completed in ~5 minutes. The fast lane is fine; only the fallback lane breaks, and only on long files.

Episode-level resume already exists — processEpisode loads an existing transcript (src/queue/consumer.ts:376) and skips transcription entirely when one is saved. The gap is chunk-level: a killed attempt's completed chunks live only in memory. Fix: cache per-chunk transcripts in KV so a retry resumes from the first uncached chunk. For the Lenny case, attempt 1 caches ~5 of 6 chunks before the kill and attempt 2 finishes in a few minutes.

Current state (verified in code, 2026-08-02)

  • Chunk loop in transcribeWithChunking (src/services/transcription.ts:886): iterate (:925), Range-fetch (:949), transcribe (:978), accumulate in memory (:987), fallback-model bookkeeping (:996). Nothing is persisted until the consumer saves the full stitched transcript (src/queue/consumer.ts:553).
  • Chunk ranges are a pure function of contentLength: calculateChunkRanges (src/lib/audio.ts:150), 15MB target, 32KB overlap — so chunk identity is deterministic across attempts for the same file.
  • TranscribeOptions (src/services/transcription.ts:699) carries only apiKey/provider/onProgress/skipValidation; the service has no storage access or episode identity (deliberately pure).
  • Consumer call site (src/queue/consumer.ts:530-533): transcribeAudio(audioUrl, { apiKey, skipValidation }) — kv and episodeId are both in scope there.
  • KV key builders at src/lib/kv.ts:9 (KV_KEYS), TTLs at :26 (TTL.JOB = 1 day).
  • Stale-job sweep: src/lib/job-status-do.ts:156 compares jobAge > TIMEOUTS.JOB_MS; TIMEOUTS.JOB_MS is 20 min (src/lib/constants.ts:317). The timeout message computes minutes from the constant, so it self-adjusts.
  • Queue config: max_batch_size = 1 (wrangler.toml:37), max_retries = 5 (:39). The 15-minute consumer wall clock is a Cloudflare platform limit, not configurable.

Target

Current Target
Retry after a wall-time kill restarts from chunk 1 resumes from first uncached chunk
Chunk transcripts in-memory only cached per chunk in KV, 1-day TTL
Whisper-lane 84-min episode never converges (5 kills → failed) attempt 1 (killed at 15 min) + attempt 2 (~5 min) → completed
Fast lane (gpt-transcribe accepts) unchanged unchanged behavior + cache writes
Stale-job sweep 20 min — false-fails resumed jobs 35 min

Work

1. Cache contract (src/services/transcription.ts, near TranscribeOptions at :699):

export interface ChunkTranscriptCache {
    /** chunkKey is `${contentLength}:${startByte}-${endByte}` — chunk identity, no episode identity. */
    get(chunkKey: string): Promise<{ text: string; model: string } | null>;
    put(chunkKey: string, value: { text: string; model: string }): Promise<void>;
}

Add chunkCache?: ChunkTranscriptCache to TranscribeOptions, and thread it through both transcribeWithChunking call sites (:819, :842) into its signature (:886). Keying on contentLength means a changed file next attempt (CDN re-encode, different length) misses cleanly — including early chunks whose byte ranges would otherwise coincide.

2. Resume logic in the chunk loop (:925-996). Insert the hit check after the chunk_processing log, before the try block (so cached chunks also skip the inter-chunk CDN sleep):

const chunkKey = `${contentLength}:${chunk.startByte}-${chunk.endByte}`;
const cached = await chunkCache?.get(chunkKey).catch(() => null);
if (cached) {
    transcriptions.push({
        text: cached.text,
        isFirst: chunk.isFirst,
        isLast: chunk.isLast,
        overlapBytes: chunk.overlapBytes,
    });
    if (cached.model !== provider.model) {
        usedFallbackModel = cached.model;
    }
    console.log(JSON.stringify({
        event: "chunk_cache_hit",
        chunkIndex: i + 1,
        totalChunks,
        model: cached.model,
    }));
    continue;
}

After a successful transcription (next to the push at :987), write best-effort — a cache failure must never fail the chunk:

await chunkCache?.put(chunkKey, {
    text: chunkResult.text.trim(),   // mirror what the push stores
    model: chunkResult.model,
}).catch((e) => {
    console.log(JSON.stringify({
        event: "chunk_cache_write_failed",
        chunkIndex: i + 1,
        error: e instanceof Error ? e.message : "unknown",
    }));
});

3. KV wiring (src/lib/kv.ts + src/queue/consumer.ts:530-533). New key builder in KV_KEYS plus a factory:

// in KV_KEYS (kv.ts:9)
chunkCache: (episodeId: string, chunkKey: string) => `chunkcache:${episodeId}:${chunkKey}`,

export function makeChunkTranscriptCache(kv: KVNamespace, episodeId: string): ChunkTranscriptCache {
    return {
        async get(chunkKey) {
            const raw = await kv.get(KV_KEYS.chunkCache(episodeId, chunkKey));
            return raw ? (JSON.parse(raw) as { text: string; model: string }) : null;
        },
        async put(chunkKey, value) {
            await kv.put(KV_KEYS.chunkCache(episodeId, chunkKey), JSON.stringify(value), {
                expirationTtl: TTL.JOB,
            });
        },
    };
}

Consumer call becomes:

const transcriptionResult = await transcribeAudio(audioUrl, {
    apiKey: env.OPENAI_API_KEY,
    skipValidation,
    chunkCache: makeChunkTranscriptCache(kv, episodeId),
});

4. Sweep threshold (src/lib/constants.ts:317): JOB_MS: 35 * 60 * 1000 with a comment explaining the math (one 15-min killed attempt + a resumed attempt + summary fits inside 35; without this, every rescued episode flaps failed→completed on Discord). Update any tests asserting the 20-minute value.

5. Docs. CLAUDE.md KV Storage Schema section: add chunkcache:{episodeId}:{contentLength}:{start}-{end} — per-chunk transcript cache for retry resume (TTL: 1 day).

6. Tests (repo norm: TDD the behavior, don't batch-edit assertions). With a Map-backed fake cache:

  • Miss path: each successful chunk lands in the cache (text + model).
  • Resume path: pre-seed chunks 1–2, mock only chunk 3's fetch + transcription → assert fetch was NOT called for chunks 1–2 (call count) and the stitched result contains all three in order.
  • Fallback bookkeeping: a pre-seeded chunk with model: "whisper-1" → result.model is "whisper-1" even when live chunks use the primary.
  • Degradation: get and put that throw → transcription completes as if uncached.
  • Identity guard: seed under one contentLength, run with another → all misses, full transcription.
  • No cache option → behavior identical to today (existing tests keep passing untouched).
  • Sweep: threshold assertion updated to 35 min.

Edge cases (accounted for, no code needed)

  • KV eventual consistency: the chunk written seconds before a kill may miss on the retry's read — it's simply re-transcribed and re-written. Idempotent.
  • All chunks cached (killed during stitch/summarize): the loop hits N caches and stitches instantly.
  • Tail-chunk-skip / partial transcripts: unchanged — only successes are cached, and the tail-failure tolerance path doesn't interact with the cache.
  • Non-chunked files: untouched. A ≤25MB single call fits any attempt (an 83.7-min whisper-1 single call measured 4m18s in the Migrate transcription from gpt-4o-mini-transcribe + whisper-1 to gpt-transcribe #46 pre-flight).
  • Manual resubmit racing a zombie attempt: both write identical content under identical keys. Harmless.
  • Cleanup: none — TTL.JOB (1 day) expires the keys; a completed episode never reads them again (episode-level transcript short-circuits first).

Verify (after merge)

  • npm test green (497+ currently).
  • Deploy is manual: npm run deploy.
  • Resubmit the motivating episode (Lenny's Podcast, feed https://api.substack.com/feed/podcast/10845.rss, episode "This CPO regrets that product management exists"). Expected: attempt 1 runs ~15 min, logs whisper_fallback_start/whisper_fallback_success per chunk, gets killed (exceededWallTime); attempt 2 logs chunk_cache_hit for the cached chunks, finishes the remainder, job goes completed — with no Discord failure alert in between (35-min sweep).
  • Spot-check a cache key mid-flight: npx wrangler kv key list --namespace-id=ee123158d5d54359b4257f8a1b678adf --prefix "chunkcache:" --remote.
  • Logs are queryable retroactively via POST /accounts/{id}/workers/observability/telemetry/query if the live window is missed.

Not in scope (considered, rejected or deferred)

  • One-queue-message-per-chunk pipeline — the architectural fix that makes the wall clock unreachable, but it needs a new message schema, chunk-completion coordination, and job-status rework. The cache gets the same outcome for a fraction of the change.
  • Skip-the-doomed-primary optimization (after the first chunk falls back, send the rest straight to whisper-1) — saves ~2s per chunk; not worth forking the per-chunk behavior.
  • Explicit cache cleanup on completion — TTL handles it.
  • Root-causing why the 4o-family decoder rejects this ordinary-looking MP3 — whisper-1 handles it; the fallback + this cache make the question academic for tldl.

References

Activity

  1. added 20 commits that reference this issue on Nov 12, 2026
  2. rianvdm commented on Nov 12, 2026

    @rianvdm
    OwnerAuthor

    Fixed in 1743b90 and deployed on 2026-11-12 (version d34f3d64).

    Each finished chunk's transcript is now written to KV with a one-day TTL, and a retry skips any chunk whose transcript it can read back, so a killed or crashed attempt resumes where it stopped. The key (chunkCacheKey) includes the planner version, container, file size, chunk index and byte or sample range, plus a strong ETag or Last-Modified when the host sends one. Without either, a same-size replacement file could still reuse old text. The stale-job sweep moved from 20 to 35 minutes, still measured from when the job was created.

    The slow lane behind this is gone too. Every chunk here fell back to whisper-1 after gpt-transcribe rejected it as "corrupted or unsupported". On Supra Insider #135 the spike traced that error to the file's LAME Info frame, which declares the whole episode's length and which the MP3 chunker copied into every chunk; this issue's episode carries the same kind of header (it declares 85 minutes). gpt-transcribe also rejected a clean 70-minute file with the same error, while 20, 30 and 45 minutes passed. The MP3 frame path now strips that header, MP3 uploads are measured frame by frame and split if they run past 15 minutes, and MP4 and ADTS are planned by duration too. MP3s the parser can't read, and other formats, keep the old byte path. The whisper-1 fallback is removed; OpenAI lists 2027-02-26 as whisper-1's removal date on its deprecations page.

    Before the final review fixes, I ran the chunk check script with --transcribe against five real episodes, including this issue's Lenny's episode ("This CPO regrets that product management exists") and Supra Insider #135. gpt-transcribe accepted all 28 uploads. After those fixes, an offline check of the Supra file still planned the same seven chunks within the cap. The design and the spike results are in the spec.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions