Repository navigation
Description
Why
On 2026-08-02 a Lenny's Podcast episode ("This CPO regrets that product management exists", episodeId 1627920305_rss_9d799ca4a3, 84 min, 77.8MB Substack-hosted MP3) could not be processed and burned through its retry budget without producing anything. The failure chain, verified live:
- Both
gpt-transcribeandgpt-4o-mini-transcribereject this file's chunks with HTTP 400"Audio file might be corrupted or unsupported"— confirmed by posting the actual first 15MB chunk to both models directly (identical error body, ~1.5s each). Not a Migrate transcription from gpt-4o-mini-transcribe + whisper-1 to gpt-transcribe #46 regression; the file looks ordinary (MPEG-1 Layer III, 128kbps, 44.1kHz stereo, 269-byte ID3v2.4 tag) and whisper-1 decodes it fine. - So every chunk takes the whisper-1 fallback at ~90–150s per 15MB chunk (measured 92–96s in the Migrate transcription from gpt-4o-mini-transcribe + whisper-1 to gpt-transcribe #46 pre-flight), instead of gpt-transcribe's ~20s.
- Six chunks at fallback speed plus fetches exceeds the 15-minute wall-clock limit Cloudflare puts on queue consumer invocations. The invocation was killed (
exceededWallTimein the observability logs), the message redelivered, and the retry restarted transcription from chunk 1 — so it can never converge. Withmax_retries = 5(wrangler.toml:39) that's up to ~75 minutes of API spend with zero output, re-paying OpenAI for the same chunks on every attempt. - Meanwhile the 20-minute stale-job sweep marked the still-working job failed ("Job timed out after 21 minutes") and fired a false Discord failure alert.
Contrast, same day: an Ezra Klein episode (58MB → 4 chunks, all on gpt-transcribe at 17–26s each) completed in ~5 minutes. The fast lane is fine; only the fallback lane breaks, and only on long files.
Episode-level resume already exists — processEpisode loads an existing transcript (src/queue/consumer.ts:376) and skips transcription entirely when one is saved. The gap is chunk-level: a killed attempt's completed chunks live only in memory. Fix: cache per-chunk transcripts in KV so a retry resumes from the first uncached chunk. For the Lenny case, attempt 1 caches ~5 of 6 chunks before the kill and attempt 2 finishes in a few minutes.
Current state (verified in code, 2026-08-02)
- Chunk loop in
transcribeWithChunking(src/services/transcription.ts:886): iterate (:925), Range-fetch (:949), transcribe (:978), accumulate in memory (:987), fallback-model bookkeeping (:996). Nothing is persisted until the consumer saves the full stitched transcript (src/queue/consumer.ts:553). - Chunk ranges are a pure function of
contentLength:calculateChunkRanges(src/lib/audio.ts:150), 15MB target, 32KB overlap — so chunk identity is deterministic across attempts for the same file. TranscribeOptions(src/services/transcription.ts:699) carries onlyapiKey/provider/onProgress/skipValidation; the service has no storage access or episode identity (deliberately pure).- Consumer call site (
src/queue/consumer.ts:530-533):transcribeAudio(audioUrl, { apiKey, skipValidation })—kvandepisodeIdare both in scope there. - KV key builders at
src/lib/kv.ts:9(KV_KEYS), TTLs at:26(TTL.JOB= 1 day). - Stale-job sweep:
src/lib/job-status-do.ts:156comparesjobAge > TIMEOUTS.JOB_MS;TIMEOUTS.JOB_MSis 20 min (src/lib/constants.ts:317). The timeout message computes minutes from the constant, so it self-adjusts. - Queue config:
max_batch_size = 1(wrangler.toml:37),max_retries = 5(:39). The 15-minute consumer wall clock is a Cloudflare platform limit, not configurable.
Target
| Current | Target | |
|---|---|---|
| Retry after a wall-time kill | restarts from chunk 1 | resumes from first uncached chunk |
| Chunk transcripts | in-memory only | cached per chunk in KV, 1-day TTL |
| Whisper-lane 84-min episode | never converges (5 kills → failed) | attempt 1 (killed at 15 min) + attempt 2 (~5 min) → completed |
| Fast lane (gpt-transcribe accepts) | unchanged | unchanged behavior + cache writes |
| Stale-job sweep | 20 min — false-fails resumed jobs | 35 min |
Work
1. Cache contract (src/services/transcription.ts, near TranscribeOptions at :699):
export interface ChunkTranscriptCache {
/** chunkKey is `${contentLength}:${startByte}-${endByte}` — chunk identity, no episode identity. */
get(chunkKey: string): Promise<{ text: string; model: string } | null>;
put(chunkKey: string, value: { text: string; model: string }): Promise<void>;
}Add chunkCache?: ChunkTranscriptCache to TranscribeOptions, and thread it through both transcribeWithChunking call sites (:819, :842) into its signature (:886). Keying on contentLength means a changed file next attempt (CDN re-encode, different length) misses cleanly — including early chunks whose byte ranges would otherwise coincide.
2. Resume logic in the chunk loop (:925-996). Insert the hit check after the chunk_processing log, before the try block (so cached chunks also skip the inter-chunk CDN sleep):
const chunkKey = `${contentLength}:${chunk.startByte}-${chunk.endByte}`;
const cached = await chunkCache?.get(chunkKey).catch(() => null);
if (cached) {
transcriptions.push({
text: cached.text,
isFirst: chunk.isFirst,
isLast: chunk.isLast,
overlapBytes: chunk.overlapBytes,
});
if (cached.model !== provider.model) {
usedFallbackModel = cached.model;
}
console.log(JSON.stringify({
event: "chunk_cache_hit",
chunkIndex: i + 1,
totalChunks,
model: cached.model,
}));
continue;
}After a successful transcription (next to the push at :987), write best-effort — a cache failure must never fail the chunk:
await chunkCache?.put(chunkKey, {
text: chunkResult.text.trim(), // mirror what the push stores
model: chunkResult.model,
}).catch((e) => {
console.log(JSON.stringify({
event: "chunk_cache_write_failed",
chunkIndex: i + 1,
error: e instanceof Error ? e.message : "unknown",
}));
});3. KV wiring (src/lib/kv.ts + src/queue/consumer.ts:530-533). New key builder in KV_KEYS plus a factory:
// in KV_KEYS (kv.ts:9)
chunkCache: (episodeId: string, chunkKey: string) => `chunkcache:${episodeId}:${chunkKey}`,
export function makeChunkTranscriptCache(kv: KVNamespace, episodeId: string): ChunkTranscriptCache {
return {
async get(chunkKey) {
const raw = await kv.get(KV_KEYS.chunkCache(episodeId, chunkKey));
return raw ? (JSON.parse(raw) as { text: string; model: string }) : null;
},
async put(chunkKey, value) {
await kv.put(KV_KEYS.chunkCache(episodeId, chunkKey), JSON.stringify(value), {
expirationTtl: TTL.JOB,
});
},
};
}Consumer call becomes:
const transcriptionResult = await transcribeAudio(audioUrl, {
apiKey: env.OPENAI_API_KEY,
skipValidation,
chunkCache: makeChunkTranscriptCache(kv, episodeId),
});4. Sweep threshold (src/lib/constants.ts:317): JOB_MS: 35 * 60 * 1000 with a comment explaining the math (one 15-min killed attempt + a resumed attempt + summary fits inside 35; without this, every rescued episode flaps failed→completed on Discord). Update any tests asserting the 20-minute value.
5. Docs. CLAUDE.md KV Storage Schema section: add chunkcache:{episodeId}:{contentLength}:{start}-{end} — per-chunk transcript cache for retry resume (TTL: 1 day).
6. Tests (repo norm: TDD the behavior, don't batch-edit assertions). With a Map-backed fake cache:
- Miss path: each successful chunk lands in the cache (text + model).
- Resume path: pre-seed chunks 1–2, mock only chunk 3's fetch + transcription → assert fetch was NOT called for chunks 1–2 (call count) and the stitched result contains all three in order.
- Fallback bookkeeping: a pre-seeded chunk with
model: "whisper-1"→result.modelis"whisper-1"even when live chunks use the primary. - Degradation:
getandputthat throw → transcription completes as if uncached. - Identity guard: seed under one
contentLength, run with another → all misses, full transcription. - No cache option → behavior identical to today (existing tests keep passing untouched).
- Sweep: threshold assertion updated to 35 min.
Edge cases (accounted for, no code needed)
- KV eventual consistency: the chunk written seconds before a kill may miss on the retry's read — it's simply re-transcribed and re-written. Idempotent.
- All chunks cached (killed during stitch/summarize): the loop hits N caches and stitches instantly.
- Tail-chunk-skip / partial transcripts: unchanged — only successes are cached, and the tail-failure tolerance path doesn't interact with the cache.
- Non-chunked files: untouched. A ≤25MB single call fits any attempt (an 83.7-min whisper-1 single call measured 4m18s in the Migrate transcription from gpt-4o-mini-transcribe + whisper-1 to gpt-transcribe #46 pre-flight).
- Manual resubmit racing a zombie attempt: both write identical content under identical keys. Harmless.
- Cleanup: none —
TTL.JOB(1 day) expires the keys; a completed episode never reads them again (episode-level transcript short-circuits first).
Verify (after merge)
npm testgreen (497+ currently).- Deploy is manual:
npm run deploy. - Resubmit the motivating episode (Lenny's Podcast, feed
https://api.substack.com/feed/podcast/10845.rss, episode "This CPO regrets that product management exists"). Expected: attempt 1 runs ~15 min, logswhisper_fallback_start/whisper_fallback_successper chunk, gets killed (exceededWallTime); attempt 2 logschunk_cache_hitfor the cached chunks, finishes the remainder, job goescompleted— with no Discord failure alert in between (35-min sweep). - Spot-check a cache key mid-flight:
npx wrangler kv key list --namespace-id=ee123158d5d54359b4257f8a1b678adf --prefix "chunkcache:" --remote. - Logs are queryable retroactively via
POST /accounts/{id}/workers/observability/telemetry/queryif the live window is missed.
Not in scope (considered, rejected or deferred)
- One-queue-message-per-chunk pipeline — the architectural fix that makes the wall clock unreachable, but it needs a new message schema, chunk-completion coordination, and job-status rework. The cache gets the same outcome for a fraction of the change.
- Skip-the-doomed-primary optimization (after the first chunk falls back, send the rest straight to whisper-1) — saves ~2s per chunk; not worth forking the per-chunk behavior.
- Explicit cache cleanup on completion — TTL handles it.
- Root-causing why the 4o-family decoder rejects this ordinary-looking MP3 — whisper-1 handles it; the fallback + this cache make the question academic for tldl.
References
- Migrate transcription from gpt-4o-mini-transcribe + whisper-1 to gpt-transcribe #46 — gpt-transcribe migration; its pre-flight measured the fallback chunk timings (92–96s per 15MB) and single-call whisper duration used above.
- Cloudflare Queues limits — 15-minute consumer wall clock.
- Incident evidence 2026-08-02: invocation outcome
exceededWallTime(started 21:02:30Z), sweep false-fail at 21:23:40Z, A/B chunk probe against both models (identical 400s).
Activity
Add design for transcribing without whisper-1 (#48)
Add implementation plan for transcribing without whisper-1 (#48)
Parse MP3 frames and Xing/Info/VBRI headers (#48)
Plan MP3 chunks on frame boundaries without the VBR header, capped at…
Require a second frame header when an MP3 buffer stops mid-file (#48)
Route every upload through the chunk planner so none exceeds 15 minut…
Remove the whisper-1 fallback; a model rejection is AUDIO_REJECTED an…
Fail rejected audio on the first attempt and put the check command in…
Raise the stale-job sweep to 35 minutes so a resumed job isn't marked…
Keep the signed audio URL out of logs and shell-quote it in the check…
Cache chunk transcripts in KV so a retry resumes instead of restarting (
Extend the chunk check script to MP3 and every container (#48)
Keep check-chunks output inside its results folder and upload chunks …
Document transcription without whisper-1 and the rejection runbook (#48)
Route check:chunks through the Worker's upload routing so small files…
Find LAME's Info tag in CRC-protected MP3 frames (#48)
Measure every MP3 upload by its frames and split one past the cap (#48)
Key cached chunk transcripts by the source's ETag or Last-Modified (#48)
Log audio hostnames, never full URLs, and only allowlisted headers (#48)
Make the retry-guard tests prove the guard, and qualify the 15MB chun…
Log an audio host's error body by size and type only, never its text (#…
Merge fix/transcribe-without-whisper: transcribe without whisper-1 (#48)
rianvdm commentedon Nov 12, 2026
Fixed in 1743b90 and deployed on 2026-11-12 (version
d34f3d64).Each finished chunk's transcript is now written to KV with a one-day TTL, and a retry skips any chunk whose transcript it can read back, so a killed or crashed attempt resumes where it stopped. The key (
chunkCacheKey) includes the planner version, container, file size, chunk index and byte or sample range, plus a strong ETag or Last-Modified when the host sends one. Without either, a same-size replacement file could still reuse old text. The stale-job sweep moved from 20 to 35 minutes, still measured from when the job was created.The slow lane behind this is gone too. Every chunk here fell back to whisper-1 after gpt-transcribe rejected it as "corrupted or unsupported". On Supra Insider #135 the spike traced that error to the file's LAME Info frame, which declares the whole episode's length and which the MP3 chunker copied into every chunk; this issue's episode carries the same kind of header (it declares 85 minutes). gpt-transcribe also rejected a clean 70-minute file with the same error, while 20, 30 and 45 minutes passed. The MP3 frame path now strips that header, MP3 uploads are measured frame by frame and split if they run past 15 minutes, and MP4 and ADTS are planned by duration too. MP3s the parser can't read, and other formats, keep the old byte path. The whisper-1 fallback is removed; OpenAI lists 2027-02-26 as whisper-1's removal date on its deprecations page.
Before the final review fixes, I ran the chunk check script with
--transcribeagainst five real episodes, including this issue's Lenny's episode ("This CPO regrets that product management exists") and Supra Insider #135. gpt-transcribe accepted all 28 uploads. After those fixes, an offline check of the Supra file still planned the same seven chunks within the cap. The design and the spike results are in the spec.