Skip to content

Send transcription hints (prompt + keywords[]) from episode metadata to gpt-transcribe #47

Open

Description

@rianvdm

Why

Podcast transcripts get proper nouns wrong: podcast names, host and guest names, product names. The transcription API's hint parameters exist for exactly this problem, and we already hold the relevant metadata (podcastName, episodeTitle, podcastAuthor) in the queue consumer at the moment we call transcribeAudio. Today we throw it away.

This is the follow-up carved out of #46 ("Not in scope: prompt, keywords[], languages[]"), kept separate so a behavior change can't hide inside the model swap. Blocked by #46: keywords[] is a gpt-transcribe-only parameter and would 400 against gpt-4o-mini-transcribe. prompt alone works on the current model, but shipping half the feature ahead of the swap means two behavior changes instead of one, so the whole thing lands after #46.

The open question #46 recorded (how array params encode in multipart form data) is answered in the docs: the speech-to-text guide's own curl example sends repeated keywords[] fields (-F 'keywords[]=premium plan' -F 'keywords[]=AC-42'), matching the timestamp_granularities[] convention. Still worth one live confirmation in pre-flight since we build FormData by hand.

API surface (verified against docs, 2026-08-02)

From the speech-to-text guide and the migration cookbook:

Param Type Models Semantics
prompt string gpt-transcribe, gpt-4o-transcribe(-mini) "Unstructured context about the recording." Cookbook: describe the topic/setting, don't list terms there
keywords string[] (multipart: repeated keywords[]) gpt-transcribe only "Literal terms you expect to hear." Hints, not required output
languages string[] (multipart: repeated languages[]) gpt-transcribe only Expected input languages; replaces the legacy language field

Validation rules that matter to us:

  • Each keyword must be a single-line literal and must not contain <, >, CR, or LF. The API rejects the request otherwise.
  • Never send both language and languages.
  • An over-long prompt is rejected. No numeric limit is documented, so keep it to a sentence.

whisper-1 supports none of this in the same form: its prompt is the old style/continuation hint (224-token cap, different semantics) and it takes a singular language. This matters for the fallback path below.

Current state (verified in code, 2026-08-02)

Line refs are current main; #46 will shift them slightly.

  • The queue consumer calls transcribeAudio(audioUrl, { apiKey, skipValidation }) at src/queue/consumer.ts:531 with metadata.podcastName, metadata.episodeTitle, and metadata.podcastAuthor all in scope. Both metadata paths populate podcastAuthor: the RSS-sourced path from monitored?.author (consumer.ts:444), the Apple-submit path via getEpisodeMetadata.
  • callWhisperApi builds the request FormData at src/services/transcription.ts:474-478 (file, model, response_format only). Chunked episodes go through transcribeWithChunking (:869), which calls callWhisperApi once per chunk (:962). Each chunk is an independent API call, so hints have to reach every call.
  • The whisper-1 fallback (:581-584) builds its own bare FormData. It fires on 400s whose body matches the "corrupted or unsupported" / "input_too_large" substrings. A hint-validation 400 would NOT match those substrings, so a bad keyword would fail the chunk outright rather than falling back. That's why sanitization happens at hint-build time, not in the API call.
  • TranscribeOptions (:682) currently carries apiKey, provider (legacy), onProgress, skipValidation.
  • Tests already inspect FormData bodies from the fetch spy (test/transcription.test.ts:464-469, :596), so asserting hint fields follows the existing pattern.
  • No language metadata exists anywhere in the pipeline. The RSS parser doesn't read the feed's <language> tag, and neither EpisodeMetadata nor the monitored-podcast KV records store one. (This is why languages[] is out of scope below.)

Design

Send per-episode hints on the primary (gpt-transcribe) call only:

  • prompt: one sentence of context, per the cookbook's "describe the recording" guidance: An episode of the podcast "<podcastName>", hosted by <podcastAuthor>, titled "<episodeTitle>". with segments dropped when a field is missing or a placeholder. The episode title's proper nouns (usually the guest's name) reach the model here.
  • keywords[]: [podcastName, podcastAuthor]. These are literal terms actually spoken in nearly every episode intro and outro. The episode title is deliberately not a keyword: keywords are for "literal terms you expect to hear," and a full editorial title isn't one.
  • Same hints on every chunk of a chunked episode.
  • whisper-1 fallback: completely unchanged. It exists to rescue chunks the primary rejected, so it stays the most minimal request we know works. keywords[] would 400 there anyway.
  • No hints when metadata is placeholder junk ("Unknown Podcast" / "Untitled Episode" from the RSS-path defaults). Send nothing rather than teach the model garbage.

Cost: a sentence of prompt plus two short keywords is tens of text-input tokens per call, a rounding error at admin-curated volume.

Caveat on measurement: the A/B harness covers summary models only; there is no transcription eval. Acceptance is the pre-flight spot check on known-bad proper nouns, not a metric.

Work

1. Add the hint builder (src/services/transcription.ts), exported for the consumer and tests:

export interface TranscriptionHints {
    prompt?: string;
    keywords?: string[];
}

/** Strip chars the API rejects in keywords (<, >, CR, LF), collapse whitespace. */
function sanitizeHintText(value: string): string {
    return value.replace(/[<>\r\n]/g, " ").replace(/\s+/g, " ").trim();
}

export function buildTranscriptionHints(meta: {
    podcastName?: string;
    episodeTitle?: string;
    podcastAuthor?: string;
}): TranscriptionHints | undefined

Rules:

  • Sanitize all three fields; treat "Unknown Podcast", "Untitled Episode", and empty strings as absent.
  • Cap the title's contribution (~200 chars) so a pathological RSS title can't produce an over-long prompt.
  • keywords = sanitized [podcastName, podcastAuthor], deduped case-insensitively (author sometimes equals the podcast name), empties dropped.
  • Return undefined when nothing usable remains, so callers can pass the result straight through.

2. Thread hints to the API call. TranscribeOptions gains hints?: TranscriptionHints. transcribeAudio passes opts.hints to both the single-shot call (:837) and transcribeWithChunking (new parameter, :869-876), which passes them to every per-chunk callWhisperApi (:962). In callWhisperApi, after the existing appends at :474-478:

if (hints?.prompt) formData.append("prompt", hints.prompt);
for (const keyword of hints?.keywords ?? []) {
    formData.append("keywords[]", keyword);
}

The fallback FormData (:581-584) gets no hint fields. Add a comment saying that's deliberate so nobody "fixes" it later.

3. Pass metadata in the consumer (src/queue/consumer.ts:531):

const transcriptionResult = await transcribeAudio(audioUrl, {
    apiKey: env.OPENAI_API_KEY,
    skipValidation,
    hints: buildTranscriptionHints(metadata),
});

4. Debug route support (src/index.ts:186). Optional podcast, title, author query params on GET /debug/transcribe, run through buildTranscriptionHints, so pre-flight can exercise hints in dev without queueing a job.

5. Observability. Add hasPrompt: boolean and keywordCount: number to the whisper_api_call_start log event. Hint content is recoverable from episode metadata, so counts are enough to see whether hints were in play when eyeballing a bad transcript in wrangler tail.

6. Tests (test/transcription.test.ts, TDD per repo norm):

  • buildTranscriptionHints: strips <>/CR/LF, collapses whitespace, drops placeholders, dedupes name/author, truncates long titles, returns undefined on junk-only input.
  • callWhisperApi FormData contains prompt and repeated keywords[] entries when hints are passed (assert via formData.getAll("keywords[]") on the fetch spy), and neither field when hints are absent.
  • The fallback call's FormData contains no prompt or keywords[].
  • Chunked path: every chunk's call carries the same hint fields.

7. Docs. One line in AGENTS.md (Request Flow step 2 and the services/transcription.ts structure entry) plus the transcribeAudio doc-comment example.

Pre-flight check (dev, before merging)

Using GET /debug/transcribe (requires ENVIRONMENT=development):

  1. One small episode with hints: confirm HTTP 200 and a normal body. This is the live confirmation of the keywords[] multipart encoding that Migrate transcription from gpt-4o-mini-transcribe + whisper-1 to gpt-transcribe #46 asked for.
  2. Find an episode whose existing prod transcript misspells the podcast, host, or guest name (browse recent transcript:{episodeId} records for candidates). Re-run it through the branch with hints and diff the proper nouns. This is the acceptance check.
  3. One chunked (>25MB) episode: confirm hints ride along on every chunk (log events) and the stitched output looks normal.

Verify (after merge)

  • npm test green.
  • Deploy is manual (npm run deploy); merging does not deploy.
  • npx wrangler tail on the next cron tick: whisper_api_call_start shows keywordCount: 2 (or 1), and whisper_fallback_start doesn't spike. A spike would mean the hinted request 400s in some new way; the logged error body will show the wording.
  • Spot-check the next few transcripts for podcast/host name spelling.

Not in scope

  • languages[] — nothing in the pipeline knows the episode language (no RSS <language> capture, no stored field), so there's no honest value to send. Hardcoding ["en"] bakes an assumption into every future podcast for zero observed benefit, and auto-detect plus the detected-languages response field already cover it. If a non-English podcast joins the archive, first capture <language> at ingest, then send it per call (and never alongside legacy language).
  • Guest names as keywords — the title's proper nouns already reach the model via prompt; promoting them to keywords means extracting names from editorial titles (heuristics or an LLM pass). Revisit only if pre-flight shows title-cased names still coming out wrong.
  • Continuation prompting across chunks (previous chunk's tail as prompt) — that's whisper-1's prompt semantics, not gpt-transcribe's context prompt, and it would interact with the overlap-dedup stitching.
  • prompt on the whisper-1 fallback — different parameter semantics (style/continuation, 224-token cap); the fallback stays minimal on purpose.
  • Retry-without-hints on 400 — sanitization makes hint rejection unreachable for expected inputs; if a rejection shows up anyway, whisper_api_error logs the full body and we handle it then.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions