Tooling for generating parallel-subtitle videos of full-length operas — original language on the left, translation on the right, word-synced to the singing. This is the repo behind the YouTube channel of the same name; example output: Tristan und Isolde with parallel subtitles.
One YAML config in, one video out:
- Libretto —
fetch_libretto.pypulls the libretto (and its translation, when available) from librettoarchive.com into a simple blank-line-separated text format;translate.pyfills in a missing translation with Claude, block-for-block. - Audio —
download_album.pyresolves a recording to its YouTube Music album (auto-generated art-tracks, in order) and downloads it with yt-dlp;demucsseparates the vocals — on a Modal GPU (separate_modal.py, minutes and cents per opera) or locally on CPU. - Transcription — ElevenLabs Scribe transcribes the vocal stems with word timestamps; each track is graded against how much singing the stem actually contains, and tracks with holes or hallucination loops are filled in from Whisper running on a Modal GPU;
detect_instrumental.pyflags purely orchestral tracks so their transcripts get ignored. - Alignment + render —
make_video.pyaligns the transcript to the libretto (Needleman-Wunsch over Levenshtein similarity), interpolates timings, pairs source/translation blocks and renders the video with moviepy; it also prints YouTube chapter markers.
- Python 3.10+,
ffmpeg, ImageMagick (convert) - The Claude Code CLI (
claude), logged in — translation runs through it .envwithELEVENLABS_API_KEYandOPENAI_API_KEY- Optional: a Modal account (
modal token new) for GPU vocal separation; without it, demucs runs locally on CPU (slow)
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txtA minimal config (configs/<opera>.yaml):
title: CARMEN
file_prefix: carmen
language: fr # language sung (ISO-639-1)
album_query: Bizet Carmen Abbado London Symphony Orchestra # or album_url: https://music.youtube.com/browse/MPREb_...
res_divisor: 4 # 4 = quick 960x540 preview, 1 = final 4KEverything else — number of tracks, which tracks are instrumental, character names for bold formatting, the translation file — is derived automatically; see CLAUDE.md for the full field list and how each is derived. Any field can still be set explicitly.
python kunstwerk.py configs/carmen.yaml # the whole pipeline
python kunstwerk.py configs/carmen.yaml --stop-after libretto # fetch + translate, then stop to eyeball the text
python kunstwerk.py configs/carmen.yaml --skip-download --skip-transcribe # re-render only
python kunstwerk.py configs/carmen.yaml --copyright-test # download the audio and make a black-screen video to test YouTube Content ID before investing in the full renderEvery stage is idempotent — it skips work whose output already exists — so re-running after a failure picks up where it left off. Each stage is also its own script (fetch_libretto.py, translate.py, separate.sh, download_album.py, detect_instrumental.py, transcribe_tracks.py, make_video.py); run them individually when you want to intervene, e.g. hand-edit aligned_words_<prefix>.csv between alignment and render.
Outputs land in output/<prefix>-<res_divisor>.mp4.
kunstwerk.py— orchestratorconfig_parser.py— config loading and derived fieldsfetch_libretto.py,translate.py— libretto acquisitiondownload_album.py,separate.sh,separate_modal.py,detect_instrumental.py— audio acquisition and separationtranscribe_tracks.py,transcribe_modal.py— transcription (ElevenLabs Scribe, quality gate, Whisper-on-Modal fallback)align.py— transcript ↔ libretto alignmentmake_video.py,video_gen/— frame generation and renderinglibretti/,configs/— per-opera inputs