Replace a video's original English vocals with Chinese voiceover on an identity timeline: the video is never re-timed — every frame keeps its original timestamp and the dubbed release has exactly the raw video's duration. Each cue's synthesized audio is fitted into its own original window (cue start → next cue's start) with a per-cue atempo; faster or slower synthesis both land in the window. The result is a second release — same picture at the same speed, Chinese audio, bilingual ZH+EN subtitles burned in.
This skill does the two creative parts the CLI can't: translating for dubbing (complete sentences written to a char budget, not the subtitle fragmentation) and running the quality gates (length gate, cold review, ear gate, post-burn review). Deterministic execution (Demucs, IndexTTS2, assembly, retime, ffmpeg) is handled by the cook CLI's cook dub subcommand. All stage scripts live inside the cook package (stages/dub/); every invocation below is a cook command. pip install -U video-cook once upgrades everything.
When to reach for this skill
You have a video that already has:
- A raw video file (
<output-root>/raw/<name>.raw.mp4) — the original, with English vocals. - A bilingual subtitle run from
video-subtitle— specificallytranscript/<name>.en.full.srt(the full-sentence English transcript, merged from whisperX fragments) andtranscript/translations.txt.
You want a Chinese-dubbed release. If you don't have these yet, run video-download then video-subtitle first — this skill reads their outputs.
What you produce
Three products ship: cooked/<name>.dubbed.mp4 (raw video untouched, Chinese dub, burned bilingual subtitles — duration identical to the raw), cloud-srt/zh.dub.srt + en.dub.srt (upload subtitles). Everything else is working files under dubbed/ — the annotated tree below maps every one of them.
The run is not done until Step 6's checks pass.
Directory layout
This skill adds dubbed/ (working directory) and writes the final products to cooked/ and cloud-srt/:
<output-root>/
├── raw/ ← from video-download (this skill reads it)
│ └── <name>.raw.mp4
├── transcript/ ← from video-subtitle (this skill reads + adds)
│ ├── <name>.en.full.srt ← full-sentence English (the dub script source)
│ ├── translations_dub.txt ← this skill writes: one Chinese line per cue
│ └── <name>.zh.dub.srt ← this skill writes
├── cooked/ ← final videos live here
│ ├── <name>.cooked.bar.mp4 ← from video-subtitle (untouched)
│ └── <name>.dubbed.mp4 ← this skill's product
├── cloud-srt/ ← upload subtitles live here
│ ├── zh.srt / en.srt ← from video-subtitle (untouched)
│ └── zh.dub.srt / en.dub.srt ← this skill's upload subtitles
└── dubbed/ ← this skill's working directory
├── _reference/
│ └── ref.wav
└── _full/
├── _segments/ ← per-cue IndexTTS2 cache
├── dub.wav ← the dub on the original clock
├── dubbing.srt ← working file (ZH, pre-shorten)
├── dubbing.short.srt ← working file (ZH, shorten output)
├── dubbing.merged.srt ← working file (ZH, post-shorten; copied to cloud-srt)
├── dubbing.en.srt ← working file (full-sentence EN)
├── dubbing.en.short.srt / dubbing.en.merged.srt ← its shorten+merge-short outputs
├── dubbing.bilingual.srt ← working file (biliteral union; what gets burned)
├── dubbing.cooked.ass ← working file (ass output, copied to burn.ass)
└── burn.ass ← the ASS actually burned
Rule: dubbed/ is the working directory; cooked/<name>.dubbed.mp4 and cloud-srt/{zh,en}.dub.srt are the products. Never touch raw/, transcript/<name>.zh.srt, cooked/<name>.cooked.mp4, or cloud-srt/{zh,en}.srt — those belong to video-subtitle. If this skill fails halfway, the bilingual cooked shipment is still complete.
The pipeline
Two stages after the creative work: synth then assemble, invoked through cook:
cook dub synth <root> <name> --python <indextts-venv>/Scripts/python.exe
cook dub assemble <root> <name> --python <indextts-venv>/Scripts/python.exe [--keep-subs]
# or both:
cook dub full <root> <name> --python <indextts-venv>/Scripts/python.exe
The steps below describe what each stage does internally (so you can verify outputs and diagnose failures); the cook dub <stage> commands are how you run them.
Step 0 — Resolve the environments
There are two Python environments in play, and confusing them is the failure mode this step exists to prevent:
- cook's environment (system Python or
~/.venvs/video-tools/) — where thecookCLI lives, with whisperX/yt-dlp/torch for the subtitle pipeline. - IndexTTS2's environment (
~/Git/index-tts/.venv) — a separate venv holdingindextts,torch,demucs, andwhisperx(the dub pipeline adds the last two on top of upstream's lockfile). These deps are heavy and isolated on purpose; do not try to install them into cook's environment.
cook runs each dub stage as a subprocess under the IndexTTS2 venv via --python, so from indextts import ... resolves there. Every cook dub command in this skill takes --python <indextts-venv>/Scripts/python.exe. Resolve the venv path once (default ~/Git/index-tts/.venv) and reuse it for the whole run.
0a. Probe both environments are reachable:
<cook-venv>/Scripts/cook doctor # whisperX/yt-dlp/ffmpeg
<indextts-venv>/Scripts/python -c "from indextts.infer_v2_5 import IndexTTS2; import demucs, whisperx; print('ok')" # indextts (v2.5) + demucs + whisperx
0b. Single-thread constraint. IndexTTS2 must run single-threaded (OMP_NUM_THREADS=1), or it produces garbage audio. cook sets this internally before importing torch, so you don't need to export it yourself.
Done when cook's doctor reports whisperX/yt-dlp/ffmpeg installed, the IndexTTS2 venv imports indextts (v2.5) + demucs + whisperx, and you know the absolute path to <indextts-venv>/Scripts/python.exe to pass as --python.
Step 1 — Separate vocals from the raw video
The original audio is one mixed track (vocals + BGM + SFX). Demucs splits it so we can extract a clean reference and check for BGM later.
cook dub separate <output-root> <name> [--model htdemucs] --python <indextts-venv>/Scripts/python.exe
Demucs lives in the IndexTTS2 venv, so --python points there. Use htdemucs (single model, ~3GB RAM), not htdemucs_ft (bag of 4 models, ~20GB RAM — OOMs on 32GB machines). Quality is slightly lower but adequate for reference extraction.
In foreground mode (default), cook moves the separated stems to dubbed/vocals.wav + dubbed/no_vocals.wav automatically; in --detach mode you move them yourself after the done marker appears.
Done when dubbed/vocals.wav AND dubbed/no_vocals.wav both exist with duration matching raw ±0.5s.
Step 2 — Extract the reference clip
IndexTTS2 needs a 14-30 second clean clip of the original speaker — it reads at most the first 15s, so a full-cap window beats a short one. IndexTTS2 clones prosody, not just timbre: the window's delivery pace becomes the whole dub's delivery pace. The default picker takes the longest continuous speech, which tends to select the slowest, most deliberate passage — often wrong. Prefer the scanner:
cook dub scan-reference <output-root> <name> \
[--full-band] [--python <indextts-venv>/Scripts/python.exe]
It ranks fixed 15s windows by syllable rate (density- and pause-filtered) and cuts the top 3 candidates (plus full-band 48k copies with --full-band). Density is only half the story — IndexTTS2 clones the window's prosody too, so rank candidates by liveliness as well: pyin F0 std and semitone range per window (a flat window dubs flat; use librosa.pyin, NOT detect_pitch_frequency, which reports voicing on silence). Authenticate any legacy reference file before reusing it: a reprocessed derivative keeps its speaker embedding and pitch statistics but loses waveform correlation, so correlation near zero does NOT mean "different speaker" — score it against an anchor clip with cook dub scan-reference --score <file> --anchor <anchor.wav> (same-speaker real audio lands near 0.94; synthesized takes 0.66-0.76 is the normal band) and compare pyin stats. Either way: the user's ear picks the reference, always — synthesize the pilot sentences once per candidate (Step 3a), prescreen each take with a small-model ASR round-trip (dropped words flag garble; traditional-Chinese output flags an accent the window bleeds into the take), score with cook dub scan-reference --score, then hand the loudness-matched wavs to the user.
Done when dubbed/_reference/ref.wav exists, is 14-30s, and has no silence gap > 0.3s (ffmpeg -af silencedetect).
Step 3 — Translate for dubbing (the agent does this)
This is where dubbing diverges from subtitles. Do not use translations.txt (the subtitle translation) — it follows whisperX's fragment cuts, which split sentences. Dubbing needs complete sentences so the Chinese flows naturally when spoken.
Read <output-root>/transcript/<name>.en.full.srt (the full-sentence English transcript). Translate each cue yourself, writing to transcript/translations_dub.txt — one Chinese line per English cue, line N = cue N.
The char budget rule — the core of dub translation. Each cue's budget is window seconds × normal speech rate, fixed before you write: the window is cue start to the next cue's start (window + following pause), the rate is measured, never assumed — cook dub measure-rate on the pilot takes the user picked (procedure: Step 3a). An assumed rate passes every gate and dubs to seconds of dead air: a 45-min lecture budgeted at an assumed 5.0 chars/s against a real 6.93 left 83 cues with >2.5s of silence and cost an 82-cue re-synthesis. Write to the budget: slightly over is fine, notably under is not — a short take now keeps its natural pace and leaves a breathing pause, so under-budget text means dead air and over-budget text means audible compression. Recalibrate on any reference or duration-factor change: a budget calibrated to a retired reference survives every gate and still dubs to dead air. Synthesize a stratified probe (5-10 typical cues per speaker across length bands) and re-check the script against windows at the measured rates — procedure in REFERENCE.md → "Rate recalibration". How to fill a budget legitimately (and what "filling" is not allowed to invent): REFERENCE.md → "Filling the char budget".
Translation principles (beyond the budget):
- Translate complete thoughts, not fragments. The English is already full sentences; your Chinese cue is one complete thought.
- Keep technical terms in English where Chinese devs do — spec, plan, prototype, agent, token, skill, session, branch, route, etc. See REFERENCE.md → "Term retention list".
- Keep English for anything shown on screen — UI labels, code, URLs, filenames. Examples in REFERENCE.md → "Term retention list".
- Translate concepts that have standard Chinese names when they aren't shown on screen — the worked examples live in the term-retention list.
- Line count must equal cue count. The sanctioned way to change counts is Step 3a's
build_merge.py, which rewrites both files together.
Then generate the pre-assembly SRT (timestamps inherited from en.full.srt):
cook dub make-srt <output-root> <name>
Self-review — two passes, before the mechanical gate: read every line aloud in your head as a spoken sentence (does it sound like something a person would say?); then scan the term list below against every line (anything on-screen stays English, standard concepts go Chinese). These are you checking your own work; the subagent review below is the cold read.
Length gate — mechanical (pure arithmetic, takes a second):
cook dub gate <output-root> <name> --rate <measured chars/s>
It cross-checks every line against its cue window three ways: short lines (≤8 syllables — IndexTTS2's narration-pace trap, rewrite fuller or let Step 3a merge), budget misses (estimated speech far over its window — text must be cut back — or far under — the window gets dead air, fill it), and the coverage total (Σ estimated speech / Σ windows — a uniformly thin translation passes every line check and still dubs to long silences; healthy 90-105%). --rate is required (the measured value; --self-measure re-derives it from existing takes). Exit 1 lists the lines and the coverage verdict; rewrite and rerun until it passes. An over-budget line is fixed by rewriting shorter or folding into its neighbour with cook dub merge — never by leaving it to be atempo-compressed.
Quality gate — fan-out subagent review (mandatory, before synth). The ear gate and this review are the only gates before hours of synthesis. Fan out a subagent with read access to both <name>.en.full.srt and translations_dub.txt, and ask it to check, for every cue:
- Translation accuracy — does the Chinese faithfully convey the English? No dropped clauses, no added content (nothing the English doesn't say), no mistranslations.
- Proper-noun spelling — names (people, products, companies) exactly as the source uses them. "Claude" not "克劳德", "IndexTTS" not "索引TTS", unless a standard Chinese name genuinely exists.
- TTS readability — will IndexTTS2 pronounce this naturally? No awkward character sequences, no orphaned punctuation, no strings that force a robotic reading.