Video Localization: voice-cloned Chinese dub

Replace a video's original English vocals with Chinese voiceover on an identity timeline: the video is never re-timed — every frame keeps its original timestamp and the dubbed release has exactly the raw video's duration. Each cue's synthesized audio is fitted into its own original window (cue start → next cue's start) with a per-cue atempo; faster or slower synthesis both land in the window. The result is a second release — same picture at the same speed, Chinese audio, bilingual ZH+EN subtitles burned in.

This skill does the two creative parts the CLI can't: translating for dubbing (complete sentences written to a char budget, not the subtitle fragmentation) and running the quality gates (length gate, cold review, ear gate, post-burn review). Deterministic execution (Demucs, IndexTTS2, assembly, retime, ffmpeg) is handled by the cook CLI's cook dub subcommand. All stage scripts live inside the cook package (stages/dub/); every invocation below is a cook command. pip install -U video-cook once upgrades everything.

When to reach for this skill

You have a video that already has:

• A raw video file (<output-root>/raw/<name>.raw.mp4) — the original, with English vocals.
• A bilingual subtitle run from video-subtitle — specifically transcript/<name>.en.full.srt (the full-sentence English transcript, merged from whisperX fragments) and transcript/translations.txt.

You want a Chinese-dubbed release. If you don't have these yet, run video-download then video-subtitle first — this skill reads their outputs.

What you produce

Three products ship: cooked/<name>.dubbed.mp4 (raw video untouched, Chinese dub, burned bilingual subtitles — duration identical to the raw), cloud-srt/zh.dub.srt + en.dub.srt (upload subtitles). Everything else is working files under dubbed/ — the annotated tree below maps every one of them.

The run is not done until Step 6's checks pass.

Directory layout

This skill adds dubbed/ (working directory) and writes the final products to cooked/ and cloud-srt/:

<output-root>/
├── raw/                            ← from video-download (this skill reads it)
│   └── <name>.raw.mp4
├── transcript/                     ← from video-subtitle (this skill reads + adds)
│   ├── <name>.en.full.srt          ← full-sentence English (the dub script source)
│   ├── translations_dub.txt        ← this skill writes: one Chinese line per cue
│   └── <name>.zh.dub.srt           ← this skill writes
├── cooked/                         ← final videos live here
│   ├── <name>.cooked.bar.mp4       ← from video-subtitle (untouched)
│   └── <name>.dubbed.mp4           ← this skill's product
├── cloud-srt/                      ← upload subtitles live here
│   ├── zh.srt / en.srt             ← from video-subtitle (untouched)
│   └── zh.dub.srt / en.dub.srt     ← this skill's upload subtitles
└── dubbed/                         ← this skill's working directory
    ├── _reference/
    │   └── ref.wav
    └── _full/
        ├── _segments/              ← per-cue IndexTTS2 cache
        ├── dub.wav                 ← the dub on the original clock
        ├── dubbing.srt             ← working file (ZH, pre-shorten)
        ├── dubbing.short.srt       ← working file (ZH, shorten output)
        ├── dubbing.merged.srt      ← working file (ZH, post-shorten; copied to cloud-srt)
        ├── dubbing.en.srt          ← working file (full-sentence EN)
        ├── dubbing.en.short.srt / dubbing.en.merged.srt  ← its shorten+merge-short outputs
        ├── dubbing.bilingual.srt   ← working file (biliteral union; what gets burned)
        ├── dubbing.cooked.ass     ← working file (ass output, copied to burn.ass)
        └── burn.ass                ← the ASS actually burned

Rule: dubbed/ is the working directory; cooked/<name>.dubbed.mp4 and cloud-srt/{zh,en}.dub.srt are the products. Never touch raw/, transcript/<name>.zh.srt, cooked/<name>.cooked.mp4, or cloud-srt/{zh,en}.srt — those belong to video-subtitle. If this skill fails halfway, the bilingual cooked shipment is still complete.

The pipeline

Two stages after the creative work: synth then assemble, invoked through cook:

cook dub synth    <root> <name> --python <indextts-venv>/Scripts/python.exe
cook dub assemble <root> <name> --python <indextts-venv>/Scripts/python.exe [--keep-subs]
# or both:
cook dub full <root> <name> --python <indextts-venv>/Scripts/python.exe

The steps below describe what each stage does internally (so you can verify outputs and diagnose failures); the cook dub <stage> commands are how you run them.

Step 0 — Resolve the environments

There are two Python environments in play, and confusing them is the failure mode this step exists to prevent:

• cook's environment (system Python or ~/.venvs/video-tools/) — where the cook CLI lives, with whisperX/yt-dlp/torch for the subtitle pipeline.
• IndexTTS2's environment (~/Git/index-tts/.venv) — a separate venv holding indextts, torch, demucs, and whisperx (the dub pipeline adds the last two on top of upstream's lockfile). These deps are heavy and isolated on purpose; do not try to install them into cook's environment.

cook runs each dub stage as a subprocess under the IndexTTS2 venv via --python, so from indextts import ... resolves there. Every cook dub command in this skill takes --python <indextts-venv>/Scripts/python.exe. Resolve the venv path once (default ~/Git/index-tts/.venv) and reuse it for the whole run.

0a. Probe both environments are reachable:

<cook-venv>/Scripts/cook doctor                                    # whisperX/yt-dlp/ffmpeg
<indextts-venv>/Scripts/python -c "from indextts.infer_v2_5 import IndexTTS2; import demucs, whisperx; print('ok')"   # indextts (v2.5) + demucs + whisperx

0b. Single-thread constraint. IndexTTS2 must run single-threaded (OMP_NUM_THREADS=1), or it produces garbage audio. cook sets this internally before importing torch, so you don't need to export it yourself.

Done when cook's doctor reports whisperX/yt-dlp/ffmpeg installed, the IndexTTS2 venv imports indextts (v2.5) + demucs + whisperx, and you know the absolute path to <indextts-venv>/Scripts/python.exe to pass as --python.

Step 1 — Separate vocals from the raw video

The original audio is one mixed track (vocals + BGM + SFX). Demucs splits it so we can extract a clean reference and check for BGM later.

cook dub separate <output-root> <name> [--model htdemucs] --python <indextts-venv>/Scripts/python.exe

Demucs lives in the IndexTTS2 venv, so --python points there. Use htdemucs (single model, ~3GB RAM), not htdemucs_ft (bag of 4 models, ~20GB RAM — OOMs on 32GB machines). Quality is slightly lower but adequate for reference extraction.

In foreground mode (default), cook moves the separated stems to dubbed/vocals.wav + dubbed/no_vocals.wav automatically; in --detach mode you move them yourself after the done marker appears.

Done when dubbed/vocals.wav AND dubbed/no_vocals.wav both exist with duration matching raw ±0.5s.

Step 2 — Extract the reference clip

IndexTTS2 needs a 14-30 second clean clip of the original speaker — it reads at most the first 15s, so a full-cap window beats a short one. IndexTTS2 clones prosody, not just timbre: the window's delivery pace becomes the whole dub's delivery pace. The default picker takes the longest continuous speech, which tends to select the slowest, most deliberate passage — often wrong. Prefer the scanner:

cook dub scan-reference <output-root> <name> \
    [--full-band] [--python <indextts-venv>/Scripts/python.exe]

It ranks fixed 15s windows by syllable rate (density- and pause-filtered) and cuts the top 3 candidates (plus full-band 48k copies with --full-band). Density is only half the story — IndexTTS2 clones the window's prosody too, so rank candidates by liveliness as well: pyin F0 std and semitone range per window (a flat window dubs flat; use librosa.pyin, NOT detect_pitch_frequency, which reports voicing on silence). Authenticate any legacy reference file before reusing it: a reprocessed derivative keeps its speaker embedding and pitch statistics but loses waveform correlation, so correlation near zero does NOT mean "different speaker" — score it against an anchor clip with cook dub scan-reference --score <file> --anchor <anchor.wav> (same-speaker real audio lands near 0.94; synthesized takes 0.66-0.76 is the normal band) and compare pyin stats. Either way: the user's ear picks the reference, always — synthesize the pilot sentences once per candidate (Step 3a), prescreen each take with a small-model ASR round-trip (dropped words flag garble; traditional-Chinese output flags an accent the window bleeds into the take), score with cook dub scan-reference --score, then hand the loudness-matched wavs to the user.

Done when dubbed/_reference/ref.wav exists, is 14-30s, and has no silence gap > 0.3s (ffmpeg -af silencedetect).

Step 3 — Translate for dubbing (the agent does this)

This is where dubbing diverges from subtitles. Do not use translations.txt (the subtitle translation) — it follows whisperX's fragment cuts, which split sentences. Dubbing needs complete sentences so the Chinese flows naturally when spoken.

Read <output-root>/transcript/<name>.en.full.srt (the full-sentence English transcript). Translate each cue yourself, writing to transcript/translations_dub.txt — one Chinese line per English cue, line N = cue N.

The char budget rule — the core of dub translation. Each cue's budget is window seconds × normal speech rate, fixed before you write: the window is cue start to the next cue's start (window + following pause), the rate is measured, never assumed — cook dub measure-rate on the pilot takes the user picked (procedure: Step 3a). An assumed rate passes every gate and dubs to seconds of dead air: a 45-min lecture budgeted at an assumed 5.0 chars/s against a real 6.93 left 83 cues with >2.5s of silence and cost an 82-cue re-synthesis. Write to the budget: slightly over is fine, notably under is not — a short take now keeps its natural pace and leaves a breathing pause, so under-budget text means dead air and over-budget text means audible compression. Recalibrate on any reference or duration-factor change: a budget calibrated to a retired reference survives every gate and still dubs to dead air. Synthesize a stratified probe (5-10 typical cues per speaker across length bands) and re-check the script against windows at the measured rates — procedure in REFERENCE.md → "Rate recalibration". How to fill a budget legitimately (and what "filling" is not allowed to invent): REFERENCE.md → "Filling the char budget".

Translation principles (beyond the budget):

• Translate complete thoughts, not fragments. The English is already full sentences; your Chinese cue is one complete thought.
• Keep technical terms in English where Chinese devs do — spec, plan, prototype, agent, token, skill, session, branch, route, etc. See REFERENCE.md → "Term retention list".
• Keep English for anything shown on screen — UI labels, code, URLs, filenames. Examples in REFERENCE.md → "Term retention list".
• Translate concepts that have standard Chinese names when they aren't shown on screen — the worked examples live in the term-retention list.
• Line count must equal cue count. The sanctioned way to change counts is Step 3a's build_merge.py, which rewrites both files together.

Then generate the pre-assembly SRT (timestamps inherited from en.full.srt):

cook dub make-srt <output-root> <name>

Self-review — two passes, before the mechanical gate: read every line aloud in your head as a spoken sentence (does it sound like something a person would say?); then scan the term list below against every line (anything on-screen stays English, standard concepts go Chinese). These are you checking your own work; the subagent review below is the cold read.

Length gate — mechanical (pure arithmetic, takes a second):

cook dub gate <output-root> <name> --rate <measured chars/s>

It cross-checks every line against its cue window three ways: short lines (≤8 syllables — IndexTTS2's narration-pace trap, rewrite fuller or let Step 3a merge), budget misses (estimated speech far over its window — text must be cut back — or far under — the window gets dead air, fill it), and the coverage total (Σ estimated speech / Σ windows — a uniformly thin translation passes every line check and still dubs to long silences; healthy 90-105%). --rate is required (the measured value; --self-measure re-derives it from existing takes). Exit 1 lists the lines and the coverage verdict; rewrite and rerun until it passes. An over-budget line is fixed by rewriting shorter or folding into its neighbour with cook dub merge — never by leaving it to be atempo-compressed.

Quality gate — fan-out subagent review (mandatory, before synth). The ear gate and this review are the only gates before hours of synthesis. Fan out a subagent with read access to both <name>.en.full.srt and translations_dub.txt, and ask it to check, for every cue:

1. Translation accuracy — does the Chinese faithfully convey the English? No dropped clauses, no added content (nothing the English doesn't say), no mistranslations.
2. Proper-noun spelling — names (people, products, companies) exactly as the source uses them. "Claude" not "克劳德", "IndexTTS" not "索引TTS", unless a standard Chinese name genuinely exists.
3. TTS readability — will IndexTTS2 pronounce this naturally? No awkward character sequences, no orphaned punctuation, no Chinese words that look like English words to the TTS (e.g., homographs).
4. Budget fit — is each line at or slightly under its char budget? (Over-budget lines must be flagged for revision, not left for atempo compression.)

Collect the review as a structured report (pass/fail per cue, with citations), fix every violation, re-run the gate, and only then proceed to synth.

Step 3a — Build the pilot (only after the user picks a reference)

The pilot is a 10-15 cue subset: stratified by window length (short / medium / long), spread across the video, and including the video's first line. It verifies the translation reads naturally, the reference works, and the char rate is real.

cook dub pilot <output-root> <name> --n <12> --python <indextts-venv>/Scripts/python.exe

This synthesizes each pilot cue with IndexTTS2, measures the actual speaking rate (chars/s) on the output, and writes them into the full project's segment cache so they aren't re-synthesized. The user listens to the pilot; if they reject the voice quality or delivery pace, pick a new reference (back to Step 2) and re-run the pilot.

If the user is happy, you now have a measured rate. Feed this to the length gate (--rate) in Step 3 above — if you had already run the gate with an assumed rate, re-run it with the measured one and adjust any lines that now miss budget.

At this point the pilot vocab-cache is warm; inside dubbed/_full/_segments/ you'll see one subdir per pilot cue with a .wav and .npy file. The full synth (Step 4) will reuse these.

Step 4 — Full synthesis

cook dub synth <output-root> <name> --python <indextts-venv>/Scripts/python.exe

This synthesizes every cue that doesn't already have a segment in _segments/, then stitches them into dubbed/_full/dub.wav on the original timeline with per-cue atempo. Skipped cues = re-synthesis avoided: each completed pilot cue saves ~15-45s of inference.

Done when dubbed/_full/dub.wav exists and dubbed/_full/dub.wav duration matches raw/<name>.raw.mp4 ±0.1s (tempo-adjusted tracks are frame-accurate; a larger mismatch signals a dropped or duplicated cue — run the ear gate below to find it).

Step 5 — Assemble the release

cook dub assemble <output-root> <name> --python <indextts-venv>/Scripts/python.exe [--keep-subs]

This:

1. Shortens the Chinese SRT so each line fits its window.
2. Merges short adjacent cues.
3. Burns the bilingual subtitles into the raw video (no re-encode of the picture track).
4. Writes cooked/<name>.dubbed.mp4 and cloud-srt/zh.dub.srt + en.dub.srt.

--keep-subs leaves working subtitle files in dubbed/_full/ for inspection.

Step 6 — Quality gate (mandatory, before shipping)

Before marking the run done, run these checks in order:

1. Ear gate — watch the full video at 2-3× speed, listening for: dropped cues (silence where speech should be), tempo artifacts (squeaky-chipmunk or mud), wrong-language bleed-through (Demucs leaked the BGM into the vocals), and any cue where the Chinese doesn't match the English line. Flag any issues and fix (re-synthesize individual cues via cook dub synth --cue <N> or fix the translation and re-run Step 4-5).

2. Character gate — every zh.dub.srt line: is each character actually a Chinese character (CJK Unified Ideographs range), proper noun in expected script, or legitimate standard punctuation? No Latin letters, no half-width forms, no invisible Unicode — except for proper nouns explicitly retained per the term list above.

3. Sync gate — confirm cooked/<name>.dubbed.mp4 duration == raw/<name>.raw.mp4 duration (ffprobe both, diff the two durations — should be identical). Check a few spot timestamps: the dubbed waveform's cue boundaries should align with the English SRT's timestamps.

4. Post-burn review — watch the burned release at normal speed for 2-3 minutes: subtitles legible? Voiceover synced to speaker's mouth movements (as much as lip-sync allows)? No subtitle-over-subtitle collisions?

5. Ship-check — verify that:

   • cooked/<name>.dubbed.mp4 exists (the product)
   • cloud-srt/zh.dub.srt exists (the upload subtitle for ZH)
   • cloud-srt/en.dub.srt exists (the upload subtitle for EN)
   • All three files are non-zero size
   • cooked/<name>.dubbed.mp4 plays back correctly (at least 5 seconds of playback in ffplay)

If all checks pass, the run is done.