What does dub-studio do?
Pipeline: video -> audio -> faster-whisper (or the user's subtitles) -> timed source lines -> your translation (JSON, one line each, within a time budget) -> Demucs removes the voice -> Kokoro voices each translated line -> time-fit (atempo, pauses, flags) -> mix with the background -> loudness -> MP4 + subtitles + report.
Skill folder (written $D below): this folder. Every command is $D/dub <command> ...; $D/dub help lists them.
Rules (read first)
- Rights.
--rightsis the user's own declaration; the tool records it but cannot check it. Only dub a video the user made or owns (--rights own), has written permission or a licence to dub (licensed), or whose speech is already a synthetic voice they may replace (synthetic). Ask if it is not clear. If the video shows or voices a real person who has not agreed, do not dub it. Never make anyone appear to say something they did not say. - No cloning. The new voice is always one of Kokoro's built-in voices, chosen by name. Never try to imitate the original speaker or any real person, and never pass a recording as a voice (the tool refuses).
- Label it. The output MP4's metadata, the VTT files and
out/<lang>/README.mdsay the dub uses a synthetic voice. Keep that label when you hand the files over, and tell the user to keep it when they publish. - Translation quality. You write the translation. Say that it is machine translation, and suggest a fluent speaker checks it before it is published.
Languages
Run $D/dub languages for the live list. Measured on this machine (Kokoro v1.0 through kokoro-onnx, espeak-ng phonemes), checked by reading the voice back with speech recognition:
| code | language | status |
|---|---|---|
en-us, en-gb | English | supported |
es | Spanish | supported |
fr-fr | French | supported |
it | Italian | supported |
pt-br | Brazilian Portuguese | supported |
hi | Hindi | fair: understandable, some words wrong |
cmn | Mandarin Chinese | experimental: most of it is not understood when read back; refused without --experimental |
ja | Japanese | does not work with kokoro-onnx (kanji are read as "Chinese letter") |
ms | Malay | not supported: Kokoro has no Malay voice |
Anything else (German, Korean, Indonesian, Arabic...) is not supported: say so plainly and do not substitute another language's voice. Any language faster-whisper knows can be the source.
Workflow
$D/dub check. Fix any MISS line (check_env.shprints the commands). Demucs is optional; without it the background falls back (see below).- Confirm the rights and the target languages with the user.
$D/dub prepare <video> --out <project> --to es,fr-fr --rights own [--from en]- Add
--srt subs.srt(or.vtt) with--from <lang>when the user has a correct script or subtitles: it skips speech recognition and uses those cue times as the lines. - The input must have a video stream and an audio track.
--frommust be a plain language code (en,pt-BR). - Running
prepareagain on the same folder is refused, becausesource.jsonmay hold corrections.--forcereplacessource.jsonand starts the translation files again. - Without it, faster-whisper
smallruns on the CPU (--model baseis faster,mediummore accurate). Readsource.jsonand fix any misheard words in it before translating (keep ids and times). --voice es=em_alexpicks a voice;--max-tempo 1.2lowers the speed-up limit;--target -23sets broadcast loudness.
- Add
- Translate. Open each
translations/<lang>.json. For every line, write the translation ofsourceintext:- One line in, one line out: never merge, split, reorder or drop lines, never change
id. - Stay within
budget.idealcharacters (fits at normal speed).budget.maxis the hard limit (spoken at the top speed-up). Shorter is better than faster: drop filler words, use the shortest natural phrasing. - Translate meaning, not words. Spoken, natural register. Keep names, product names and numbers. Do not add facts.
- Plain text only: no brackets, stage directions, markup or emoji.
- One line in, one line out: never merge, split, reorder or drop lines, never change
$D/dub lint <project>: fix every ERROR (too long, empty, missing). Read the WARNings (long lines borrow the pause after them; dropped numbers; untranslated lines) and shorten where you can.$D/dub build <project> [--separation ...]: background, voice, fit, mix, loudness, mux. Prints each language's MP4, loudness and how many lines were sped up, shifted or flagged. AFLAGline means the translation is too long even with the pauses: shorten it and build again (voices are cached, only changed lines are re-voiced). Lines it pushed late are reported asdelayed, not flagged: fix the flagged line, not them.- Look at it. Read
out/<lang>/sheet.png: the dub voice row should sit in the source line slots; blue (shift) is fine, red (overflow) and purple (delayed) are not. Readout/<lang>/report.md. $D/dub verify <project> --asr: checks length, loudness (measured with ffmpeg's ebur128: within 0.5 LU of the target, true peak at most 0.5 dB over the ceiling, since AAC encoding can add a little), flags and the synthetic-voice label in the MP4, and reads the dub back with speech recognition. You cannot hear the result: this read-back score is how you check the voice says the right words. Lines under 60% are listed; look at them.- Hand over: the MP4s, the subtitle files and the report paths, the measured loudness, the read-back scores, and the label sentence.
What goes where
dub.json: settings and the rights declaration.fit.maxTempo(1.25),fit.gap(0.12 s between lines),fit.maxEarly(0.3 s a line may start early),loudness.target(-16 LUFS) andceiling(-1.5 dBTP),mix.background/mix.voice(dB),mix.duck(dB, for the duck fallback),keepOriginal(true keeps the original audio as a second, non-default track),separation.source.json: the timed source lines.translations/<lang>.json: your translations.out/<lang>/:<name>.<lang>.mp4,<name>.<lang>.srt|vtt(dub only),<name>.<lang>+<src>.srt|vtt(bilingual),report.md|json,sheet.png,README.md.work/: cache (extracted audio, background, voiced lines), keyed to the exact source file, the background method and the duck level. Safe to delete;verify --asrthen needsdub buildagain, because it readswork/<lang>/voice.wav.
Background (music and effects)
separation in dub.json or --separation:
demucs(default when installed): htdemucs on the CPU, about 15 to 25 s for a 20 s clip here. Best result. The weights are downloaded from a pinned Hugging Face revision as safetensors and checked against pinned SHA-256 values; demucs' pickle fallback is never used.center: removes what is panned to the middle (usually the voice). Only for real stereo; it also removes centred bass and mono instruments.duck: keeps the original audio and turns it down (mix.duck, -18 dB) while the original voice speaks. The original voice stays faintly audible: tell the user.none: voice only.autopicks demucs, else centre (stereo with real side content), else duck, and the report says which was used.
Troubleshooting
- "too long" errors: shorten the line; do not raise
maxTempoabove about 1.3, it sounds rushed. - Lines drift late (
shift sgrows in the report): an earlier line overflowed; shorten it. - Words misheard in
source.json: edit the text there directly (keep ids and times) and do not rerunprepare(that refuses, and--forcewould replace your edits and the translations).dub lintthen warns on each translation whosesourceno longer matches: update those translations. - The video has burned-in text or captions in the source language: they stay. Say so; the dub only replaces the sound.
- Mouths on screen will not match: this is a voice-over style dub, not lip-sync.
dub preparerefuses: read the message (rights, language, missing audio, bad subtitles) and fix that; do not work around the rights check.