Communitygithub.com

dub-studio

Dub a video on this machine, translating and re-voicing it while keeping its music and sound effects, with no paid service. Transcribes locally with faster-whisper (or reads an SRT/VTT), has you translate line by line into JSON with a time budget per line, removes the original voice with Demucs (or a fallback), voices each line with a built-in Kokoro synthetic voice, fits lines to their slots (speed-up, then pauses, else a flag), remixes at a set loudness (-16 LUFS default) and writes an MP4, bilingual SRT/VTT, a timing report and a sheet. Use when the user wants to dub, translate, re-voice or localise a video or voice-over into Spanish, French, Italian, Brazilian Portuguese, Hindi or English, or wants a "Spanish version" of a video. Only for videos the user owns, has rights to, or whose voice is already synthetic; never clones a voice. Malay, Japanese, German and other languages without a working Kokoro voice are not supported.

dub-studio란 무엇인가요?

dub-studio is a Claude Code agent skill that dub a video on this machine, translating and re-voicing it while keeping its music and sound effects, with no paid service. Transcribes locally with faster-whisper (or reads an SRT/VTT), has you translate line by line into JSON with a time budget per line, removes the original voice with Demucs (or a fallback), voices each line with a built-in Kokoro synthetic voice, fits lines to their slots (speed-up, then pauses, else a flag), remixes at a set loudness (-16 LUFS default) and writes an MP4, bilingual SRT/VTT, a timing report and a sheet. Use when the user wants to dub, translate, re-voice or localise a video or voice-over into Spanish, French, Italian, Brazilian Portuguese, Hindi or English, or wants a "Spanish version" of a video. Only for videos the user owns, has rights to, or whose voice is already synthetic; never clones a voice. Malay, Japanese, German and other languages without a working Kokoro voice are not supported.

지원 대상~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/VinVirtual/dub-studio/tree/HEAD

즐겨 사용하는 AI에게 물어보기

이 에이전트 스킬이 미리 로드된 새 채팅을 엽니다.

문서

dub-studio은(는) 무엇을 하나요?

Pipeline: video -> audio -> faster-whisper (or the user's subtitles) -> timed source lines -> your translation (JSON, one line each, within a time budget) -> Demucs removes the voice -> Kokoro voices each translated line -> time-fit (atempo, pauses, flags) -> mix with the background -> loudness -> MP4 + subtitles + report.

Skill folder (written $D below): this folder. Every command is $D/dub <command> ...; $D/dub help lists them.

Rules (read first)

  • Rights. --rights is the user's own declaration; the tool records it but cannot check it. Only dub a video the user made or owns (--rights own), has written permission or a licence to dub (licensed), or whose speech is already a synthetic voice they may replace (synthetic). Ask if it is not clear. If the video shows or voices a real person who has not agreed, do not dub it. Never make anyone appear to say something they did not say.
  • No cloning. The new voice is always one of Kokoro's built-in voices, chosen by name. Never try to imitate the original speaker or any real person, and never pass a recording as a voice (the tool refuses).
  • Label it. The output MP4's metadata, the VTT files and out/<lang>/README.md say the dub uses a synthetic voice. Keep that label when you hand the files over, and tell the user to keep it when they publish.
  • Translation quality. You write the translation. Say that it is machine translation, and suggest a fluent speaker checks it before it is published.

Languages

Run $D/dub languages for the live list. Measured on this machine (Kokoro v1.0 through kokoro-onnx, espeak-ng phonemes), checked by reading the voice back with speech recognition:

codelanguagestatus
en-us, en-gbEnglishsupported
esSpanishsupported
fr-frFrenchsupported
itItaliansupported
pt-brBrazilian Portuguesesupported
hiHindifair: understandable, some words wrong
cmnMandarin Chineseexperimental: most of it is not understood when read back; refused without --experimental
jaJapanesedoes not work with kokoro-onnx (kanji are read as "Chinese letter")
msMalaynot supported: Kokoro has no Malay voice

Anything else (German, Korean, Indonesian, Arabic...) is not supported: say so plainly and do not substitute another language's voice. Any language faster-whisper knows can be the source.

Workflow

  1. $D/dub check. Fix any MISS line (check_env.sh prints the commands). Demucs is optional; without it the background falls back (see below).
  2. Confirm the rights and the target languages with the user.
  3. $D/dub prepare <video> --out <project> --to es,fr-fr --rights own [--from en]
    • Add --srt subs.srt (or .vtt) with --from <lang> when the user has a correct script or subtitles: it skips speech recognition and uses those cue times as the lines.
    • The input must have a video stream and an audio track. --from must be a plain language code (en, pt-BR).
    • Running prepare again on the same folder is refused, because source.json may hold corrections. --force replaces source.json and starts the translation files again.
    • Without it, faster-whisper small runs on the CPU (--model base is faster, medium more accurate). Read source.json and fix any misheard words in it before translating (keep ids and times).
    • --voice es=em_alex picks a voice; --max-tempo 1.2 lowers the speed-up limit; --target -23 sets broadcast loudness.
  4. Translate. Open each translations/<lang>.json. For every line, write the translation of source in text:
    • One line in, one line out: never merge, split, reorder or drop lines, never change id.
    • Stay within budget.ideal characters (fits at normal speed). budget.max is the hard limit (spoken at the top speed-up). Shorter is better than faster: drop filler words, use the shortest natural phrasing.
    • Translate meaning, not words. Spoken, natural register. Keep names, product names and numbers. Do not add facts.
    • Plain text only: no brackets, stage directions, markup or emoji.
  5. $D/dub lint <project>: fix every ERROR (too long, empty, missing). Read the WARNings (long lines borrow the pause after them; dropped numbers; untranslated lines) and shorten where you can.
  6. $D/dub build <project> [--separation ...]: background, voice, fit, mix, loudness, mux. Prints each language's MP4, loudness and how many lines were sped up, shifted or flagged. A FLAG line means the translation is too long even with the pauses: shorten it and build again (voices are cached, only changed lines are re-voiced). Lines it pushed late are reported as delayed, not flagged: fix the flagged line, not them.
  7. Look at it. Read out/<lang>/sheet.png: the dub voice row should sit in the source line slots; blue (shift) is fine, red (overflow) and purple (delayed) are not. Read out/<lang>/report.md.
  8. $D/dub verify <project> --asr: checks length, loudness (measured with ffmpeg's ebur128: within 0.5 LU of the target, true peak at most 0.5 dB over the ceiling, since AAC encoding can add a little), flags and the synthetic-voice label in the MP4, and reads the dub back with speech recognition. You cannot hear the result: this read-back score is how you check the voice says the right words. Lines under 60% are listed; look at them.
  9. Hand over: the MP4s, the subtitle files and the report paths, the measured loudness, the read-back scores, and the label sentence.

What goes where

  • dub.json: settings and the rights declaration. fit.maxTempo (1.25), fit.gap (0.12 s between lines), fit.maxEarly (0.3 s a line may start early), loudness.target (-16 LUFS) and ceiling (-1.5 dBTP), mix.background / mix.voice (dB), mix.duck (dB, for the duck fallback), keepOriginal (true keeps the original audio as a second, non-default track), separation.
  • source.json: the timed source lines. translations/<lang>.json: your translations.
  • out/<lang>/: <name>.<lang>.mp4, <name>.<lang>.srt|vtt (dub only), <name>.<lang>+<src>.srt|vtt (bilingual), report.md|json, sheet.png, README.md.
  • work/: cache (extracted audio, background, voiced lines), keyed to the exact source file, the background method and the duck level. Safe to delete; verify --asr then needs dub build again, because it reads work/<lang>/voice.wav.

Background (music and effects)

separation in dub.json or --separation:

  • demucs (default when installed): htdemucs on the CPU, about 15 to 25 s for a 20 s clip here. Best result. The weights are downloaded from a pinned Hugging Face revision as safetensors and checked against pinned SHA-256 values; demucs' pickle fallback is never used.
  • center: removes what is panned to the middle (usually the voice). Only for real stereo; it also removes centred bass and mono instruments.
  • duck: keeps the original audio and turns it down (mix.duck, -18 dB) while the original voice speaks. The original voice stays faintly audible: tell the user.
  • none: voice only. auto picks demucs, else centre (stereo with real side content), else duck, and the report says which was used.

Troubleshooting

  • "too long" errors: shorten the line; do not raise maxTempo above about 1.3, it sounds rushed.
  • Lines drift late (shift s grows in the report): an earlier line overflowed; shorten it.
  • Words misheard in source.json: edit the text there directly (keep ids and times) and do not rerun prepare (that refuses, and --force would replace your edits and the translations). dub lint then warns on each translation whose source no longer matches: update those translations.
  • The video has burned-in text or captions in the source language: they stay. Say so; the dub only replaces the sound.
  • Mouths on screen will not match: this is a voice-over style dub, not lip-sync.
  • dub prepare refuses: read the message (rights, language, missing audio, bad subtitles) and fix that; do not work around the rights check.

관련 스킬