Multi-role / Dialogue Dubbing (Multi-voice Dubbing)
Synthesize "multi-person dialogue / multi-role scripts" into multi-voice audio tracks—each character gets a voice matching their persona, no longer "one voice throughout". The core engine is skills/shared/scripts/multivoice.py: it calls tts.py (free multi-voice via edge-tts) or voice_clone.py (cloud expressive provider / cloned voices) line by line, feeds each line's emotion into the provider's true emotion channel, and uses ffmpeg to concatenate everything into one track while generating aligned speaker-labeled subtitles.
Creativity (who says what, with what emotion) is produced by you, the LLM; voice mapping is in the cast. Deterministic IO (line-by-line synthesis/concatenation/subtitles) is handled by the engine. The output
voice.mp3can be directly used as narration forauto-short-video/assemble.pyor added to any video;voice.srtis a subtitle track with speaker names.
Dubbing Quality Tiers (Important: Cures "flat AI reading")
edge-tts has no emotion engine; it can only adjust speed and pitch, and no matter what you do, it still sounds like a robot. To sound "human", you must use a cloud provider with emotion channels (user brings their own key, no GPU needed):
Engine (engine in cast) | Quality | Requirements | Emotion Mechanism |
|---|---|---|---|
edge (default, free fallback) | ⚠️ Flat, robotic—draft only | No key, external network | Only rate/pitch/volume micro-adjustments |
clone+openai-compatible→SiliconFlow CosyVoice2 (recommended) | Good, natural Chinese | VOICE_API_KEY (cheap / free credits for new users) | Inline `< |
clone+gemini | Good, truly free | GEMINI_API_KEY (Google free tier, requires proxy in China) | Natural language prefix |
clone+minimax / dashscope | Good, emotional | Provider-specific keys | Emotion enumeration / instruct |
Recommended: SiliconFlow (cloud CosyVoice2, no GPU needed, most stable for Chinese). Configure in .env:
VOICE_PROVIDER=openai-compatible
VOICE_BASE_URL=https://api.siliconflow.cn/v1
VOICE_API_KEY=<your key>
VOICE_MODEL=FunAudioLLM/CosyVoice2-0.5B
VOICE_INSTRUCT_MODE=inline
For a character in cast: --engine clone --provider openai-compatible --voice-id FunAudioLLM/CosyVoice2-0.5B:alex (8 voices: alex/anna/benjamin/…).
- Line-by-line emotion auto-drives performance: Each line in
lines.jsonhas anemotion(angry / collapsing in tears / sneering / gentle…) → the engine converts it to the corresponding emotion parameter per provider. The more specific you write, the better the acting. - Want it "human-like" → give at least the main character / key characters a cloud provider (
engine=clone); only with a key will you get emotions; without a key it automatically falls back to edge (flat) with a warning. - Provider configuration is at the top of
voice_clone.py(env vars per provider);voice_clone.py check --provider <name>validates offline that the keys are complete.
Who would use it
Short drama dialogue (short-drama already delegates internally), paper Q&A with two speakers (paper-explainer: speaker + questioner), interview/podcast scripts, audio dramas, any spoken content with multiple speakers. For a single speaker's full narration, use tts-voiceover.
Input
| Field | Required | Description |
|---|---|---|
| cast.json | Yes | Casting table: each speaker → voice (edge voice + pitch/rate, or cloned voice). Includes entries for "Narrator / Main Speaker" |
| lines.json | Yes | Line-by-line dialogue: ordered [{speaker, text, emotion}] (speaker must match a name in cast; emotion e.g. cold/angry/nervous/gentle, automatically matched to tone) |
| Output path | No | voice.mp3 (default agreed with caller); voice.srt with same name |
Execution Steps
1. Casting (cast.json)—Read the casting guide to assign voices
First read skills/shared/references/voice-casting.md (voice ↔ persona archetype mapping + distinguishing same-gender voices + independent narrator + emotional prosody).
Assign a voice matching each speaker's gender/age/temperament/identity. The narrator / main speaker should be separate and different from all characters:
python skills/shared/scripts/multivoice.py cast init --cast <path>/cast.json
python skills/shared/scripts/multivoice.py cast add --cast <path>/cast.json \
--name Lin Ce --role male_lead --voice zh-CN-YunxiNeural --rate=-5% --pitch=-3Hz --note "Cool male lead" # Use = for negative values
python skills/shared/scripts/multivoice.py cast add --cast <path>/cast.json \
--name Su Wan --role female_lead --voice zh-CN-XiaoxiaoNeural --note "Gentle female lead"
python skills/shared/scripts/multivoice.py cast check --cast <path>/cast.json # Validate: voices valid / narrator independent / no voice collisions
cast.json can also be written directly (it's just JSON). If you want a character to sound human with emotion (cures AI flatness) → use a cloud provider for that character:
python skills/shared/scripts/multivoice.py cast add --cast <path>/cast.json \ --name CEO --role male_lead --engine clone --provider minimax --voice-id <your voice id> --note "Cloned / expressive voice"The corresponding provider key must be set in
.env(voice_clone.py check --provider minimax); if the key is missing, that character automatically falls back to edge (flat) with a warning.
2. Line-by-line dialogue (lines.json)
Split the script / dialogue into ordered lines:
{"lines":[
{"speaker":"Presenter","text":"This paper addresses a key problem.","emotion":"neutral"},
{"speaker":"Questioner","text":"Wait, why can't existing methods work?","emotion":"surprised"},
{"speaker":"Presenter","text":"Because they ignore temporal dependencies.","emotion":"firm"}
]}
speaker must match a name in cast (if not, it falls back to the narrator voice with a warning); emotion is optional.
3. Synthesize multi-voice track + subtitles
python skills/shared/scripts/multivoice.py dub \
--cast <path>/cast.json --lines <path>/lines.json -o <path>/voice.mp3
Produces voice.mp3 (each character has an independent voice; emotions automatically modulate prosody) + voice.srt (with speaker names, timed to actual line durations).
dub prints how many voice types were used—confirm ≥2 (a multi-person dialogue should not have only one voice).
4. Use in a video
voice.mp3as narration +voice.srtas subtitles, feed intoauto-short-video/scripts/assemble.py(image/video compositing).- Or mix with BGM (audio-mix), add to an existing video (video-editing).
Fallback
- No external network / edge-tts fails → cannot synthesize; inform the user truthfully (multi-voice depends on edge-tts).
- cast specifies a cloned voice but the key is missing / fails → that character automatically falls back to a free edge voice with a warning; does not block the process.
- Speaker not in cast → fallback to narrator voice with a warning (suggest adding to cast).
Profile Awareness
- With Profile:
style.mdinfluences voice timbre selection (character temperament → voice);preferences.mdfilters lines for redlines. - Without Profile: Use default voice-casting.md mapping.
Rules
- Multi-person dialogue must use multiple voices: each speaker has an independent voice; narrator / presenter is separate—never use one voice throughout (
cast check+dubvoice count guard). - To sound human, use a cloud provider: edge is only for drafts; final / key characters use
engine=clone+provider (with a real emotion channel), no GPU needed. - Voice matches the character: follow voice-casting.md archetype mapping + distinguish same-gender voices with pitch/rate.
- Emotion annotations must be specific: each line in lines should have an emotion (angry / collapsing in tears / sneering / gentle…), which feeds into the provider's emotion channel to drive performance.
- Deterministic persistence: save cast.json / lines.json; to change lines or swap voices, simply re-run
dub—no need to start over.
References
See EASEL-META.md. Multi-voice dubbing (cast + line-by-line lines + per-character voice synthesis + ffmpeg concatenation + speaker subtitles) is self-developed by Easel; underlying wrappers are edge-tts (tts.py) and cloud cloning (voice_clone.py).