Product Film AI Voiceover: From 'A Bit Off' to Professional Commentary Quality
This skill comes from a real rework. A 2-minute 43-second English-language hackathon product film (11 lines of narration synced to visuals) received user feedback that the voiceover "still sounds a bit off." After 9 versions, over a dozen TTS engines and voices, calibrated evaluation with large-sample statistics, the final process uses MOSS-TTS one-shot continuous generation + content-validated sentence splitting + classic post-processing that doesn't damage the voice, earning user approval.
Evaluation methodology in references/evaluation.md, acoustic processing trade-offs in references/acoustic-polish.md, reusable scripts in scripts/.
1. Diagnose What's 'Off': Usually Not Audio Quality, But Prosody
[!IMPORTANT] Modern TTS already surpasses average recordings in audio quality. Both UTMOS and Distill-MOS scores are higher than clean human readings (LibriSpeech), so 'off' almost always comes from prosody. Using public-domain professional commentary (NASA documentaries, vocals separated with Demucs) as a reference, key measured features:
| Feature | Professional Commentary | Sentence-by-Sentence Clone TTS (Initial Version) |
|---|---|---|
| Pitch variation (F0 std, semitones) | 4.6 | 2.7 (flat) |
| End-of-phrase fall (semitones) | 0.7 | 2.3 (sinks at end) |
| Speaking rate (syllables/s) | 4.2 | 4.55 (a bit fast) |
| Average pause (s) | 0.39 | 0.54 |
Use scripts/prosody_compare.py, scripts/end_fall.py, and scripts/take_features.py to compare candidate and reference on the same feature set—rely on data, not impressions.
2. Generation: Read the Full Script in One Go, Don't Stitch Sentences
- Continuous generation first. When generating sentence-by-sentence, each sentence starts its pitch, stress, and breathiness from scratch—stitching them together sounds 'off.' Have the TTS read the entire script at once, with blank lines between sentences. The model sees the full context, so sentences connect naturally. This skill uses MOSS-TTS v1.5 clone mode with a ~15-second reference recording of appropriate tone, generating 4 variants with different random seeds—each ~2 minutes 10 seconds, with 2–6% whole-text character error rate.
- Candidate comparison results (same evaluation):
- MOSS continuous one-shot: prosody distance to professional commentary 0.29 (closest), fewest defects;
- Qwen3-TTS VoiceDesign sentence-by-sentence with performance instructions, multiple tries selected by prosody: distance 0.37, naturalness similar to MOSS;
- OpenAI gpt-audio: distance 0.57;
- Sentence-by-sentence clone: distance 0.89.
- Known pitfalls:
- MOSS-VoiceGenerator (text description of voice) stops early on long text, around 33 seconds—cannot be used for full script generation.
- Gemini TTS requires "AUDIO PROFILE / DIRECTOR'S NOTES / TRANSCRIPT" segmented format, otherwise it reads instructions aloud; free tier: 10 calls/day per model.
- OpenAI voices scored lower on pitch variation in evaluations. If using commercial AI voices, must credit them in the credits.
- Select seed by prosody features: Among seeds with acceptable character error rate, pick the one with smallest distance to professional reference. If one sentence is bad (e.g., the final tagline sinks into a bubbly voice), replace only that sentence with a version from the same reference voice but a different seed.
3. Sentence Splitting: Validate by Content, Don't Trust Timestamps Alone
Continuous audio must be split back into sentences for scene alignment. scripts/align_split.py does:
- Use SenseVoice for word-level timestamps, align to script by edit distance.
- Cut at silence between sentence pairs.
- Validate by content: Try cutting at every silence within 1.5 seconds around the intended split point, re-recognize both halves, and select the position with the lowest sum of character error rates between the two halves.
Relying only on timestamps leads to errors: TTS often pauses mid-sentence (e.g., "Just | sentences.") but reads across sentence boundaries, and word-level timestamps lag actual start times. Without validation, "does | it" and "Just | sentences" would both get split at the wrong place, causing the first word of the next sentence to be cut off.
4. Post-Processing: Only Fix Quantifiable Issues, Don't Touch Pitch
| Processing | Decision |
|---|---|
Normalize pauses based on punctuation (pause_fix_en.py: 0.10s within phrases, 0.25s for commas, 0.33s for colons, 0.45s for sentence ends; keep 0.8s beat for short taglines) | Keep: only shorten pauses, don't lengthen—brings them closer to professional commentary |
Uniform loudness (−20 LUFS), high-pass at 70 Hz, −2 dB at 300 Hz, light de-essing, 6% small-room reverb (vo_post.py) | Keep: audio quality scores don't drop |
| Limiter | Remove: Distill-MOS drops by ~0.14; use linear gain control for peak management |
| WORLD vocoder or Praat PSOLA to raise sentence endings, boost pitch variation | Abandon: Though features got closer to professional, rigorous evaluation scores actually dropped (WORLD processing made things worse—strongest setting: −0.30, confidence interval [−0.61, −0.03]) |
Principle: Fix sentence endings and pitch variation by generating more variants and selecting the best one—don't post-process pitch.
5. Mixing: Don't Let Music 'Pump' Between Sentences
scripts/duck_curve.py: when silence between sentences is shorter than 2.5s, keep music pulled down (−11 dB); start pulling down 0.25s early, recover with 0.9s single-pole smoothing. The original 0.3s sliding average caused music to swell in every pause, making listeners feel like they're gasping. Final mix: −14 LUFS, peak below −1 dBFS; ASR character error rate of the narration in the mix: 0–6.5%.
6. Scene Timing
- If narration doesn't fit the scene, animation segments can be slowed by ~5%, screen recording segments can pause longer—don't compress the voice.
- Dense commas often make TTS sound stiff. Adjust wording without changing meaning, e.g., change "Here it is, live, on real footage" to "running live on real footage."
7. Release Criteria
- Sentence-by-sentence character error rate ≤ 0.08, re-tested in the mix.
- Each sentence separately inspected by Gemini for defects (one audio at a time): no clicks, missing words, mispronunciations, or bubbly sentence endings.
- Prosody distance to professional commentary, and rigorous naturalness test (single-sample, large-sample, bootstrap confidence interval) not worse than the reference—see evaluation doc.
- Final user audition. AI evaluation alone cannot prove it "doesn't sound off"—see next section.
8. Lessons About 'AI Evaluation'
- Gemini pairwise comparisons are unreliable: 86% of the time it selects the first-played audio; reversing order gives opposite conclusion.
- Multi-sample ranking is unstable: Two rounds of ranking on the same set gave contradictory results.
- Scores without reference are untrustworthy: A real amateur recording scored only ~5, lower than TTS's ~6.
- Usable method: Single-sample rigorous tests with professional commentary as reference, combined with large samples and confidence intervals. Even so, "sounds human" classification has high noise (same audio could score 0/8 or 3/8, and even professional actors get classified as AI).
- AI evaluation opinions must be verified: For example, a claim that "the mix is pumping" contradicted direct measurement of gain curves—trust measurements over impressions.