OmniVoice
Overview
Generate audio locally via the OmniVoice Studio MCP server. Tools: generate_speech, list_voices, list_personalities, list_languages, check_health. Resources: voice://{id}, history://recent.
Prerequisites — Backend Must Be Running
The MCP tools all hit $OMNIVOICE_API_URL (default http://localhost:3900). If the backend is down, every tool returns a connection error. Install + boot:
git clone https://github.com/debpalash/OmniVoice-Studio.git "$OMNIVOICE_HOME"
cd "$OMNIVOICE_HOME"
uv sync
VIRTUAL_ENV="$(pwd)/.venv" uv pip install 'mcp[cli]'
Then:
scripts/check-health.sh # exit 0 if up
scripts/start-backend.sh # boot in background (MPS/CUDA auto-detected)
First synthesis call lazy-downloads the k2-fsa/OmniVoice model (~2.4 GB) from HuggingFace — cached on subsequent boots.
Task Index — Pick the Right Tool
| Task | Tool | Notes |
|---|---|---|
| Verify backend is up | check_health | Returns `{"status":"ok","device":"mps |
| Text → audio with a saved voice | generate_speech(text, profile_id) | Returns base64 WAV. profile_id="demo0001" is the bundled demo voice |
| Text → audio without a clone (voice design) | generate_speech(text, instruct="…") | Omit profile_id; pass an instruct like "warm middle-aged female narrator, calm pace" |
| Multilingual narration | generate_speech(text, language="es") | Any ISO 639 code or "Auto" |
| List existing voices | list_voices | Returns id, name, type, personality |
| List personality presets | list_personalities | Returns narrator / casual / news-anchor / etc. with their instruct strings |
| List supported languages | list_languages | 646 total; returns 20 popular + the full count |
For non-trivial decisions (which engine to use, when to pick OmniVoice over kokoro / Edge TTS / ElevenLabs), see references/engines-comparison.md.
For MCP wiring details, backend lifecycle, troubleshooting, and a clean teardown, see references/mcp-setup.md.
Common Workflows
1. One-shot narration with the demo voice
# As called through the MCP client (your agent will do this for you):
result = generate_speech(
text="Hello — this is OmniVoice generating speech locally.",
profile_id="demo0001",
language="English",
steps=16, # 8 = fast/draft · 16 = balanced · 32 = quality
)
# result is JSON with audio_id, generation_time_s, audio_duration_s, format, wav_base64
Benchmark: 4.2 s of audio in ~24 s server-side on Apple Silicon MPS at 16 diffusion steps.
2. Save the WAV to disk and play
Tool returns base64 PCM WAV (16-bit, mono, 24 kHz). Decode + write:
import base64, json
payload = json.loads(result_text) # parse JSON the tool returns
open("out.wav","wb").write(base64.b64decode(payload["wav_base64"]))
On macOS: afplay out.wav. Convert to MP3 with ffmpeg -i out.wav -codec:a libmp3lame -b:a 128k out.mp3.
3. Voice clone — end-to-end recipe
Cloning needs a 3-10 second reference clip the model will use as a speaker embedding. The MCP server does NOT expose profile creation — it only reads existing profiles. Two paths to create one:
Path A — bundled helper (macOS, recommended for fresh clones):
scripts/record-reference.sh ~/Downloads/my-ref.wav 12 1
# args: output_path raw_duration_sec mic_index
# Default mic_index=1 (MacBook built-in); list devices via:
# ffmpeg -f avfoundation -list_devices true -i ""
The script gives audible countdown + start/stop cues via macOS say + /System/Library/Sounds/Ping.aiff so the user knows when to speak (terminal stdout is buffered — text "speak now" prompts arrive too late). It records a longer raw window, then trims to ~10 seconds of speech via silenceremove + atrim, plays back for verification, and prints the next-step curl command.
Path B — manual:
# 1. Record (mono, 24 kHz native — matches model's internal rate)
ffmpeg -f avfoundation -i ":1" -t 12 -ac 1 -ar 24000 raw.wav
# 2. Trim leading silence + take first 10 sec of speech
ffmpeg -i raw.wav \
-af "silenceremove=start_periods=1:start_silence=0.05:start_threshold=-40dB,atrim=end=10" \
-ac 1 -ar 24000 ref.wav
# 3. Verify
ffmpeg -i ref.wav -af volumedetect -f null - 2>&1 | grep volume # max should be > -20 dB
afplay ref.wav
POST to /profiles (multipart/form-data — required fields: name, ref_audio):
curl -X POST http://127.0.0.1:3900/profiles \
-F "name=carlos-clone" \
-F "[email protected]" \
-F "ref_text=The exact text spoken in the clip" \
-F "language=English" \
| python3 -m json.tool
# returns { "id": "abc12345", "name": "carlos-clone" }
Once created, pass profile_id to generate_speech (via MCP) or directly via POST /generate. Profiles persist in SQLite + reference-audio files at ~/Library/Application Support/OmniVoice/voices/<id>.<ext> (the backend preserves the uploaded extension — .wav if you uploaded a WAV, .mp3 if MP3, etc.). State persists across backend restarts.
Reference clip tips that materially affect quality:
| Factor | Why it matters |
|---|---|
| Single speaker | Mixed speakers blur the embedding |
| Clean speech, no music/noise | Model embeds the noise too |
| Natural prosody (avoid pangrams) | Diffusion samples replicate prosody, not just timbre |
| 3-10 sec is the sweet spot | < 3 s lacks information; > 10 s adds compute without quality gain |
Match ref_text to what's spoken | Improves alignment, especially on noisy refs |
language correct | Wrong language → cross-lingual transfer artifacts |
| Loudness peak ≥ -15 dB | Quiet refs work but normalize poorly |
4. Voice design (no reference clip)
Skip profile_id; provide an instruct string describing the desired voice:
generate_speech(
text="Welcome to the future of agentic systems.",
instruct="warm middle-aged female narrator, calm authoritative pace, documentary style",
)
Get pre-made instructs via list_personalities and copy the one matching the brief (narrator, casual, news-anchor, etc.).
5. Video dubbing (web UI only)
The MCP server does not expose the dubbing endpoint. The full transcribe → translate → re-voice → mux pipeline lives behind the desktop UI (bun run desktop in $OMNIVOICE_HOME) and the /dub/* REST routes. When the user asks to dub a video, point them to the UI; surface this skill only for the synthesis primitives above.
When NOT to use OmniVoice
- Fast English-only narration on weak hardware →
kokoro-ttsis ~10× smaller and 2× realtime on CPU (see references/engines-comparison.md) - Lowest-friction one-off TTS → Edge TTS needs no install or backend
- Highest possible quality regardless of cost → ElevenLabs still wins on English narration polish; OmniVoice ties or wins on multilingual + cloning
- Real-time streaming dictation → use the OmniVoice desktop widget (
⌘+⇧+Space), not the MCP server
Resources
- references/engines-comparison.md — Decision tree across OmniVoice / kokoro / Voicebox / Edge TTS / ElevenLabs / cloud APIs
- references/mcp-setup.md — MCP wiring, backend lifecycle, env vars, troubleshooting
- scripts/check-health.sh —
curl /health, exit 0/1 - scripts/start-backend.sh — Start uvicorn on 127.0.0.1:3900 with health probe
- scripts/stop-backend.sh — Clean shutdown via
kill -TERMon the bound PID - scripts/record-reference.sh — macOS-only: record + trim + verify a reference clip for cloning, with audible cues (
say+ system beeps) that bypass terminal output buffering
Backend Swagger / OpenAPI: http://127.0.0.1:3900/docs (when backend is up).
Upstream: github.com/debpalash/OmniVoice-Studio — FSL-1.1-ALv2 (free for personal/internal/non-commercial; auto-converts to Apache-2.0 two years after each release).