talking-clip

What does talking-clip do?

A clip is four steps: write the line, make the voice, plan the acting, build and render. Output: a HyperFrames project and an mp4 (vertical 1080x1920 by default).

1. line    you write it, in the character's voice
2. voice   scripts/tts.py          (ElevenLabs costs credits; say and none are free)
3. plan    you write plan.json     (gestures and expressions on the words)
4. video   scripts/build_clip.py -> scripts/verify.py -> scripts/render.py

Paths are relative to this package's root: the folder that holds .claude-plugin/, two levels above this file (resolve symlinks first). Requires the public HyperFrames skills (hyperframes, hyperframes-cli): read hyperframes once before editing any composition by hand.

0. Preflight (always first)

Run python3 scripts/setup.py doctor (add --json to parse it). It lists every dependency, what it is for, and the exact install command for this machine; it exits 1 when a required item is missing.

• Required missing (ffmpeg, Node.js 18+, the HyperFrames skills, and the ElevenLabs key or say when that voice is configured): tell the user in one line what each one is for and propose the printed command. Run it only after the user says yes. Never type or ask for an API key yourself: tell the user to export ELEVENLABS_API_KEY in their shell.
• Optional missing: offer them once and say what each unlocks. Playwright gives web captures with element zoom boxes and full-page shots, tesseract (OCR) gives the secrets scan on screenshots, and ASR (mlx-whisper / openai-whisper) gives the audio check and exact word timings. Skip them if the user declines; the matching checks then become manual review steps.
• HyperFrames skills (hyperframes, hyperframes-cli, hyperframes-registry, media-use): if missing, recommend npx hyperframes skills. Read hyperframes (and hyperframes-cli for render problems) before building or editing any composition.

Setup (first run, or when the user wants to change something)

Look for a config: python3 scripts/setup.py show. If there is none, or the user asks to change their character, voice, render or styles, interview them (one short question at a time, offer the default) and then write it with flags:

Question	Flag	Options
Which parts?	--features	character, explainer, or both
Which character?	--character	the bundled example (pkg:characters/host/character.json) or a path to theirs (stored absolute)
Narration language	--language	ISO code; with say a clear voice for it is picked (setup.py voices --lang <code> lists them, --say-voice overrides)
Which voice?	--voice	elevenlabs (their key in ELEVENLABS_API_KEY, their --voice-id), say (macOS, free), none
Credit guards (ElevenLabs)	--max-chars, --max-seconds, --daily-credit-cap	numbers they choose; empty = no guard
How to render?	--render	local (free, this machine), cloud (HyperFrames cloud, needs npx hyperframes auth login), preview (Studio only)
Size	--size	1080x1920, 1920x1080, 1080x1080

python3 scripts/setup.py init --features character --voice say --render local --yes

init runs the doctor first and offers the same installs. If a config already exists it asks before overwriting, or needs --force when not interactive.

Their own character: python3 scripts/setup.py new-character --name "<name>" --out ./characters/<name> --activate copies the example and makes it active (no second init). Then help them fill the keys it prints: description, personality, speaking_style.rules, rig.colors (or sprites), expressions / poses / gestures, set, voice_moods.

Never ask for, print or write the API key. If ELEVENLABS_API_KEY is missing, tell the user to export it in their shell. The config schema is in config/config.schema.json; the character file format is in the README ("Your own character").

Choose a style for this clip

There is no house look. For each clip, pick the style from the topic, the audience and the user's brand or taste: the set ({"type": "color"|"gradient"|"image", ...}), the framing (close, medium, wide) and the motion intensity (calm, normal, lively). Ask the user when the choice matters, for example when the clip is for a brand, a public channel or a series. Vary the style between clips instead of repeating the last one. If the config has a brand kit, its background wins over your choice. Write it to a file and pass --style:

{"set": {"type": "gradient", "colors": ["#FFF4E6", "#FFD9B3"]}, "framing": "medium", "motion": "lively"}

The same goes for the words: match the tone to the audience within the character's personality, and do not reuse the same opener or closing line from one clip to the next.

1. Write the line

Read the character file named in the config (character key): personality and speaking_style.rules are the voice. Follow them; the user's request wins where they conflict.

• Length: at most styles.talking_clip.max_seconds. At speaking_style.chars_per_second (about 14) a 20 s clip is under ~260 characters. One idea, one closing line.
• Language: the user's, unless they ask otherwise.
• ElevenLabs models with audio tags (v3 and later) accept emotion tags in English inside brackets before the words they colour: [excited], [calm], [whispers], [laughs], [pause]. say and none ignore them (they are stripped); act it with gestures instead.

2. Make the voice

python3 scripts/tts.py estimate --text "<line>" --max-seconds 20
python3 scripts/tts.py speak --text "<line>" --out out/voice.wav [--mood excited] [--confirm]

• ElevenLabs: always run estimate first and tell the user the cost; speak refuses without --confirm, and refuses when a guard the user set (max chars, max seconds, daily credit cap) would be broken. --takes 2 makes alternatives. Word timings land in out/voice.wav.words.json.
• --mood picks a mood from the character's voice_moods (ElevenLabs settings + optional ffmpeg filter).
• say / none: free. With none there is no audio; pass --text to the builder instead.

3. Plan the acting

python3 scripts/timings.py out/voice.wav --text "<line>" gives word start times. Write plan.json:

{"expression": "neutral",
 "events": [{"t": 0.0, "gesture": "hello"},
            {"t": 2.1, "gesture": "explain", "dur": 2.5},
            {"t": 5.0, "expression": "happy"},
            {"t": 6.2, "gesture": "wink"}]}

• gesture and expression ids come from the character file (gestures, expressions); each gesture has a description and a typical dur. Read them before planning.
• Mouth, blinks, breathing and idle sway are automatic. You only place the acting beats.
• Place a strong gesture 0.2-0.3 s before the word it goes with. One reaction per beat; never two gestures in the same second. End on a calm, held gesture (it is held to the end automatically).

4. Build, verify, render

python3 scripts/build_clip.py --audio out/voice.wav --plan out/plan.json --style out/style.json --project out/clip
python3 scripts/render.py out/clip --out out/clip.mp4 --quality draft
python3 scripts/verify.py all --project out/clip --text "<line>" --video out/clip.mp4

Verification is required (details in the explainer skill's "Verify" section, same loop): build must be clean, render gives a contact sheet you must look at (mouth moving, gestures where planned, nothing cut off), audio checks the words with ASR and the duration limit. Fix and re-run from the failed stage. Each verify.py all run is one iteration; at verify.max_iterations it exits 2 and you tell the user what still fails instead of shipping. render stays review until you look at the contact sheet and record it with verify.py mark render pass|fail --project out/clip --note "...", then run verify.py all again (a rebuild invalidates the mark).

Then render the final with the configured backend: python3 scripts/render.py out/clip --out out/clip.mp4 (render.max_mb re-encodes to fit an upload limit). Never commit mp4 or wav files.