Explain with motion
Result: one MP4 in which the speaker stays visible and the panel below draws what they explain. The explanation stays theirs – the graphics make the structure visible, they do not repeat the words.
Talk to the user in their language, in plain words. They know their content; they may not know video terms. Show them the plan before you cut and before you build, because cutting and building take minutes and their feedback is cheapest early.
All scripts live in scripts/ next to this file. <skill> below means this skill's folder.
0. Check the machine (once)
python3 <skill>/scripts/doctor.py --fetch-model
Needs ffmpeg, Node 20+, whisper.cpp (whisper-cli), Python with numpy. The script names the install
command for anything missing. Everything runs locally; no account is needed (Auphonic is optional, step 8).
1. Intake
Ask only what you cannot infer, ideally in one round:
- the recording (path) and which passage ("where I talk about the customer and the new software").
- who speaks (names, and the role line for the lower third, e.g. "in conversation with …").
- format: 4:5 feed (default), 9:16 reels, 16:9 YouTube.
- look: their brand (colours, fonts, logo, or a website to read them from) or the calm default.
- end card: key sentence, and optional call to action ("More in the course … at …") with logo.
- what the video is for (a post? a course?) – it decides what the graphics should generalise.
2. Transcribe and find the passage
python3 <skill>/scripts/transcribe.py <recording> --work <work> --lang de
Read <work>/transcript.txt and locate the passage. Choose it so it stands on its own: start at the
first sentence of the story (not mid-thought), end on a sentence that lands. Then plan the cuts:
remove stumbles, false starts, long "uh"s, side remarks addressed to the host ("you wrote a great
article about …"), anything that needs the original conversation to make sense. Keep short host
turns when they move the story.
Write <work>/cut.json (format in the header of scripts/cut.py) and show the user the plan in
plain words – where it starts and ends, what is removed and why, resulting length. Cut after OK:
python3 <skill>/scripts/cut.py <work>/cut.json
python3 <skill>/scripts/probe.py seams <work>
Read every seam: is the first and last word complete? Fix doubtful seams with
probe.py env/probe.py say and overrides (see references/pitfalls.md, Cutting), cut again.
3. Captions
Write <work>/captions.txt from the transcript of the cut: wording as spoken, recognition errors
fixed (names, terms, umlauts, punctuation), | between cards (one card = what fits in two short
lines, roughly 4-9 words, broken at phrase boundaries), *word* for the few words worth stressing.
Mark words you could not verify and ask the user about them later instead of guessing silently.
python3 <skill>/scripts/captions.py <work> # add --check for a per-card listening check
Merge cards the script reports as too fast. If merging would make a card longer than two lines, move a few words from the fast card to its neighbour instead of merging it whole.
4. Scene plan (show before building)
Read references/design-rules.md now – the rules there come from real feedback and decide whether the
video feels right. Then plan 12-20 scenes for ~5 minutes (fewer for shorter clips): for each phase of
the story one picture, start/end on phrase boundaries in captions.json, every scene following the
previous one without a gap. references/scene-recipes.md lists metaphors that worked, with working code.
Show the plan as a short table – time, what is said, what the picture shows – and ask for changes. Ask explicitly where the speaker places key items in any model they use (e.g. which zone a thing belongs to): placement is meaning.
5. Build
python3 <skill>/scripts/new_project.py <project> --work <work> --layout 4:5
Then edit in <project>: brand.json, project.json (lower third, footer, end card – formats in
references/brand.md), and the scenes:
src/scenes.html– one<div id="sN" class="clip scene" data-start data-duration data-track-index="3"><div class="sc">…</div></div>per scene, coordinates inside the panel (PANEL_W×PANEL_H)src/scenes.js– animation with the helpersfin, fade, fout, pop, grow, draw, pulse, morph, sceneOut, boil, centre, person, worry, relaxon the shared timelinetl(all defined insrc/template/helpers.js, times in seconds of the clip)src/scenes.css– scene-specific CSS
Start from examples/change-story/ (a complete 5-minute example: read the scene you need, not the whole
file). Rebuild with python3 tools/build.py . – never edit index.html directly, it is generated.
Captions are read from the work dir, so re-running captions.py is picked up by the next build.
Things that are easy to miss:
- Lower third: set
lower_third.startto a moment when the named person is on screen (calls cut to whoever speaks). If the footage has burned-in name labels bottom-left (Zoom, Riverside), use"position": "top-left". - Never an empty panel: something enters within ~0.5 s of every scene start.
- Call
boil(start, end)for every time range with people figures; enlarge figures for tall panels with CSS (e.g.#s2 .p svg{width:160px;height:233px}). - The last scene needs no
sceneOut– the end card fades in over it.
6. Check and look
python3 <skill>/scripts/render.py <project> --snap 10,45,90,… --no-render
Fix every error and contrast warning. Look at the contact sheet(s) – one frame per scene, taken
right after its last element landed. Check against references/design-rules.md §5 (logic of placement)
and references/pitfalls.md. Offer the live preview for the user's own look:
npx hyperframes preview --background in the project (they can click any element and edit it there).
7. Render and verify
python3 <skill>/scripts/render.py <project> --out renders/<name>.mp4
The script reports picture/sound offset (must be 0 ms), whether the video moves, and loudness (target −14 LUFS). Rendering takes about 1–5× the video length depending on the machine; run it in the background for longer videos.
8. Optional: cleaner sound
Only if the user wants it and has an Auphonic key (AUPHONIC_API_KEY):
python3 <skill>/scripts/audio_polish.py <work> # makes equally loud before/after samples
python3 <skill>/scripts/audio_polish.py <work> --apply <render.mp4> # after the user picked "polished"
Never use voice re-synthesis ("studio voice") unless asked – it changes how people sound.
9. Hand over
Tell the user where the MP4 (and captions.srt for platforms that take subtitle files) is, what you
could not verify (uncertain words), and offer one round of changes. Iterations are cheap now: edit
src/scenes.*, rebuild, re-render. Keep earlier renders with a version suffix instead of overwriting.