JacobLinCool/video-studio-mlx

Studio Demo Video

Build a ~90–180s narrated demo of whatever product the user names — a macOS app, a website, a CLI, anything you can screen-record. The only fixed piece is the narration backend: the video-studio-mlx Studio API (POST /studio/tts), which clones a voice and validates each line. The subject of the demo is arbitrary and changes every time. Keep that separation in mind: don't assume the demo target is video-studio-mlx itself (that's just the example in references/).

Core principle: narration-first, exact-duration anchoring

Generate the narration before any footage. /studio/tts returns each segment's exact duration and per-word timestamps — those drive the whole timeline and every focus effect. Then record footage generously and fit each clip to its segment. Cutting first and voicing later forces fragile re-timing.

You usually cannot play or hear the output. Treat that as a hard constraint: verify the video by extracting frames, and verify audio content/sync by running ASR on slices of the finished file. Build everything so it can be checked this way.

Pipeline

1. Write a per-scene storyboard — narration text, on-screen action, caption, and (per scene) which elements the narration names so focus can point at them. Aim ~2.1 words/sec (≈170 words ≈ 80s). Copy references/storyboard.example.json to <work>/storyboard.json. Word the narration so each named element is actually visible and in order in that scene's footage. Write for the ear: spell words the way they're spoken, not written — expand abbreviations and initialisms ("regular expression" not "regex", "version three" not "v3"), avoid bare symbols (~, /, &), and write numbers/units as words when unsure. The TTS reads the literal text and ASR validates against it, so written-only shorthand both mispronounces and skews WER.
2. Generate narration — scripts/generate_narration.py <storyboard.json>. POSTs each scene to /studio/tts, cloning every scene from one locked voice (the user's ref_audio, or a voice instruct it designs once — see below), ASR-validates it, and writes narration/manifest.json with per-segment duration_s, wer, and per-word words. (See references/studio-api.md.)
3. Record the target — one clip per scene of the app/site (see Recording), plus a named element-rect map for the elements focus will point at.
4. Compose — scripts/compose.py <storyboard.json>: frames the footage on a gradient stage, burns Pillow captions, slow-zooms hero shots, applies narration-anchored focus (zoom + ring), xfades scenes, lays narration on the computed timeline, muxes.
5. Verify — filmstrip every scene; extract a frame at each focus's spoken-word time and confirm the highlight sits on the named element; ASR two or three slices to confirm the right line lands on the right scene.

Prerequisites

• The Studio API reachable (default http://127.0.0.1:8100). Setup, cloning, and the align/word-timestamp contract are in references/studio-api.md.
• A 3–15s reference wav (24 kHz mono) plus its exact transcript for cloning — confirm the transcript with /asr first (cloning quality depends on audio↔text alignment, not on a "recommended" length).
• One voice for the whole video. Each scene is a separate /studio/tts call, so the voice only stays consistent if every call clones one shared reference. If the user supplies a reference, clone from it; otherwise set a voice design instruct in the storyboard and generate_narration.py designs one clip up front and clones every scene from it (cached, so re-runs keep the same voice). With neither, each scene is synthesized independently and the timbre drifts between scenes — avoid it for real videos.
• ffmpeg with avfoundation (macOS capture), zoompan, xfade, gblur. drawtext is often missing (no libfreetype) — captions are rendered with Pillow.
• For website targets: Playwright (or Puppeteer) for deterministic recording + element boxes.

Recording the target

Each scene needs two things from the target product:

1. a screen-recorded clip (record longer than the narration — results and spinners land seconds after a click), and
2. a named element-rect map: for every element the narration will name, its rect in footage pixels (the coordinate space the composed clip uses).

The target's UI toolkit decides how you get these. Two backends:

A macOS app — AppleScript + ffmpeg avfoundation

• Record avfoundation "Capture screen N" and live-crop to the app window. Probe once (record ~1.5s, extract a frame, confirm crop=W:H:X:Y). The display is Retina, so physical crop = DPR × logical (DPR is usually 2).
• Drive with System Events. Find the reliable primitives for this app's toolkit — they differ per app. General cautions that recur:
  • Coordinate clicks often don't register on SwiftUI/AppKit lists; use AX select/AXPress on the element instead.
  • Segmented controls expose as AXRadioButtons; query them by role.
  • AX queries right after a view switch can race (empty results) — retry, or avoid in-session navigation entirely.
  • If navigation is flaky, relaunch the app already in the needed state (via launch args / env vars / a URL scheme the app supports) and only trigger the one on-camera action. Cheap and deterministic.
• -capture_cursor 0 (AX actions don't move the pointer, so the cursor stays out of frame). For a visible cursor, synthesize one in post (see Focus).
• Element map: query each element's AX position+size and convert to footage px: (global − window_origin) × DPR. If the toolkit hides field rects, measure from a gridded footage frame. Trust the footage frame over stale AX numbers if they disagree (scroll state differs); verify each focus frame.

A website — Playwright (recommended over screen capture)

• Record with recordVideo; set recordVideo.size equal to the viewport and normalize the webm to your fps before composing.
• Coordinate space (the #1 footgun): Playwright records at the viewport size, so when recordVideo.size == viewport, footage px == CSS px. deviceScaleFactor: 2 only sharpens rendering — it does NOT scale the recorded video, and a recordVideo.size larger than the viewport just pads the extra area black (the page stays top-left at 1×, it is NOT scaled up to fill). So boundingBox() gives footage px directly — do NOT multiply by DPR. (If you ever record at a different size, scale rects by recordedSize / viewport.) Always verify by overlaying one known rect on an extracted frame before recording all scenes — a wrong DPR assumption silently puts every focus ring in the wrong place.
• Capture every rect at one settled scroll position: do all scrolling first, let it settle, then read all boundingBox()es, then hold. Rings live in footage coords and must stay glued as the camera zooms/pans. Emit the interaction track from the page clock if you want auto focus.
• Stabilize the page first: disable CSS animations / prefers-reduced-motion, dismiss cookie banners, hide the scrollbar, and waitForLoadState('networkidle') so nothing pops in on camera. Auto-auth via an addInitScript that seeds localStorage before navigation, when the app supports it.
• Motion: for long pages, capture a full-page screenshot and pan it (a vertical Ken-Burns) rather than recording jittery live scroll.

Real-model footage — show genuine output, hide the wait. When a scene must run a slow model live (real captions/chat/inference), record the real flow but time-warp the waits: mark each click → result window (a monotonic clock from page start), then compress ONLY those windows with ffmpeg (trim+setpts+concat; action stays 1×) and freeze the last frame (tpad=stop_mode=clone) so any scene duration plays out on the result. Warm the model on a throwaway request first — a cold load can be 10–40× a warm call (e.g. a vision model: ~38s cold → ~5s warm). Word + time the narration over the warped clip; keep these scenes plain-play (the action carries the motion). Full recipe + snippets: references/playwright-recording.md.

A fully worked macOS example (exact selectors, env hooks, crop rect, element rects, and a turnkey recorder) is in references/video-studio-mlx-app.md — that example happens to demo the same app that hosts the Studio API, but treat its specifics as illustrative; your target will have its own.

Composition (scripts/compose.py)

Config-driven, target-agnostic. Each scene becomes a normalized 1080p clip: footage scaled into a framed card on a dark gradient stage (soft shadow), a Pillow caption (lower-third pill for steps, big title+subtitle scrim for hero), clip + caption fades, then a shared finishing grade (color + S-curve + cool shadows + vignette + film grain + de-banding) so disparate captures read as one directed film ("grade": false disables it, or pass a block to tune strength/tint). Hero scenes get a cinematic treatment: an eased (ease-out) drifting Ken Burns push on the still, a frosted darkened backdrop plate behind the card, and a staged title-then-subtitle fade-in. Clips chain with per-scene xfade transitions (default fade; set transition/xfade_dur per scene — fadeblack reads well into a hero). Audio: narration is adelay-ed onto the master timeline and amix-ed (normalize=0); an optional music bed is sidechain-ducked under the VO and the final mix is loudnorm-mastered to −16 LUFS (VO-only is still mastered).

Focus effects — narration-anchored (zoom + ring follow the words)

Focus must point at what the narration is naming, exactly when it says it — so it's authored as phrase → element, timed from word timestamps, never hand-placed seconds and never pinned to one element for a whole scene. A scene declares a view (which element-rect map) and a focus list (element/view names are whatever your target app has):

"view": "main",
"focus": [
  { "say": "search",   "element": "search_field" },
  { "say": "results",  "element": "result_list" },
  { "say": "generate", "element": "primary_button", "zoom": 1.7 }
]

How compose resolves it:

• When — resolve_phrase finds the contiguous run of words matching say in the segment's words and returns [start,end]. Repeated words aren't ambiguous: the word list is positional — pick by occurrence (default first). Plays at clip-time lead + word_start.
• Where — element is a rect in elements.json under the scene's view (or a map:key to reference a shared map, e.g. a persistent sidebar).
• Camera — multi-keyframe zoompan (focus_camera), absolute on-based smoothstep. Between two focuses it glides directly target→target by default; it only returns to full view when the next target is BOTH far (outside the previous zoomed viewport) AND the gap is long (focus_gap_full, ~2.2s) — so an adjacent/already-visible element just pans a little instead of a jarring zoom-out/zoom-in. Pans are constant speed (duration = distance ÷ focus_pan_speed px/s, floored by focus_move_min): the camera holds the prev framing, then slides at a fixed px/s arriving as the next word starts — near and far pans feel the same, not "slow then whippy." "focus_pan": true forces direct glide everywhere.
• Ring — drawn in footage coords before the camera, so it stays glued to the element as it zooms/pans; faded in/out around the phrase.
• Zoom and ring are decoupled and zoom auto-clamps to fit (fit_zoom): zoom is reduced so the highlighted element stays fully framed, and a wide target drops to ring-only (z=1) so it isn't cropped. Don't pair them by default — both exist only to point the eye; pick what fits the target's size. ring:false gives a gentle zoom with no ring.

Supporting data & extensions:

• Element map (elements.json): named rects per view in footage px. See Recording for how to obtain them; verify by checking each focus frame.
• Capture bigger than you present so zoom stays sharp — the camera crops the native footage down to the on-stage size, so ~1.7× still has real pixels. On macOS the Retina capture already oversamples. On the web, deviceScaleFactor does NOT enlarge the recorded video (see Recording) — to get oversampling, record at a viewport/recordVideo.size larger than the on-stage card (e.g. capture 1920-wide, present into a ~1456 card), not by bumping DPR.
• The recorder can emit an interaction track (emit_track.py) so track_to_effects auto-derives a zoom+ring per click; inline focus is the precise path. Structure is open for cursor/spotlight overlays using the same rects + word times.

Design note (why phrase→element, not API-side cue markers): word timestamps are already a positional list, so the compositor owns "which element / which occurrence" — the Studio API stays a clean primitive (audio + word times) usable to demo any product. The footage must actually show the named element (unscrolled/static) at that moment; word the narration so it does.

Timeline math (keep video and audio consistent)

• scene duration d_i = narration_i + pad_i (pad ≈ 2 step / 3.4 hero).
• scene start on master is accumulated per transition: start_0 = 0, start_i = start_(i−1) + d_(i−1) − T_i, where T_i is the xfade duration INTO scene i (per-scene xfade_dur, default xfade; clamped below both adjacent clips). With every T_i = T this reduces to Σ d_(<i) − i·T.
• place narration i at start_i + lead (lead ≈ 0.8, after the incoming crossfade) — the same recomputed start_i, so audio stays synced. Ensure tin_i + d_i ≤ source clip length.

Verification (you can't watch/listen)

• Frames: one frame per scene + at each focus's spoken-word time + at transition midpoints; hstack into filmstrips and read them. Confirms content, captions, focus placement, clean transitions, and a non-black ending.
• Sync: ffmpeg -ss <t> -t 5 … slice.wav at known scene starts (or focus word-times), POST to /asr, confirm the expected line. This is the only check of the composition/encode layer — the Studio API validates each segment in isolation and can't see your assembled timeline, so an off-by-one or swapped file would otherwise pass silently.

Iteration (cheap re-dos)

• Reword/retime one line: edit its narration, rerun generate_narration.py (re-validates + re-aligns) then compose.py. Focus auto-re-anchors to the new word times; other segments are untouched.
• Re-shoot one scene: re-record that clip, rerun compose.py.
• Tune pacing or focus: adjust pad/tin, or a focus's zoom/ring/say.

Why a separate ASR pass when the Studio API already validates

The Studio API validates content ("does this segment say its script?") on the clean wav, per segment — trust it. Your final ASR-on-slices validates placement ("did the right line land at the right second after adelay/amix/xfade/encode, and does the focus sit on the named element?"), which the API structurally can't know. Different layers, different bugs.