themosthappypiano/reel-style-studio

Transcript-Driven Video Editing

Edit video by deciding cuts from a word-level transcript, generating motion graphics as HTML, and compositing with ffmpeg. The NLE is optional and comes last.

Use when asked to AI-edit raw footage, auto-cut a talking-head video, build an automated editing pipeline, or generate motion graphics programmatically.

Canonical pipeline lives at ~/Projects/editor-agent on the Mint laptop roughcut.py → HyperFrames → render.py. Its style/DIRECTOR.md is the per-project editing brain and carries a running gotchas log — read it before a job and append to it after.

instead — HyperFrames CLI is already installed there with working examples. Scaffold a new proof-of-style project with:

HYPERFRAMES_SKIP_SKILLS=1 npx hyperframes init <name> --non-interactive \
    --skip-transcribe --resolution portrait

Then npm run check (lint+runtime+layout+motion+contrast) and npm run render -- -o renders/<name>.mp4 inside the new project dir.

Always-on rules

1. Verify every render visually. Extract a still and look at it with vision_analyze. ffmpeg exits 0 on overlays that are off-screen, mistimed, or entirely absent — a clean exit code is not evidence the edit is correct.

2. Sample a settled frame, not the boundary. Check ~2–2.5s into a graphic's own timeline. A still grabbed mid-animation looks broken when the animation is working, and will send you chasing a bug that does not exist.

3. Never invent a visual style. If a brief lacks look-and-feel direction, ask targeted batched questions (style reference, colour mood, animation beats, weight) before generating. Execution is autonomous; creative direction is not. Leave unspecified brand values as clearly-labelled PLACEHOLDER, never as plausible-looking invented defaults.

4. Keep the LAST take when a speaker restarts a sentence — it is almost always the best one, and silently choosing an earlier take is a creative decision the user did not delegate.

5. Report what you cut for content reasons. Pacing cuts are yours; removing an idea is the user's call.

6. "Edit" means graphics, not just cuts. Delivering a silence-trimmed cut when asked to edit a video reads as not having done the job. Unless told otherwise, an edit includes at least an opening/title graphic and one graphic carrying the content's structure. Propose the graphics up front instead of offering cutting alone and waiting to be asked for more.

7. A first graphics pass is a draft. Refine one graphic at a time with specific direction. Batch-generating many graphics produces generic output.

8. Source motion from the block registry before hand-rolling it. Hand-written CSS keyframes and linear slide-ins look amateur beside designed motion, and the user will say so. Check references/hyperframes-block-registry.md first; when authoring custom, copy its patterns (back.out entrances, autoAlpha +scale, staggers, SVG stroke-dashoffset connector draws).

9. Claude has no colour vision. Never build a colour grade from scratch — feed an existing LUT or ask for a reference.

10. Deconstruct a reference reel before generating. First obtain a word-timed transcript and a low-rate contact sheet for the whole reel; classify each beat by speech trigger, face/proof/full-screen coverage, authentic capture versus constructed illustration, typography, brand micro-UI, and transition. Then sample uncertain beats at source frame rate (ffmpeg -ss <start> -i <video> -vf fps=<source_fps> ...) before describing their motion; a half-second contact sheet hides two-frame fades, menu states, and mascot/bubble details. Report observed frame states separately from inferred easing or cursor actions, and ask the user only about movement that remains ambiguous. Preserve reusable editorial functions, not another creator's footage/assets. See references/visual-proof-layout-modes.md for layout vocabulary.

11. Delete a retained third-party reference clip once its written study is extracted. Saving someone else's video into the project as a style reference is fine only until the .md analysis exists — after that the video file itself is someone else's face/voice/footage with no licence to keep. If unsure whose footage it is, compare an extracted frame against the creator's public photos before deciding what to keep; then rm the source file and retain only the written notes.

12. Use an image-first approval gate for generated visual beats. Present the original start stills, and where relevant matched end stills, for approval before spending credits on Flow/Kling animation; the still determines the visual language and is cheaper to correct than a rendered clip.

13. Separate brand truth from generated motion. Pull named brands, product marks, repository screenshots, and real UI from authoritative or user-supplied sources; composite them as post-animation overlays or screen captures. Image/video models mutate logos and small text, so never present generic generated cards when the transcript names the actual software. 13b. Classify every beat REAL / MOCK / TEMPLATE and hand the user a manifest of the REAL rows before building. REAL = any claim about the user's own business, product, or results (metrics, dashboards, CRM state, execution history, "runs every day") — never drawn, reconstructed, or estimated. MOCK = a concept with no real artifact, or one where the artifact is the thing being criticised — design it, and design it well. TEMPLATE = a card already built, refilled with new copy. Returning the whole table (beat · spoken words · tag · what is needed) lets the user redirect a beat tagged wrong and record only what is genuinely missing; starting a build with an unresolved REAL beat forces the choice between stalling and quietly faking it. Depth: references/visual-asset-sourcing.md. 13c. Route any beat carrying legible type or UI chrome away from generative image/video. Generative motion and upscaling smear small text, and these cards are read rather than watched — author them in HTML/CSS where type stays crisp and animate with transforms (zoom, stagger, typewriter). Generative tools earn their place on atmosphere and texture under an abstract line, never on a proof object. 13d. A creator's recurring visual is one asset placed many times, not a fresh generation per beat. Build each recurring noun in the user's scripts ONCE as a composition and refill it; keep it with the project's worked examples so the next video copies rather than rewrites.

14. Do not use a creator's captioned reference Reel as the base for a subtitle prototype. Its burned-in captions and prior graphics collide with the new editorial layers, making a technical render look like a finished style test; use clean talking-head footage, or show the overlay as a standalone proof until clean footage exists.

15. Before building the first proof sequence, batch-ask what's still undecided rather than assuming. At minimum: whether presenter footage exists yet (build the proof-card mechanics standalone if not — a face is not required to validate hierarchy/depth/motion), where to build (which machine/toolchain already has working examples), and whether to scope to one beat or storyboard the whole script first. Getting this from one clarify call up front is faster than generating something and finding out the scope was wrong.

16. Distinguish a creator's thumbnail grammar from their in-video grammar. A dark/red-block thumbnail does not prove their actual reel uses a dark canvas; inspect the specific moving reel before choosing a palette or layout. Keep distinct reference registers separate unless the user asks to blend them. See references/nick-saraev-visual-grammar.md.

17. Keep the visual state changing at least every 4-5 seconds — this user has explicitly rejected long face-only stretches ("we need visuals constantly"). A hook or CTA beat with no real screenshot to show is not licence to leave the speaker alone on screen for 8+ seconds; use the illustrated-concept-card exception (rule in references/visual-proof-layout-modes.md) for beats that have no real-source equivalent, so no gap in the storyboard is visual-free. Plan the beat list first and check for any face-only span over ~5s before building.

18. Search for an existing render before building anything from a raw clip. Find prior project folders and renders, then frame-match the raw against each project's source/face video (a low mean-abs-diff on downscaled grayscale frames = same take). When a prior version exists, patch and re-render IT and stay in that project's toolchain; a fresh reimplementation in a different framework discards the finished work's craft and the user will recognise the downgrade immediately.

19. Check card elements against the caption band after the first render. Card layout and caption layout are authored separately and collide silently, hiding exactly the counter or figure the beat exists to show. Extract a native-res frame of every card beat; when something collides, move the card's element, not the caption. On beats whose card already prints the sentence, caption only the words the card does not show — the same words twice on one frame reads as a rendering bug.

20. Verify a product name or a self-contradicting figure against the audio before it reaches the screen. ASR mis-renders product spellings confidently at every beam size, and a phrase that contradicts itself is a misspeak to resolve with the user, not to render. Stop a proof section before any unverified claim about the user's business rather than shipping it as a placeholder.

21. A npm run check pass with 0 errors is not evidence a clip's exit animation actually plays. Two silent-but-real failure modes surfaced from real builds: a GSAP tween scheduled on a clip whose data-duration ends before that tween's start time (the element unmounts before the fade-out ever fires — it just vanishes), and a GSAP scale/rotate tween fighting a CSS transform already used for static centering/tilt on the same element (GSAP overwrites the whole transform, silently discarding the centering). See references/hyperframes-compositions.md for both fixes.

22. Keep the speaker's face centred in every vertical crop. A centre-crop of a landscape webcam leaves the face off-centre and the speaker drifts, so cut in landscape and reframe with scripts/facetrack.py; check centring at the start, middle, and end. Recipe: references/layered-screen-mode-compositing.md.

23. Mix screen modes per beat; play the user's recordings as video. Default to the Saraev mix (full-screen proof for concrete claims, full-frame face for summaries and the CTA, split only for lists), not a permanent split. When the user supplied a screen recording of what the line describes, that beat plays the real footage. Mode table + recipe: references/visual-proof-layout-modes.md, references/layered-screen-mode-compositing.md.

24. Use the user's named music file when given one. Fetch it from their machine (e.g. ssh host 'cat "<path>"' > local.mp3); a partial copy of a long track is a valid file once trimmed with ffmpeg, so a timed-out transfer of a 1-hour mix still covers a 1-minute reel.

Working style for long builds

• Post a one-line status every few steps. Long silent tool chains lose the user; they will interrupt with "what are you doing". Say what just worked, what is next, and keep moving.
• Stop before spending money or committing to an architecture. When a build hits a paid dependency or a fork in approach, present the options with real prices and stop for a decision. Do not build past the fork on an assumption.
• Exhaust the free path before recommending any purchase, and say plainly when hardware would not help. Most of this pipeline's value (transcription, cutting, graphics, rendering) is free and CPU-only; the NLE is the one paid, most swappable piece.
• Verify the source workflow before cloning it. When asked to build something modelled on a public creator's system, pull their actual transcripts and check the claims — second-hand build prompts drift from what the creator really does and go stale against vendor policy changes. See references/source-workflow-verification.md.

Procedure

1. Transcribe (word-level is mandatory)

./transcription/venv/bin/python transcription/transcribe.py \
    media/raw/<clip>.mp4 --model small --outdir media/transcripts

Emits .json (canonical: segments + per-word start/end/score), .srt, .txt. Roughly real-time on CPU with small.

• Never pass --no-align when a cut will follow. Segment-level timestamps leave dead air at every line head and tail; the whole method depends on cutting on words.
• Always use the project venv (./transcription/venv/bin/python). System python lacks the ML stack.
• Whisper mis-hears proper nouns (product names, brands, tool names) consistently. Spell-check them before burning captions; timing-based cutting is unaffected.

2. Plan the cut

./transcription/venv/bin/python roughcut.py media/transcripts/<clip>.json

Writes <clip>.plan.json — editable JSON. Tweak a timestamp and re-render without re-transcribing.

Flag	Default	Effect
--pad	0.12	breathing room each side; raise ~0.2 for calm, drop ~0.06 for punchy
--max-gap	0.45	silence longer than this becomes a cut
--min-seg	0.40	discard keeps shorter than this
--keep-filler	off	retain "um", "so", "like"
--no-dedupe	off	retain stutters

• Never cut mid-word — snap all boundaries to word timestamps.
• Do not pad to zero. Hard-cutting every gap makes speech sound robotic.
• Strip filler only when sentence-initial. "it works like this" must keep its "like"; blanket filler lists butcher natural rhythm.
• Sanity-check compression against the job type: 40–70% kept for a normal edit, 15–25% for an intro or hook. Outside the band for the job at hand, re-check --max-gap before rendering.
• Silence removal alone plateaus around 60–75% kept and cannot reach intro length. Going below that means dropping whole ideas — a content decision, not a timing one. See references/content-level-editing.md for the authored-plan technique and intro-tuned flags.

3. Generate graphics (HyperFrames)

Read the transcript before designing anything. Graphics should encode the content's real structure — material that enumerates four steps wants a four-step tracker, not a decorative flourish. Structure discovered in the transcript is also what justifies a graphic's existence to the user.

Check framing before choosing a position. Extract a still from the source, see which region of the frame the subject leaves empty, and place graphics there. A centred overlay on an off-centre talking head covers the speaker's face. In a vertical proof-plus-face state, preserve the source camera crop as the lower visual anchor instead of automatically putting the speaker in a generic rounded panel; place one proof object above it only when the frame genuinely has room. Use a full-screen proof visual when the viewer must read UI/code/compare states, then cut directly back to the speaker.

Source branded proof before generating a concept visual. When narration names software, first locate its official logo/brand asset or use a user-supplied screenshot. Build the label/logo in the local composition after Flow/Kling motion, then animate only the original background, connector, highlight, crop, or state change. If the user supplies a repository or product capture, treat it as the canonical proof surface and animate cursor/highlight/crop states instead of asking an image model to recreate the interface.

Make the first approval pass a real proof sequence, not a batch of generic concept cards. After studying a reference style, build one short scene from an actual screenshot or product asset: face → proof card → focused crop/highlight → full-screen result → face. It tests hierarchy, depth, and motion together before credits or time are spent producing a whole visual library.

See references/hyperframes-compositions.md for the authoring contract — the three attributes that silently break a render, and the verification command. Each composition needs its own directory holding a copy of the scaffolded hyperframes.json, meta.json and package.json; produce several graphics for one video by copying those config files into sibling dirs and rendering each.

See references/hyperframes-block-registry.md for pulling ready-made blocks and the motion patterns that make custom work look designed.

See references/visual-proof-layout-modes.md for the six-mode layout vocabulary (face_full, face_plus_proof, proof_plus_face_cutout, proof_full, brand_reset, cta_dark) and the real-proof-over-generated-illustration rule that governs which mode and which asset to pick per beat.

See references/nick-saraev-visual-grammar.md for an observed thumbnail register and a protocol for analyzing his moving reels. Do not transfer a thumbnail's dark/red-block look to the in-video edit without checking the actual source reel.

3b. Captions / subtitles

See references/subtitle-caption-styles.md for the registry-first word-level caption workflow — which of the 17 registry caption components matches this pipeline's style (caption-editorial-emphasis for the default flowing-text- plus-one-serif-emphasis look, caption-highlight for punchy per-word flash beats), which to skip (anything neon/glow), and how to wire the real WhisperX transcript into either one.

4. Render

./transcription/venv/bin/python render.py media/transcripts/<clip>.plan.json \
    -o media/renders/final.mp4 \
    --overlay "media/graphics/<g>[email protected]" --music bed.mp3 --music-db -22

See references/ffmpeg-compositing.md for the overlay timing and scaling traps — both fail silently with exit code 0.

5. Log what you learned

Append new gotchas to the project's style/DIRECTOR.md as symptom → cause → fix.

Choosing an NLE (or skipping one)

Default to no NLE for deterministic transcript-driven UI animation; use an NLE/Fusion for interactive refinement and finishing when it adds value. Before promising agent-driven NLE integration, check references/nle-scripting-access.md for current edition, bridge, codec, and host constraints.