ninjasweb/video-engine-skill

Video engine

A proven pipeline for turning scripts into finished, publishable videos with code. It is style-agnostic: the engine (voice, timing, animation primitives, QA, render) stays the same; the look, pacing, characters and sound are designed per project from the brief and the client's references.

If the user already has a project built with this engine, read it before building anything new: its layout doc (VIDEOS.md / SAGAS.md), the engine contract (src/engine/video.ts or saga.ts), one style (src/styles/<style>/: kit.ts, Video.tsx) and one finished scene. Reuse its engine instead of starting over. Two layouts have proven themselves:

• Series: episodes that share a character, voice and format (e.g. a kids math saga with TTS and a mascot).
• Independent videos: one-off explainers, each with its own topic, supplied voice, assets and maybe its own look (e.g. tech explainers in a dark diagram style, rendered in 16:9 and 9:16).

Non-negotiable principles

1. The audio is the clock. Generate/import voice first, align it per word, then animate to the words. Never speed up or trim speech to hit a duration.
2. Every visual event is anchored to a spoken word (at(block, 'exact phrase', {word, n})). If the phrase isn't in the alignment, fail loudly — never invent timestamps.
3. Pure function of time. Scene state = f(t). No state between frames → preview == render at any frame, and QA can sample any instant.
4. Deterministic textures. Seeded noise in the object's local coordinates → no boiling/flicker. Measure it (frame diff in static regions ≈ 0).
5. Validate before rendering; review frames before and after. Automatic checks catch logic/timing bugs; a contact sheet catches composition bugs (overlaps, crowding, hidden elements). Both are required.
6. Spend money only with consent. TTS calls cost credits: dry-run (blocks + characters) → ask → generate. Cache by hash of every input that affects the audio.
7. Report honestly. State what was verified automatically, what was looked at, and what was NOT (e.g. "I did not listen to the audio").
8. Many videos, one engine — from the first video. Shared engine + reusable styles + one self-contained folder per video (or per series), with that video's own scripts, voice, scene, thumbnail, assets and per-video components. The engine never imports a video or a style; every command takes the video. See references/project-layout.md.

Workflow

0. Place the video in the project layout — new repo: set up the multi-video layout before writing any scene; existing repo: npm run video -- new / npm run saga -- new (works on a folder the client already filled with audio + script). See references/project-layout.md.
1. Brief & format — audience, goal, platform, duration, aspect ratio, language/locale, monetization constraints (e.g. YouTube "made for kids"). See references/formats-and-niches.md.
2. Style bible — infer from client references or propose 2–3 directions; lock palette, typography, texture, character, motion language, sound palette in a style.json/doc. See references/style-design.md.
3. Script as data — blocks with tts_text (with voice tags), subtitle_text, optional question/wait, visual instructions, expected answers/facts. Validate with a JSON schema.
4. Voice — ElevenLabs with timestamps (or import manual clips + local forced alignment). See references/voice-and-alignment.md.
5. Timeline — place clips, measured gaps/waits, master voice track normalized, per-frame voice envelope, SRT. See references/timeline-and-audio.md.
6. Scene script — per video, a declarative builder using the kit (objects, moves, labels, equations/text reveals, poses, SFX, decor). See references/animation-remotion.md.
7. Validate — schema, waits, "nothing revealed before it's said", counts/answers, no overlaps, safe areas, subtitle band free. See references/qa-and-validation.md.
8. Stills review — contact sheet of 12–15 key instants; fix and repeat.
9. Render — background, logged, memory-safe concurrency; verify with ffprobe + loudness. See references/render-and-performance.md.
10. Deliverables — MP4, <name>.<lang>_<REGION>.srt, thumbnail (mandatory, generated by the pipeline itself — never a separate optional step; per format: 1280×720 for 16:9, 1080×1920 cover for 9:16 with key content centered for 1:1/3:4 crops; fail the run if missing, check its size in output QA, and look at it), master audio, word timings, event timeline, QA report, upload metadata (titles/descriptions with real chapters/hashtags/tags).

Stack (default; justify deviations)

• Remotion 4 (React + SVG) for deterministic frame rendering, Studio/Player preview, CLI batch. Check license (free for individuals/≤3 employees).
• Chromium GL angle on macOS (≈30% faster). Hardware encoders rarely help (bottleneck is drawing) — measure.
• ElevenLabs POST /v1/text-to-speech/{voice}/with-timestamps (char alignment). Library voices need a paid plan via API.
• stable-ts (Whisper) in a Python venv for local forced alignment of manual clips; ElevenLabs forced-alignment API as the precise alternative.
• ffmpeg for mixing, loudness (EBU R128), remux/faststart, contact sheets, flicker measurement.
• Fonts: OFL (Google Fonts) stored locally with license files. SFX/music: synthesize procedurally or use licensed assets; keep copies + license table.

Where things go wrong (learned the hard way)

• ctx.at() must return the first word of a phrase by default; repeated phrases need n (e.g. "una más" said twice).
• Overlap checks must use the object's scaled size; scale changes after validation can break it — re-validate everything after engine changes.
• Objects created by "transfer/take out" helpers must also get a remove time when the scene changes, or they leak into later scenes.
• Moving objects during a question's wait window breaks "freeze during wait"; move after the wait ends.
• Never show a number/answer the narration hasn't said yet (in scene, labels, equations, subtitles). Split subtitle cues at number words.
• Render concurrency: ~2.6 GB RAM per Chromium instance; 6 on an 8 GB Mac → massive swap and a stalled render. Default to floor(totalmem/2.6GB).
• Burned-in subtitles need a reserved band; validate that no object enters it.
• Long titles in fixed boxes overflow — auto-size by length.
• Check each clip's content (transcribe → compare to script, map numerals ↔ words) before trusting imported audio; clips may be missing or out of order. Do the same for generated TTS ("listen" by proxy: Whisper transcript vs script, all numbers present).
• Remotion bundle() copies public/ (all voices) to the OS temp dir on every call and never deletes it; many stills runs filled a 460 GB disk. Always rm -rf the bundle when the script ends. Watch free disk during long batches.
• SVG filters use the element's bounding box: a perfectly horizontal or vertical path (zero-area bbox) with a filter renders NOTHING. Balance arms, number-line ticks and guide lines vanished this way. Straight axis-aligned strokes need no filter.
• A question's data may need to stay visible (prices, data policy) while the answer must not. If a price equals the numeric answer, remove the price before the question.
• A flat single-video repo has to be refactored as soon as the second video arrives. When that happens, capture a baseline first: stills md5, thumbnails, JSON/SRT deliverables and the voice md5. After moving, everything must be byte-identical. Move files and never delete them; park leftovers in _legacy/ for the user to review.
• staticFile() only reaches public/: per-video media lives in the video folder and is synced into public/videos/<slug>/ before each bundle. A new video needs a placeholder timeline (no blocks) so the static import compiles.
• Client scripts often mix narration with visual direction (e.g. NARRACIÓN / REFERENCIA VISUAL — NO NARRAR). Align only the narration, and keep the visual notes per section in script.json, because they are the fidelity spec for the scene.
• The supplied audio may not say what the supplied script says (the narrator improvised, a later take). Always read the free transcription before aligning, and align against what is actually said in a separate file (narracion.md) next to the client's script. Forced alignment against the wrong text silently produces wrong timings.
• Technical explainers need fidelity, not decoration. Compute what can be computed: real photos as tensors, real convolutions, the real noising formula per pixel, a real sampler update. Label the rest as illustrative ("esquema didáctico", "salida ilustrativa"), keep a references/decisiones.md, and say the same in the upload copy. Check product claims against primary sources before drawing them. For example, Imagen is pixel-space cascaded diffusion, not latent.
• Formulas: render the client's TeX with MathJax to SVG paths (no font dependency) in a pipeline step, then draw them as a formula element.
• Diagrams that build left→right are never centered unless the group slides as it grows. Use an automatic recenter that slides before a newcomer enters and after a leaver's fade. Keep headers on CX fixed, and group entrances ≥ 0.6 s apart. For chains, draw empty slots first.
• Seamless loops:
  • Use no fades.
  • Make lead + tail equal a natural pause.
  • Use music generated to last exactly the video length.
  • End on an exact copy of the opening element: a scaled copy differs by antialiasing.
  • Verify with a first-frame vs last-frame diff without subtitles.
• Raster photos inside SVG go in <foreignObject><Img/></foreignObject>, because Remotion waits for <Img>. A bare SVG <image> can render blank frames.
• A generic prop dispatcher with keyframed numeric params (clock minutes, tilt, level, explode offset) scales to dozens of new objects without touching the renderer. Build a prop review sheet (dev still) before writing scenes.

Starting a new project

0. New video or series in an existing engine repo (preferred): use its scaffolding command (e.g. npm run video -- new --video <slug> or npm run saga -- new --saga <slug>). It creates the folder from _template/, registers it, and optionally copies a base style to src/styles/<new-style> as a starting point to restyle.
1. New repo: set up the multi-video layout from references/project-layout.md before writing any scene: package.json, tsconfig.json, remotion.config.ts, src/engine/ (scene model, word cues, formats, subtitles, the video contract), one style in src/styles/<style>/, scripts/ (timeline, validate, stills, render, thumbs, check, the new command), tools/ (alignment, procedural assets), a _template/ folder, the registry and Root.tsx. Then npm install, and create a Python venv with stable-ts only if supplied audio must be aligned.
2. Adapt the script schema to the niche's block structure (roles such as hook, teach, example, question/wait, recap).
3. Implement the style bible: palette, fonts (local, licensed), filters, background texture (generated by a script), character or presenter, SFX/music.
4. Render a 12–15 s sample from real audio → user approval → then scale.
5. Keep project rules in the project's CLAUDE.md (costs/consent, naming conventions, language, review steps) and the layout in a VIDEOS.md / SAGAS.md.

Reference files

• references/style-design.md — infer/propose a visual identity, style bible, consistency rules.
• references/voice-and-alignment.md — ElevenLabs, caching, errors, manual clips, forced alignment.
• references/timeline-and-audio.md — clip placement, waits, loudness, music ducking, SFX, subtitles.
• references/animation-remotion.md — scene data model, authoring kit, character rig, motion language, preview.
• references/textures-and-rendering-looks.md — seeded SVG filter recipes for many looks.
• references/qa-and-validation.md — automatic rules, visual review, output checks.
• references/render-and-performance.md — render settings, memory, queues, deliverables, upload metadata.
• references/formats-and-niches.md — aspect ratios, safe areas, niche playbooks.
• references/project-layout.md — many videos per repo: per-video folders, reusable styles, the video contract, assets sync, migrating a flat repo.