jblxo/programmatic-video-skill

Programmatic Video Generation Skill

A deterministic, zero-boilerplate workflow for AI coding agents to design, visually QA, score, and render 60fps 1080p MP4 animated videos directly from a single HTML5 Canvas/SVG file, a Python procedural audio script, and a headless Playwright → FFmpeg pipe.

Paths. ${CLAUDE_SKILL_DIR} is the directory containing this SKILL.md. In Claude Code it expands automatically; in Cursor, Codex, OpenCode, or other agents, replace ${CLAUDE_SKILL_DIR} with the resolved path to skills/programmatic-video/.

---

Why Deterministic Frame-Stepping?

Never use real-time browser screen recording (MediaRecorder, requestAnimationFrame wall-clock capture, or CSS transitions running in real time) to produce videos with an AI agent:

1. Wall-clock capture drops frames whenever CPU load spikes, causing audio/video drift and stutter.
2. CSS animations run on wall-clock time, making it impossible to seek to t = 14.20s for instant visual QA.
3. Blind rendering wastes minutes: rendering 5,000 frames before checking if a title overflows its card guarantees wasted work.

Instead, every video built with this skill obeys three rules:

• Rule 1 — Pure Function of Frame Index: The entire scene is driven by window.__setFrame(frameIndex, timeSeconds, fps) where t = frameIndex / fps. Every position, opacity, counter, and particle is a deterministic math function of t (using a seeded PRNG like mulberry32, never Math.random()).
• Rule 2 — Multimodal Contact-Sheet QA Before Full Render: Before encoding the full 60fps MP4, capture 8–12 key timestamps (render.mjs --timestamps ...), stitch them into a single contact-sheet.png (contact-sheet.py), and inspect that image using your multimodal Read/View tool to fix text clipping, overlap, or low contrast.
• Rule 3 — Sample-Accurate Audio + Zero-Disk-Bloat FFmpeg Pipe: Generate a multi-segment stereo WAV soundtrack + timestamped SFX cues (synth-audio.py), then stream PNG buffers from Playwright directly into ffmpeg -f image2pipe (render.mjs).

---

End-to-End Agent Workflow (6 Steps)

Step 1: Storyboard & Scene Budget

Design a concrete timeline table before writing code. Group the video into 4–8 second scenes with explicit transition timestamps (T_cut) that both scene.html and synth-audio.py share:

Scene	Time (t)	Frames (60fps)	Visual Focus	Audio / SFX Cue
01 — Hook / Alert	0.0s – 7.0s	0 – 420	CRT terminal boot, typewriter logs, CTA button click at 2.7s	Sub pulse + UI click/unlock cue at 2.7s, riser into 3.4s
02 — Hero Metrics	7.0s – 15.0s	420 – 900	3D perspective synthwave grid + 3 spring-physics counter cards	Full 124 BPM synthwave bass + stereo arpeggiator drop at 7.0s
03 — Deep Telemetry	15.0s – 23.0s	900 – 1380	Split-screen progress bars + animated semi-circular gauge	Filter sweep + high-hat drive, sub impact at 15.0s
04 — Roll Call & Finale	23.0s – 30.0s	1380 – 1800	Bento grid of contributor cards + "JOB'S DONE!" banner & confetti	Finale chord + voice/SFX cue at 25.2s, master fade-out 28.5–30.0s

Step 2: Author scene.html with window.__setFrame(f, t, fps)

Write a self-contained 1920x1080 HTML file using HTML5 <canvas> (and/or inline <svg> + DOM overlays) that exposes:

window.__setFrame = function(frameIndex, timeSeconds, fps = 60) {
  if (window.__rafId) {
    cancelAnimationFrame(window.__rafId);
    window.__rafId = null;
  }
  const t = typeof timeSeconds === 'number' ? timeSeconds : frameIndex / fps;
  renderSceneAtTime(frameIndex, t);
};

• Always include a seeded PRNG (mulberry32(seed)) for stars, particles, confetti, or noise so frame f is bit-identical across runs.
• Wait for document.fonts.ready (handled automatically by render.mjs).
• See references/visual-patterns.md for copy-pasteable recipes (3D perspective grid, spring counters, CRT terminal, gauges, bento cards, confetti physics).

Step 3: Run Fast Timestamp Snapshots & Build contact-sheet.png

Never run a full 60fps render first. Sample 1–2 timestamps per scene (plus transition midpoints) and build a single contact sheet:

# 1. Capture 9 keyframe PNGs in ~3 seconds
node "${CLAUDE_SKILL_DIR}/scripts/render.mjs" \
  --html ./scene.html \
  --timestamps 1.8,3.0,5.2,8.5,12.5,16.5,20.5,24.5,28.0 \
  --out-dir ./qa-frames

# 2. Stitch them into a labeled grid image
python3 "${CLAUDE_SKILL_DIR}/scripts/contact-sheet.py" \
  --dir ./qa-frames \
  --cols 3 \
  --thumb-width 560 \
  --title "VISUAL QA CONTACT SHEET" \
  --out ./contact-sheet.png

Step 4: Multimodal Self-Review Loop (Mandatory!)

Call your image Read / View tool on ./contact-sheet.png (and any individual ./qa-frames/t-*.png that looks dense). Check every tile against the checklist in references/visual-qa-loop.md:

• Does any text overflow or collide with card borders, badges, or progress bars?
• Are all numbers, labels, and HUD headers high-contrast and legible at 1080p?
• Did every scene transition land cleanly without an unintended empty/black frame?

If you spot any issue in contact-sheet.png, fix scene.html and re-run Step 3 before moving on.

Step 5: Synthesize Procedural Soundtrack & Mix WAV SFX Cues

Generate a stereo 44.1 kHz WAV soundtrack whose transition risers and SFX cues match the exact timestamps in scene.html:

python3 "${CLAUDE_SKILL_DIR}/scripts/synth-audio.py" \
  --duration 30.0 \
  --bpm 124 \
  --style synthwave \
  --transitions 3.4,7.0,15.0,23.0,25.2 \
  --cue "25.2:./sfx/jobs-done.wav:+1.5" \
  --out ./soundtrack.wav

For custom multi-segment compositions or integrating themed sound packs (such as jblxo/claude-code-sound-packs), read references/audio-synthesis.md.

Step 6: Render Final 60fps H.264 MP4 via FFmpeg Pipe

Once visual QA and audio synthesis pass, run render.mjs in full pipe mode:

node "${CLAUDE_SKILL_DIR}/scripts/render.mjs" \
  --html ./scene.html \
  --duration 30 \
  --fps 60 \
  --audio ./soundtrack.wav \
  --crf 18 \
  --out ./final-video.mp4

Verify the output container with ffprobe:

ffprobe -v error -show_entries format=duration,size:stream=codec_name,width,height,r_frame_rate -of default=noprint_wrappers=1 ./final-video.mp4

---

Reference Playbooks (Loaded on Demand)

Topic	Reference File
Canvas/SVG visual primitives, easing functions, seeded PRNG, 3D synthwave grid, gauges, bento cards	references/visual-patterns.md
Multimodal contact-sheet QA loop, common layout defects, timestamp selection strategy	references/visual-qa-loop.md
Procedural Python/NumPy audio synthesis, chord progressions, risers, ducking, WAV SFX cue mixing	references/audio-synthesis.md