zizhongxiao-svg/code-video-skill

code-video: Have a Large Language Model Write Code to Draw Explainer Videos Frame by Frame

The visuals are not generated by a video model or recorded by a camera: the model writes a webpage (Canvas / SVG / Three.js / WebGL), the page exposes __render(t), a script captures screenshots frame by frame by frame number, and then ffmpeg encodes the result into an mp4. The benefit is reproducible, editable, absolutely accurate text and numbers, and zero generation cost.

The entire process is three steps: Pick a style → Write a detailed prompt → Check with sparse frames first, then render the full video after corrections.

0. Preparation (First Time)

cd <skill directory>
npm install                     # Install playwright-core
npx playwright-core install chromium-headless-shell   # Or set CHROME=/path/to/chrome
# ffmpeg / ffprobe must be on PATH; Chinese fonts recommended: Noto Sans CJK (fonts-noto-cjk / noto-fonts-cjk)
pip install edge-tts numpy scipy   # Optional: narration, subtitles, background music composition

1. Pick a Style: Match the Style to the Subject

First look at what subject you have, then go to references/styles/ and read the corresponding style file (each file contains: suitable subjects, concise prompt, full prompt, production notes, factual basis, and example source code in examples/<name>/).

Subject	Try First	Also
Anecdotes, historical stories	Shadow puppetry piying	Pixel pixel, Comic comic
Children's science, nature growth	Picturebook picturebook	Flat MG flatmg
Text, artifacts, dynasties	Hanzi evolution hanzi	Ink painting ink
Routes, envoys, migrations	Old map oldmap	Isometric flow isometric
Mechanical structures, product disassembly	3D explode explode3d	Whiteboard whiteboard
Production processes, assembly lines	Isometric flow isometric	Flat MG flatmg
Size, depth, distance	Scale scale	Data infographic infographic
Eras, orders of magnitude	Data infographic infographic	Scale scale
Theorems, formulas	Math visualization math	Line art lineart
Single principle, cause and effect	Whiteboard whiteboard	Line art lineart
Slogans, intro/outro	Kinetic typography kinetic	—

All 16 styles are listed in references/styles/README.md. If the user asks for a specific style, read that file directly.

2. Write a Detailed Prompt: A One-Liner Only Sets the Style

A one-line prompt like "make a 10-second shadow puppet video" only tells the model the style—it doesn't say what the story is, which numbers to use, or how long each segment is. Follow the skeleton below to clearly describe four things (the full prompt in each style file follows this structure):

Render a 1920×1080, 30fps, roughly 12-second "style" explainer animation frame by frame using Canvas, specifying colors, texture, and fonts.
Subject: "..." (describe the concept): break it down into several shots, what happens in each shot, what text appears on screen, and end with a one-line punchline.
Use these numbers: ..., with sources written into notes.txt; don't make precise numbers if uncertain.
Each frame is determined only by time t (no timers; use a fixed seed for random numbers).
First extract 4–6 frames to check that subtitles are large enough, labels don't overlap or go off-screen, then render the full video after corrections.

Then add physical/factual rules that the model might get wrong for the subject. Example: in the story of Cao Chong weighing an elephant, the ship's draft must follow the load—otherwise the mark and waterline won't match. If you don't write this rule, the model will draw it incorrectly.

3. Write the Page: Mandatory Page Contract

Each video gets its own directory <work>/<name>/, with at least index.html. It's recommended to also include prompt.txt (the prompt used) and notes.txt (sources for numbers and facts). You can start from templates/canvas/index.html.

• The page must be 1920×1080, overflow:hidden.
• It must include this line (render.sh reads it via regex): const FPS = 30, DUR = 12;
• window.__info = { fps: FPS, duration: DUR, frames: Math.round(FPS * DUR) };
• window.__render = t => { ...; return { seg: '<shot name>', tau: t.toFixed(2) }; } Each frame is determined only by t: do not use CSS animation/transition, setTimeout, requestAnimationFrame, Date.now(), or Math.random(); use a fixed seed for randomness (e.g., mulberry32); particles, jitter, physics must all be directly calculable from t, with no state accumulation between frames.
• After fonts and resources have loaded, set window.__ready = true (for WebGL pages, set it to true after the first frame renders).
• Do not access external networks (the frame-capture script intercepts all external requests). Use fonts already installed on the system.
• Must be readable on mobile: body labels ≥ 36px, titles ≥ 64px, optional small footer text in the bottom-left corner.
• For Three.js pages: run scripts/fetch_three.sh <name> to download to <name>/vendor/, then use <script type="importmap">{ "imports": { "three": "./vendor/three.module.js", "three/addons/": "./vendor/jsm/" } }</script>; set the renderer with preserveDrawingBuffer: true, and call renderer.render() (or composer) synchronously inside __render(t).

For details and common pitfalls, see references/page-contract.md.

4. First Show Sparse Frames to the Model, Then Render the Full Video After Corrections

scripts/render.sh <work>/<name> 30,120,210,300   # Capture only these frames -> <work>/_preview/<name>/

Review each image with an image viewer, checking against this checklist. If you find issues, fix the code and re-extract (at least one iteration):

• Text: Is it large enough? Are any labels overlapping, going off-screen, or obscured? Is all Chinese text rendered in real fonts (no tofu blocks)?
• Facts: Do the numbers, dates, place names, and formulas on screen match notes.txt? For illustrative scenes, label them as "示意图" (schematic diagram).
• Composition: Is the main subject large enough (for distant shots, the protagonist should fill at least 1/4 of the frame height)? Is the frame too empty? Does the color scheme fit the subject?
• Motion: Did the key actions actually happen (compare two adjacent frames)? Are transitions jarring?
• WebGL: Are there black blocks/spots or flickering?

If everything looks good, render the full video:

scripts/render.sh <work>/<name>          # -> <work>/<name>.mp4
N=8 scripts/render.sh <work>/<name>      # Number of parallel frame capture processes (default: min(6, cores/2))
GPU_ANGLE=vulkan scripts/render.sh ...   # For WebGL with GPU (faster, but continuous capture may produce black spots; do a sparse check first)

The encoder is chosen automatically: if NVENC is available, use h264_nvenc, otherwise libx264. Frame capture is mainly CPU-bound; the time-consuming part is usually "model writes the page, you check the screenshots, and fix the code", not the rendering itself.

5. Optional: Narration, Subtitles, Background Music, Preview Image

# Narration + subtitles (default: free edge-tts; synthesize the entire narration at once for consistent voice)
python3 scripts/tts.py script.txt -o out/narration --voice zh-CN-YunyangNeural --rate -5%
# Background music: compose your own (no copyright issues) or use royalty-free libraries (see references/narration.md)
python3 scripts/chiptune.py out/bgm.wav 30          # 8-bit chiptune
python3 scripts/beat.py out/beat.wav --bpm 120 --beats 20   # Beat-accurate drum track, aligned to frames
# Multiplex: music automatically ducks under voice, loudness normalized to -16 LUFS; --burn burns subtitles into frames
scripts/mux.sh out/<name>.mp4 out/<name>.final.mp4 --voice out/narration.mp3 --music out/bgm.wav --srt out/narration.srt --burn
# Preview GIF / poster frame
scripts/gif.sh out/<name>.mp4 2.5 4          # 4 seconds starting at 2.5 seconds
scripts/gif.sh --poster out/<name>.mp4 6.0   # Capture one frame as jpg

If you plan to burn in subtitles, leave the bottom 1/5 of the frame empty—don't draw your own subtitle bar (or use SUB_MARGIN=60 to raise the subtitles). Target loudness: -16 LUFS; voice peaks may be slightly lower (true peak limit -1.5 dBTP). For audio choices, custom TTS integration, and beat alignment, see references/narration.md.

6. Creating Multiple Videos at Once (Style Comparisons, Series)

Each video gets its own directory and prompt—independent of each other. You can distribute the work to multiple sub-agents to write pages and self-check in parallel. The main agent is only responsible for unified rendering, concatenation, and final review. When instructing each sub-agent, be clear about: the page contract, directory conventions, the sparse-frame self-check list, that facts must be written into notes.txt, and that after completion they should report duration and a self-assessment.

Before You Start, Be Clear

• All factual numbers must be real and verifiable; when making videos about breaking news or new categories, only include publicly available information, and mark the corner with "示意图" (schematic diagram).
• Do not reuse example numbers as general facts for other subjects; re-verify numbers when the subject changes.
• Model-generated music (e.g., MusicGen weights) often has non-commercial licenses; for commercial release, use your own compositions or explicitly licensed music.