video-flow: Code-Rendered General Explainer Video Workflow
In a nutshell: First, think clearly about 'what to teach,' then turn that into a calculable timeline, and finally let the browser draw each frame. Final product = Canvas/HTML
render(t)+ headless Chrome frame capture + FFmpeg + optional TTS/music/subtitles.
This skill breaks down "a video" into 8 checkable stages. Each stage has an output and an acceptance checkpoint; do not proceed to the next step if any step is unsatisfactory.
0. When to Use / When Not to Use
Use:
- Turn a PDF, PPT, DOCX, article, webpage, video, or a topic into a 1–5 minute explainer video;
- Need a video where "text is clear, data is accurate, every frame is editable, and the process is reproducible" (structural advantage of code rendering over generative video models);
- Need multiple versions (portrait/square/landscape) or multi-language versions;
- Need narration, subtitles, music, and want the entire workflow repeatable.
Do not use:
- Need real people/live-action/realistic footage → use video generation models or live footage; this skill only handles "the code layer";
- Just want to cut existing footage → use editing tools/agents;
- Only want to generate an MP4 without willing to define storyboards and script → the result will feel templated; it is recommended to do steps 1–2 first.
1. Hard Rules (Violate any, redo)
- Every frame is a pure function of time: The page exposes
window.render(t); givent, the frame is uniquely determined.- Prohibit
Date.now(), unseeded random,requestAnimationFramedriven state, CSStransition/animation, cross-frame caching. - For cumulative animations (particles/water), push from 0 to
twith fixed steps or switch to analytical formulas.
- Prohibit
- The timeline is the single source of truth: Storyboards, narration, music, sound effects, and subtitles all derive from the same
timeline.json. - Storyboard before code: Without
storyboard.mdandscript.json, do not write scene code. - Do not fabricate source facts: Numbers, years, names, and formulas must come from the source material; mark uncertainties with
[to verify]and list them separately at delivery. - No branding/watermarks by default: Only if the user provides a logo/brand name; default footer only shows course/topic information.
- Encode only once: Complete mixing and loudness in the WAV stage; perform only one final AAC encode; do not re-encode repeatedly.
- Must pass QA before delivery: Check for dropped frames, black frames, loudness, duration, subtitles, and audio-sync; see step 8.
2. Input and Output
Input (choose any, can be combined)
| Type | Processing Method |
|---|---|
scripts/ingest.py extracts text; for scans/garbled text, automatically rasterize to PNG and let the model "read the pages" | |
| PPTX | Extract title, body, and notes per slide; optionally export key slides as images |
| DOCX / Markdown / TXT | Directly extract as source/source.md |
| Web URL | Fetch main text; preserve source link |
| Video | ffprobe metadata + key frame extraction; if no ASR, human supplements content |
| Bare Topic | A sentence/outline from the user; first have the user confirm "facts that must appear" |
Project directory contract
<project>/
project.json # Metadata: title/audience/goal/aspect ratio/style/voice/music/brand
source/ # Source materials (source.md, pages/*.png, meta.json)
storyboard.md # Storyboard table: what each scene covers, visuals, duration, narration, key points
script.json # Machine-readable: scenes[] (type/script/visuals/vo/duration)
build/ # index.html + lib.js + theme.js + scenes.js + timeline.js
audio/ # vo/*.wav, bgm, mix.wav
frames/ # Frame-by-frame JPEGs
out/ # final.mp4 / final.srt / poster.png
qa/ # report.md + contact sheets + key frames
Use scripts/init_project.py to generate the skeleton at once.
3. Workflow (8 Steps)
Step 0: Environment Check
python scripts/check_env.py
Required: Python 3.10+, FFmpeg/ffprobe, a Chromium-based browser, numpy/scipy.
Recommended: edge-tts (online neural voice), pypdf, python-pptx, python-docx, Pillow.
Install what is missing per the output prompts; if not installable, degrade (no narration/system voice/hand-written audio).
Step 1: Read Source Material, Extract "Teaching Structure"
Answer 5 questions (write into project.json's brief):
- Who is the audience? What do they already know?
- After watching, they should be able to say/do what? (1–3 testable goals)
- What is the most easily misunderstood/hardest point?
- What facts/formulas/data must appear? What are the sources for each?
- Duration and aspect ratio? Platform (classroom/mobile/social media)?
Do not follow the original page order. Restructure using one of the 4 storytelling structures from references/storytelling.md:
- Causal chain (recommended for science/engineering);
- Problem → Solution → Validation;
- Misconception → Correction → Transfer;
- Timeline/process.
Step 2: Choose Style and Aspect Ratio
Select 1 set from references/styles.md (silicon / paper / blueprint / neon / swiss), and define the aspect ratio from references/formats.md.
Rules:
- Content leaning toward "chips/engineering/data" → silicon / blueprint;
- Leaning toward "humanities/biology/history" → paper;
- Leaning toward "launch/marketing/youthful" → neon / swiss;
- One screen, one concept; portrait mode must re-layout, do not crop landscape to portrait.
Step 3: Write Storyboard and Script
Output storyboard.md (for humans) and script.json (for machines). Each scene must include:
type(title/concept/process/compare/formula/bars/quote/recap/custom)kicker(subtitle),headline(one-sentence conclusion),points(≤3 items)vo(voiceover, 1–2 sentences; leave blank if no narration)visual(specific data for charts/tables/flows/formulas)duration(can be empty, automatically determined by narration length)
5 hard constraints for writing copy:
- On-screen text ≠ narration script: On-screen only key words/conclusions; narration explains.
- At most 1 conclusion + 3 supporting points per screen; title ≤ 18 chars, body text ≤ 22 chars per line.
- Change every 2–4 seconds (element/camera/emphasis), but no particle explosions.
- Write a "forbidden list" first: gradient background + centered big text, fade-in for all elements, full-screen small text, meaningless glows.
- Mark factual sentences with sources individually; add
[to verify]for uncertainties.
Step 4: Generate Narration and Timeline
# Optional: skip if no narration, just write `duration` for each scene
python scripts/tts.py --project . --voice zh-CN-XiaoxiaoNeural --rate +3%
python scripts/timeline.py --project .
tts.pyoutputsaudio/vo/<id>.wav(48 kHz), automatically applies light EQ/compression; supports--from-dirfor human recordings.timeline.pyuses measured narration durations to generatebuild/timeline.jsandtimeline.json:scene.duration = head + voDuration + tail; for scenes without narration, use explicitduration.- Afterwards, only modify
script.jsonand re-run these two steps; do not manually edit the timeline.
Step 5: Build the Page and Implement Scenes
Using the template:
python scripts/init_project.py --dir my-video --title "..." --format 16:9 --style silicon
# Edit build/content, generated by timeline.py; generic scenes are already implemented, just modify script.json
- Generic scenes (8 types) are in
build/scenes.jsreadingCONTENT; sufficient for most explainer videos. - For specialized graphics (chips/circuits/anatomy/maps/molecules), write
build/custom.js:window.CUSTOM['s3'] = function(u, D, sc, ctx){ /* use helpers from lib.js to draw */ };typetocustominscript.json. - Work on one scene at a time: first render 3 still frames (
scripts/render.py --still 1.2,2.4,3.6), check for text overflow/overlap/subject size, then continue.
Step 6: Rendering
python scripts/render.py --project . --fps 30
# Render only a segment: --start 8 --duration 4
# Reuse an existing Chrome: --cdp http://127.0.0.1:9333
# Let the script launch Chrome: --launch-chrome
Outputs frames/f00000.jpg..., supports resumable rendering (skips existing frames >2KB).
Headless Chrome and sandbox: If the runtime environment prohibits Chrome's IPC/pipe, run in an allowed environment, or connect to an already launched Chrome via
--cdp.
Step 7: Music, Sound Effects, Mixing, Subtitles
python scripts/audio.py --project . # Synthesize BGM + place narration + duck + master
python scripts/srt.py --project . # Generate out/final.srt
- No narration:
audio.py --no-bgmor use only user-provided music. - User provides music: place in
audio/music.*;audio.pyautomatically loops/cuts and ducks. - Target loudness default: -14.2 LUFS, true peak ≤ -1.5 dBFS; encode audio only once.
Step 8: Encoding + QA + Delivery
python scripts/encode.py --project . # frames+mix+srt → out/final.mp4
python scripts/qa.py --project . # qa/report.md + contact sheet of key frames
QA must fully pass (see references/qa.md):
- Duration/aspect ratio/frame rate/audio track/subtitle stream correct;
- No black frames ≥0.25 seconds; key frames have no blanks, overlapped text, or overflow;
- Loudness within target ±1 LU, true peak ≤ -1.5 dBFS;
- First frame is not pure black; last scene has a conclusion;
- Fact list matches source materials;
[to verify]items listed separately.
Delivery: out/final.mp4, out/final.srt, poster.png, storyboard.md, script.json, qa/report.md, rendering time, and known shortcomings.
4. Generic Scene Types (Implemented in Template)
| type | Use Case | Key Fields |
|---|---|---|
title | Opening/chapter | title/subtitle/kicker |
concept | One concept + 3 key points | headline/points/visual |
process | Process/steps (3–5 steps) | steps[] |
compare | Side-by-side comparison | left/right |
formula | Formula + explanation | formula/notes[] |
bars | Data comparison | items[{label,value}] |
quote | Key quote/conclusion | quote/author |
recap | Summary map | nodes[]/takeaway |
custom | Specialized graphic | Write your own custom.js |
When a new type is needed: add a window.SCENES_IMPL['type'] in scenes.js and document the data contract in references/pipeline.md.
5. Aspect Ratios and Safe Zones
| Aspect Ratio | Dimensions | Safe Zone | Title Font Size | Body Font Size |
|---|---|---|---|---|
| 16:9 | 1920×1080 | Left/Right 96 / Top/Bottom 64 | 72–120 | 28–40 |
| 9:16 | 1080×1920 | Left/Right 64 / Top/Bottom 160 | 64–96 | 32–44 |
| 1:1 | 1080×1080 | All sides 72 | 56–88 | 28–40 |
| 4:5 | 1080×1350 | All sides 80 | 60–92 | 30–42 |
Portrait mode requires re-layout: title at top, graphic in middle, conclusion at bottom; do not compress horizontally. The template uses layout() to switch based on aspect ratio; see references/formats.md.
6. Common Failures → Fixes
| Symptom | Root Cause | Fix |
|---|---|---|
| Black/white screen | JS error, render undefined, canvas size 0 | render.py --still, check console; explicitly set canvas width/height |
| Wrong font/first frame fallback font | Font not loaded | Wait for document.fonts.ready before setting READY=true; use system fallback for CJK |
Same t gives different results / flickers during parallel execution | Used rAF/timer/random/cross-frame state | Convert to pure function; hash(seed); closed-form spring |
| Text overlap/overflow | No per-scene still frame check | 3 still frames per scene + 360px wide key frame for portrait |
| Audio-video desync | Audio starts from 0 vs video starts at some point; TTS per-sentence concatenation | Same timeline; whole-segment TTS; -ss placed before -i |
| Loudness inconsistency | Multiple loudnorm/transcoding | Only static gain in WAV stage; one AAC encode |
| Rendering too slow | 4K/subframes/brush/WebGL | First render 960×540 sample; subframes only for fast motion; parallel workers |
| Chrome fails to start | Sandbox limits IPC/pipe | Run in allowed environment, or use --cdp to connect to an already running Chrome |
7. Extension Points
- New style: copy
templates/styles/*.json, modify color palette/fonts/background type; add a row instyles.md. - New scene: register in
build/scenes.js; or write specialized graphics inbuild/custom.js. - Human voice-over: Place recordings as
<scene-id>.wavintoaudio/vo_human/;tts.py --from-dironly does resampling/light processing. - Multi-language: one
script.<lang>.json+ corresponding TTS voice; timeline can reuse the same visuals (note text length differences). - Multi-aspect ratio: same
script.json, re-init/render with differentproject.format; layout adapts vialayout(). - Branding: provide logo text/colors/footer in
project.json.brand; disabled by default.
8. Reference Documents
references/pipeline.md: Full technical pipeline,render(t)contract, CDP rendering, FFmpeg parameters, timeline data model.references/inputs.md: Extraction and OCR fallback for PDF/PPT/DOCX/URL/video.references/storytelling.md: 4 teaching structures, pacing table, copy rules.references/styles.md: Color palettes, fonts, animations, and suitable topics for the 5 style sets.references/formats.md: Aspect ratios, safe zones, portrait re-layout rules.references/audio.md: TTS options, human recording, music synthesis, ducking, loudness, single encode.references/qa.md: Acceptance checklist and automated check commands.