video-flow: Code-Rendered General Explainer Video Workflow

In a nutshell: First, think clearly about 'what to teach,' then turn that into a calculable timeline, and finally let the browser draw each frame. Final product = Canvas/HTML render(t) + headless Chrome frame capture + FFmpeg + optional TTS/music/subtitles.

This skill breaks down "a video" into 8 checkable stages. Each stage has an output and an acceptance checkpoint; do not proceed to the next step if any step is unsatisfactory.

0. When to Use / When Not to Use

Use:

Turn a PDF, PPT, DOCX, article, webpage, video, or a topic into a 1–5 minute explainer video;
Need a video where "text is clear, data is accurate, every frame is editable, and the process is reproducible" (structural advantage of code rendering over generative video models);
Need multiple versions (portrait/square/landscape) or multi-language versions;
Need narration, subtitles, music, and want the entire workflow repeatable.

Do not use:

Need real people/live-action/realistic footage → use video generation models or live footage; this skill only handles "the code layer";
Just want to cut existing footage → use editing tools/agents;
Only want to generate an MP4 without willing to define storyboards and script → the result will feel templated; it is recommended to do steps 1–2 first.

1. Hard Rules (Violate any, redo)

Every frame is a pure function of time: The page exposes window.render(t); given t, the frame is uniquely determined.
- Prohibit Date.now(), unseeded random, requestAnimationFrame driven state, CSS transition/animation, cross-frame caching.
- For cumulative animations (particles/water), push from 0 to t with fixed steps or switch to analytical formulas.
The timeline is the single source of truth: Storyboards, narration, music, sound effects, and subtitles all derive from the same timeline.json.
Storyboard before code: Without storyboard.md and script.json, do not write scene code.
Do not fabricate source facts: Numbers, years, names, and formulas must come from the source material; mark uncertainties with [to verify] and list them separately at delivery.
No branding/watermarks by default: Only if the user provides a logo/brand name; default footer only shows course/topic information.
Encode only once: Complete mixing and loudness in the WAV stage; perform only one final AAC encode; do not re-encode repeatedly.
Must pass QA before delivery: Check for dropped frames, black frames, loudness, duration, subtitles, and audio-sync; see step 8.

2. Input and Output

Input (choose any, can be combined)

Type	Processing Method
PDF	`scripts/ingest.py` extracts text; for scans/garbled text, automatically rasterize to PNG and let the model "read the pages"
PPTX	Extract title, body, and notes per slide; optionally export key slides as images
DOCX / Markdown / TXT	Directly extract as `source/source.md`
Web URL	Fetch main text; preserve source link
Video	ffprobe metadata + key frame extraction; if no ASR, human supplements content
Bare Topic	A sentence/outline from the user; first have the user confirm "facts that must appear"

Project directory contract

<project>/
  project.json          # Metadata: title/audience/goal/aspect ratio/style/voice/music/brand
  source/               # Source materials (source.md, pages/*.png, meta.json)
  storyboard.md         # Storyboard table: what each scene covers, visuals, duration, narration, key points
  script.json           # Machine-readable: scenes[] (type/script/visuals/vo/duration)
  build/                # index.html + lib.js + theme.js + scenes.js + timeline.js
  audio/                # vo/*.wav, bgm, mix.wav
  frames/               # Frame-by-frame JPEGs
  out/                  # final.mp4 / final.srt / poster.png
  qa/                   # report.md + contact sheets + key frames

Use scripts/init_project.py to generate the skeleton at once.

3. Workflow (8 Steps)

Step 0: Environment Check

python scripts/check_env.py

Required: Python 3.10+, FFmpeg/ffprobe, a Chromium-based browser, numpy/scipy. Recommended: edge-tts (online neural voice), pypdf, python-pptx, python-docx, Pillow. Install what is missing per the output prompts; if not installable, degrade (no narration/system voice/hand-written audio).

Step 1: Read Source Material, Extract "Teaching Structure"

Answer 5 questions (write into project.json's brief):

Who is the audience? What do they already know?
After watching, they should be able to say/do what? (1–3 testable goals)
What is the most easily misunderstood/hardest point?
What facts/formulas/data must appear? What are the sources for each?
Duration and aspect ratio? Platform (classroom/mobile/social media)?

Do not follow the original page order. Restructure using one of the 4 storytelling structures from references/storytelling.md:

Causal chain (recommended for science/engineering);
Problem → Solution → Validation;
Misconception → Correction → Transfer;
Timeline/process.

Step 2: Choose Style and Aspect Ratio

Select 1 set from references/styles.md (silicon / paper / blueprint / neon / swiss), and define the aspect ratio from references/formats.md. Rules:

Content leaning toward "chips/engineering/data" → silicon / blueprint;
Leaning toward "humanities/biology/history" → paper;
Leaning toward "launch/marketing/youthful" → neon / swiss;
One screen, one concept; portrait mode must re-layout, do not crop landscape to portrait.

Step 3: Write Storyboard and Script

Output storyboard.md (for humans) and script.json (for machines). Each scene must include:

type (title/concept/process/compare/formula/bars/quote/recap/custom)
kicker (subtitle), headline (one-sentence conclusion), points (≤3 items)
vo (voiceover, 1–2 sentences; leave blank if no narration)
visual (specific data for charts/tables/flows/formulas)
duration (can be empty, automatically determined by narration length)

5 hard constraints for writing copy:

On-screen text ≠ narration script: On-screen only key words/conclusions; narration explains.
At most 1 conclusion + 3 supporting points per screen; title ≤ 18 chars, body text ≤ 22 chars per line.
Change every 2–4 seconds (element/camera/emphasis), but no particle explosions.
Write a "forbidden list" first: gradient background + centered big text, fade-in for all elements, full-screen small text, meaningless glows.
Mark factual sentences with sources individually; add [to verify] for uncertainties.

Step 4: Generate Narration and Timeline

# Optional: skip if no narration, just write `duration` for each scene
python scripts/tts.py --project . --voice zh-CN-XiaoxiaoNeural --rate +3%
python scripts/timeline.py --project .

tts.py outputs audio/vo/<id>.wav (48 kHz), automatically applies light EQ/compression; supports --from-dir for human recordings.
timeline.py uses measured narration durations to generate build/timeline.js and timeline.json: scene.duration = head + voDuration + tail; for scenes without narration, use explicit duration.
Afterwards, only modify script.json and re-run these two steps; do not manually edit the timeline.

Step 5: Build the Page and Implement Scenes

Using the template:

python scripts/init_project.py --dir my-video --title "..." --format 16:9 --style silicon
# Edit build/content, generated by timeline.py; generic scenes are already implemented, just modify script.json

Generic scenes (8 types) are in build/scenes.js reading CONTENT; sufficient for most explainer videos.
For specialized graphics (chips/circuits/anatomy/maps/molecules), write build/custom.js:
```
window.CUSTOM['s3'] = function(u, D, sc, ctx){ /* use helpers from lib.js to draw */ };
```
Set the scene's type to custom in script.json.
Work on one scene at a time: first render 3 still frames (scripts/render.py --still 1.2,2.4,3.6), check for text overflow/overlap/subject size, then continue.

Step 6: Rendering

python scripts/render.py --project . --fps 30
# Render only a segment: --start 8 --duration 4
# Reuse an existing Chrome: --cdp http://127.0.0.1:9333
# Let the script launch Chrome: --launch-chrome

Outputs frames/f00000.jpg..., supports resumable rendering (skips existing frames >2KB).

Headless Chrome and sandbox: If the runtime environment prohibits Chrome's IPC/pipe, run in an allowed environment, or connect to an already launched Chrome via --cdp.

Step 7: Music, Sound Effects, Mixing, Subtitles

python scripts/audio.py --project .            # Synthesize BGM + place narration + duck + master
python scripts/srt.py --project .              # Generate out/final.srt

No narration: audio.py --no-bgm or use only user-provided music.
User provides music: place in audio/music.*; audio.py automatically loops/cuts and ducks.
Target loudness default: -14.2 LUFS, true peak ≤ -1.5 dBFS; encode audio only once.

Step 8: Encoding + QA + Delivery

python scripts/encode.py --project .           # frames+mix+srt → out/final.mp4
python scripts/qa.py --project .               # qa/report.md + contact sheet of key frames

QA must fully pass (see references/qa.md):

Duration/aspect ratio/frame rate/audio track/subtitle stream correct;
No black frames ≥0.25 seconds; key frames have no blanks, overlapped text, or overflow;
Loudness within target ±1 LU, true peak ≤ -1.5 dBFS;
First frame is not pure black; last scene has a conclusion;
Fact list matches source materials; [to verify] items listed separately.

Delivery: out/final.mp4, out/final.srt, poster.png, storyboard.md, script.json, qa/report.md, rendering time, and known shortcomings.

4. Generic Scene Types (Implemented in Template)

type	Use Case	Key Fields
`title`	Opening/chapter	`title/subtitle/kicker`
`concept`	One concept + 3 key points	`headline/points/visual`
`process`	Process/steps (3–5 steps)	`steps[]`
`compare`	Side-by-side comparison	`left/right`
`formula`	Formula + explanation	`formula/notes[]`
`bars`	Data comparison	`items[{label,value}]`
`quote`	Key quote/conclusion	`quote/author`
`recap`	Summary map	`nodes[]/takeaway`
`custom`	Specialized graphic	Write your own `custom.js`

When a new type is needed: add a window.SCENES_IMPL['type'] in scenes.js and document the data contract in references/pipeline.md.

5. Aspect Ratios and Safe Zones

Aspect Ratio	Dimensions	Safe Zone	Title Font Size	Body Font Size
16:9	1920×1080	Left/Right 96 / Top/Bottom 64	72–120	28–40
9:16	1080×1920	Left/Right 64 / Top/Bottom 160	64–96	32–44
1:1	1080×1080	All sides 72	56–88	28–40
4:5	1080×1350	All sides 80	60–92	30–42

Portrait mode requires re-layout: title at top, graphic in middle, conclusion at bottom; do not compress horizontally. The template uses layout() to switch based on aspect ratio; see references/formats.md.

6. Common Failures → Fixes

Symptom	Root Cause	Fix
Black/white screen	JS error, `render` undefined, canvas size 0	`render.py --still`, check console; explicitly set canvas width/height
Wrong font/first frame fallback font	Font not loaded	Wait for `document.fonts.ready` before setting `READY=true`; use system fallback for CJK
Same `t` gives different results / flickers during parallel execution	Used rAF/timer/random/cross-frame state	Convert to pure function; `hash(seed)`; closed-form spring
Text overlap/overflow	No per-scene still frame check	3 still frames per scene + 360px wide key frame for portrait
Audio-video desync	Audio starts from 0 vs video starts at some point; TTS per-sentence concatenation	Same timeline; whole-segment TTS; `-ss` placed before `-i`
Loudness inconsistency	Multiple loudnorm/transcoding	Only static gain in WAV stage; one AAC encode
Rendering too slow	4K/subframes/brush/WebGL	First render 960×540 sample; subframes only for fast motion; parallel workers
Chrome fails to start	Sandbox limits IPC/pipe	Run in allowed environment, or use `--cdp` to connect to an already running Chrome

7. Extension Points

New style: copy templates/styles/*.json, modify color palette/fonts/background type; add a row in styles.md.
New scene: register in build/scenes.js; or write specialized graphics in build/custom.js.
Human voice-over: Place recordings as <scene-id>.wav into audio/vo_human/; tts.py --from-dir only does resampling/light processing.
Multi-language: one script.<lang>.json + corresponding TTS voice; timeline can reuse the same visuals (note text length differences).
Multi-aspect ratio: same script.json, re-init/render with different project.format; layout adapts via layout().
Branding: provide logo text/colors/footer in project.json.brand; disabled by default.

8. Reference Documents

references/pipeline.md: Full technical pipeline, render(t) contract, CDP rendering, FFmpeg parameters, timeline data model.
references/inputs.md: Extraction and OCR fallback for PDF/PPT/DOCX/URL/video.
references/storytelling.md: 4 teaching structures, pacing table, copy rules.
references/styles.md: Color palettes, fonts, animations, and suitable topics for the 5 style sets.
references/formats.md: Aspect ratios, safe zones, portrait re-layout rules.
references/audio.md: TTS options, human recording, music synthesis, ducking, loudness, single encode.
references/qa.md: Acceptance checklist and automated check commands.

lian4211/video-flow

video-flow 是什么？

在你喜欢的 AI 中提问

文档