Qsnh/video-make

Make Videos with Code

Overview

One video = finalized script → sentence-by-sentence TTS → sentence-by-sentence ASR validation → timeline → HTML animation → frame-by-frame screenshots → ffmpeg composition.

Core principle: The timeline comes from the audio. First, voice each sentence, validate it, and measure the true start and end times of each subtitle segment. Then, attach all visual animations to these time points. Do not hardcode seconds into the animation. If the voice-over changes, simply re-run the timeline; the rendering takes frame-by-frame __seek(t) snapshots of the page – deterministic and reproducible.

Use cases: narration/tutorials (2–5 min), product promos (20–40 sec), knowledge videos with subtitles and chapter markers. Not suitable for: live-action editing, videos requiring real footage, or pure music videos.

Pipeline

Each step has a gate. Do not proceed to the next step until the current one passes.

#	Step	Output	Gate	Reference
0	Initialize	Working directory	bash scripts/init.sh <dir> succeeds	Below
1	Topic selection, fact-check, script	script.md, audio/make_script.py	Facts checked (run experiments first); user explicitly approves script before starting voice-over	references/script.md
2	Voice-over	audio/raw/NN.wav → audio/clips/NN.wav (1.3x)	Each sentence passes ASR; problematic sentences re-recorded	references/voice.md
3	Timeline	build/narration.wav, build/timeline.json	Subtitle segmentation aligns with speech	references/timeline.md
4	Assets	Real interface screenshots, real terminal output	Consistent with narration; use example.com for example domains	references/capture.md
5	Composition	comp/index.html	Review each frame composite individually	references/composition.md
6	Rendering	dist/video.mp4	Duration matches timeline, loudness -14 LUFS	references/render.md
7	Cover, copy	dist/cover-bili.png, dist/cover-douyin.png, dist/copy.txt	Covers within platform crop-safe zones	references/publish.md
8	Delivery, cleanup	Send to user; archive source files	User confirms save before cleanup	Below

Initialization

Run bash scripts/init.sh ~/work/<project>/video-<episode> to copy template/ to the working directory, copy scripts/ to tools/, and install playwright-core. The template includes a 30-second demo that runs out of the box. Before starting, replace these example contents:

• script.md, audio/make_script.py: Your episode's script; define chapters only once in make_script.py's CHAPTERS – pauses between chapters, chapter bars, and chapter timestamps are derived from it.
• audio/direction.txt: TTS director prompt describing the [scene].
• asr-prompt.txt: The episode's terminology list (standard notation, with pronunciation if needed).
• comp/index.html: Scenes and animations; comp/cover-*.html for cover content.

Common Commands

Execute inside the working directory:

Command	Purpose
python3 audio/make_script.py	Generate audio/script.json and audio/lines.json
python3 tools/tts.py audio/lines.json audio/raw	Voice each sentence; reuse existing audio for identical text, just re-run after script changes
bash tools/speed.sh audio/raw audio/clips 1.3	Change speed without pitch shift
python3 tools/asr.py audio/clips --en	Sentence-by-sentence recognition: plain / term-prompted / English mode
python3 tools/takes.py 3,7 3	Record 3 takes for sentences 3 and 7, then score them
python3 tools/takes.py audio/var.json 2	Record 2 takes for each variant of the same sentence
python3 tools/tts.py audio/lines.json audio/raw --adopt 3 audio/takes/03-0-1.raw.wav	Adopt a specific take
python3 tools/build_audio.py audio/clips build audio/script.json	Concatenate voice-over, generate timeline
python3 tools/align_chunks.py audio/clips build/timeline.json audio/script.json	Align subtitle segments to actual speech
node tools/render.mjs --timeline=build/timeline.json --stills=2,8.5,20	Extract frames to build/stills (clears first)
bash tools/contact.sh build/stills build/sheets	Create 2x2 grids, view with Read
bash tools/render-par.sh build/timeline.json build/narration.wav dist/video.mp4 8	8-process parallel rendering and mixing
node tools/render.mjs --html=comp/cover-4x3.html --width=1440 --height=1080 --png=dist/cover-bili.png	Generate cover
python3 tools/bili_kw.py "nginx configure ssl certificate" "baota https"	Topic research: Bilibili top 6 videos by keyword views

Dependencies: ffmpeg, Node 18+, local Google Chrome, Python 3 + requests + openai-whisper (small model). TTS uses Xiaomi MiMo. API key read from environment variable MIMO_API_KEY or a file pointed to by MIMO_ENV_FILE; if neither exists, ask the user. To use a different TTS, just ensure each sentence produces one WAV file.

Most Common Mistakes

• Voicing without approval. Send the script to the user first; "continue"/"start"/"okay" counts as approval. If wording is changed for pronunciation, list every change at delivery.
• Writing subtitles and voice-over as one text. Subtitles are for reading (Nginx, 443, 1.3); voice-over is for TTS (Engine X, four-four-three, one point three). In make_script.py, use P(subtitle, pronunciation) to keep them separate; the concatenated pronunciation must equal the full TTS text of that sentence.
• Only listening to the full mix. Full-sentence ASR misrecognizes English terms; always validate sentence by sentence. Score takes only by plain score and --en result. The term-primed mode (--prime) can wrongly "correct" mispronunciations – see voice.md for judging rules.
• Hardcoding seconds in animations. After the first sentence starts, always use T(sentence, segment) for animations; speed changes or re-recording will shift timings.
• Empty page start. If the page is blank for 2–3 seconds after a card appears, it looks stuck; content must show up within the first subtitle segment of the chapter.
• Multiple fill: both animations on the same element, or duplicate IDs. WAAPI won't error, but the element may disappear or freeze in the wrong state – see composition.md.
• Using file:// to open the page for rendering. CSS masks, etc. won't work; render.mjs includes a local HTTP server.
• Cover cropped by platform. Bilibili cuts one image into both 4:3 and 16:9; Douyin 9:16 is cropped to 3:4 – place content only in the safe zone.
• Directing traffic. Do not include real URLs, QR codes, or contact info in the video, cover, or copy. Do not say "you can check out/search for". example.com and similar reserved domains may appear in demo interfaces and terminal output. For cross-platform videos, do not use a specific content platform as an example. Do not mention prices in promos.
• Reporting success before checking output. Send progress messages only after the command finishes and you verify the output.

Delivery and Cleanup

1. Delivery: final video, Bilibili 4:3 cover, Douyin 9:16 cover, dist/copy.txt (title, tags, description, chapter timestamps). By default, the Bilibili video is also posted to Douyin and other platforms, so vertical cover and Douyin copy are included. If the user explicitly states only one platform, deliver only that set. List every wording change made for pronunciation, and note any uncertain timings for the user to review.
2. If the file exceeds the chat tool's limit (often 30 MB), send a crf 22 preview first and explain where the high-quality version is located. If sending a large file times out and you resend, warn the user they may receive two copies.
3. After the user confirms assets are saved: archive reusable source files (comp/ and its assets, audio/make_script.py, variant specs, script.md, copy.txt, direction.txt, asr-prompt.txt, screenshot scripts, lab/) to the user's template library (if no convention, ask the user where to store, e.g., ~/work/<project>/video-template/<episode>/). Use diff -r to verify consistency, then delete the working directory, temporary browser profiles, containers, dev servers (kill by PID after verifying the command line), and any frontend dependencies installed only for screenshots. Keep the whisper model cache and Docker images.