Remotion Beat-Sync Short Video Pipeline

What does Remotion Beat-Sync Short Video Pipeline do?

Goal: Given a topic and a BGM, produce a 88–90s / 16:9 / no voiceover / bilingual subtitles / every scene cut on phrase downbeat final video, plus three cover versions, a publishing package, and a fact-check list. Everything is code; changing subtitles means changing strings, and re-rendering takes ~35 minutes.

When to Use

• "Turn this topic into a Douyin knowledge video / game theory commentary"
• "Make a beat-sync video with [BGM]"
• "Turn this novel passage into a video" (fictional IP, see references/fact-check.md)
• Needs complete delivery: final video + covers + publish copy + fact-check

When NOT to Use

• Need voiceover narration (this pipeline is voiceover-less subtitle storytelling; voiceover would require TTS track alignment first)
• Need live action / AIGC footage mashup (this pipeline is pure code SVG graphics)
• Only need image + text, no video

---

Part 0: Style DNA (Read First – All Based on Real Testing)

Dimension	Rule	Why
Aspect Ratio	1920×1080 / 30fps (portrait opens separate Composition)	Account locked to 16:9; data shows higher content efficiency
Duration	88–90 seconds (~42 bars)	Falls in the "60–90s per episode" test range
Audio	No voiceover, BGM + synthesized sfx	Style requirement; also avoids TTS alignment cost
Subtitles	Bilingual two lines, Chinese ≤22 chars, each line stays ≥2.2s	Account style (Douyin general recommendation is single language; this is a channel choice)
Hook	Second 1 = a complete conflict sentence + a concrete object	2-second dropout rate is the biggest leak
Narrative	One character + one problem + one causal chain	Avoid "concept gallery" and "group montage"
Structure	Hook → Title → Rule → Trap → Counterattack → Backfire → Reveal → Elevation → Ending	See references/script-template.md
Rhythm	Each scene hard-cuts from phrase downbeat; key actions land on beats	Sync comes from musical structure, not manual alignment
Visuals	Dark background + metal/wood/gold textures + large fonts + few labels	Readable on phone thumbnail
Safe Zone	Scene elements y ≤ 850; subtitles y≈870–1000; top-left header y 50–95	Most common mistake in real testing
Ending	Must provide a screenshot-able checklist (save point) + one question	Save rate is the biggest weakness
Rigor	Fictional IP must be marked "illustration only"; facts layered by source	See references/fact-check.md

---

Part 1: 8-Step Workflow (Execute Step by Step, No Skipping)

Step 1: Project Initiation (Topic Selection + Confirmation)

1. If the user hasn't given a topic, first offer 3 candidates (each specifying: one-sentence core / second-1 hook / why it can be visualized / save point / risk), let the user choose.
2. Candidates must verify existence before submission: fictional IP must be found by chapter name or entry; knowledge-type must have a source.
3. After the user picks, collect the original source material for that topic (original paragraphs are best; second-hand summaries must have source hierarchy, see Step 8).

Step 2: Audio Track & Beats (Determines Everything Timing)

# ① Extract audio (mp4/mp3 both supported; no system ffmpeg → auto uses imageio-ffmpeg binary)
python scripts/extract_audio.py input.mp4 work/            # → work/audio.f32 + work/music.mp3
# ② Beats / downbeats / segments / tonality
python scripts/analyze_beats.py work/audio.f32 src/beats.json
# ③ Scan energy, select window (key step)
python scripts/scan_energy.py src/beats.json work/audio.f32 --seconds 89

• Output gives: START_BAR / END_BAR, per-bar energy graph, percentage of silence dip within window.
• Place the reversal on the first downbeat after the silence dip – this is the most valuable second of the entire video.
• Don't cut off the ending decay segment (it leaves a dead tail).
• If a song has been used by the series before, prioritize an unused second half to avoid visual repetition.

Step 3: Write Script (src/script.ts)

• Each scene declares bars (number of bars it occupies); the sum of all bars must equal END_BAR − START_BAR (otherwise placeScenes scales proportionally and warns).
• Subtitles are written in subs: {zh, en, at}; at is the beat number within the scene (can be half-beats like 4.5). One line per 4–5 beats is most comfortable.
• Sound effects are written in cues: {id, beat, sfx, vol, impact}; impact:'big' = white flash + screen shake + chromatic aberration + music sidechain ducking for 40 frames.
• Rhythm budget: number of sentences × 2.5–3s ≈ total duration. 89 seconds ~ 30 sentences.

Step 4: Visual Kit & Scenes (src/*/kit.tsx + src/*/scenes.tsx)

• Write all kit parts before writing scenes: a topic usually has 6–10 parts (e.g., iron box / lock / key / shredder / microwave / character silhouette / statue / lamp / lottery ticket / counter / spirit coin).
• All parts are pure SVG with coordinates fixed on 1920×1080, reused via translate/scale.
• Scene components only arrange parts by beat: const bf = (n) => beatFrame(scene, n); entries always use easing progress like pp(f, bf(n), 8).
• Check list in references/visual-kit.md.

Step 5: Project Setup

One project houses multiple episodes: src/<episode>/{script.ts,timeline.ts,fx.tsx,kit.tsx,scenes.tsx,Main.tsx}, register a second <Composition> in src/Root.tsx, each using its own beats.json and music.mp3.

• The second episode must have its own beat shell (Shell / ImpactFlash in fx.tsx must use the episode's own cueFrame, otherwise impact points will mismatch).
• Audio uses <Audio startFrom={OFF}>; content wraps <Sequence from={-OFF}> so scene internals always use "music frame" timing.
• See references/pipeline.md for details.

Step 6: Still-Frame Self-Check (Don't Render Without This)

node <node> node_modules/@remotion/cli/remotion-cli.js still <Comp> out/s1.png --frame=<scene mid frame>

Render one frame per scene, visually check four things:

1. Does subtitle overlap scene elements (most common)
2. Element overlap (label on label, label on graphic)
3. Safe zone (element y ≤ 850)
4. Text readability at phone thumbnail size (Chinese ≥24px, main title ≥120px)

Step 7: Render & Final Check

node <node> node_modules/@remotion/cli/remotion-cli.js render <Comp> out/<name>.mp4 --codec=h264 --crf=18 --concurrency=6
python scripts/check_mp4.py out/<name>.mp4      # duration / track count / resolution / audio params

check_mp4.py doesn't need ffprobe: it directly parses mp4 boxes, confirming vide + soun dual tracks, correct duration, 1920×1080, AAC 48kHz.

Step 8: Delivery

python scripts/make_covers.py --src out/<still frame>.png --crop x0,y0,x1,y1 \
    --title main title --kicker subtitle --sub one-liner --out-prefix out/Cover_XXX

Delivery package (4 items):

1. Final mp4 (copied to workspace root, filename includes topic)
2. Three cover versions (16:9 / 3:4 / 9:16) – never use automatic frame cropping
3. Publishing package (body text + pinned comment + cover plan + 24-hour judgment table), template in references/cover-and-publish.md
4. Fact-check list (essential for fictional IP) + shot list (timecode table)

---

Part 2: Iron Rules (All Discovered Through Real Testing)

1. Frame 1 is not a title card, it's a hook: a complete conflict sentence + a concrete object ("The hostage is locked in the microwave, the countdown is ticking").
2. Subtitle safe zone: scene elements always within y ≤ 850, subtitles at 870–1000. Check this first during still-frame self-check.
3. One line of subtitle ≤22 chars, stays ≥2.2s; Chinese main line + English translation two lines.
4. Ending must be screenshot-able: a three-step/three-signal checklist + "Save this for later"; endings with only emotion won't get saves.
5. Silence dip = reversal point: scan_energy.py tells you which bar the dip is in; place the strongest reversal right after it.
6. Big impact is scarce: impact:'big' triggers white flash + screen shake + chromatic aberration + music ducking; 5–8 times per video is enough, more becomes numb.
7. No uniform sliding: all entries use easing (Easing.out(cubic)); labels exit with slight displacement + scaling.
8. Fonts: don't use "subset fonts trimmed by old copy" (new copy will miss characters) – use system font stack 'Noto Sans SC','Microsoft YaHei' directly.
9. public/ is copied entirely: only keep used sfx (e.g., 148 → 16 files, saves 60MB per render).
10. Remotion's ETA is inflated early: don't be scared by "3 hours remaining"; stable real speed ~1.35 fps, 2700 frames ≈ 35 minutes.
11. Three-layer annotation for fictional IP: original text / reader compilation / my speculation – only write consensus of first two layers in the video, rest goes to fact-check list.
12. Don't hardcode numbers in the video ("four zodiacs" → "several zodiacs"), don't hardcode chapter numbers ("Chapter 359" → "around Chapter 359").

---

Part 3: Directory Conventions

<project>/
├── public/music.mp3              # BGM for this episode
├── public/music.f32(+.json)      # For beat analysis (can be deleted)
├── public/sfx/*.wav              # Timbre-matched sound effects
├── src/beats.json                # Output from analyze_beats
├── src/script.ts                 # Master copy list (change content here only)
├── src/timeline.ts               # Scene→downbeat mapping + OFF/TOTAL
├── src/Root.tsx                  # Register Composition
├── src/fx.tsx                    # Screen shake / white flash / chromatic aberration / bilingual subtitles
├── src/<episode>/**              # Second+ episodes: independent script/timeline/fx/kit/scenes/Main
├── tools/                        # Scripts for this skill
└── out/                          # Still frames + final video + covers

Part 4: Reference Files

• references/pipeline.md – Environment setup and full command collection (includes all pitfalls of this machine: no ffmpeg / scipy / npm)
• references/script-template.md – Script structure template + beat conversion table + real timecodes for two sample episodes
• references/visual-kit.md – Visual part specifications, safe zone, color scheme, common overlap pitfalls
• references/sfx-and-mix.md – Sound effect synthesis (by tonality), sidechain ducking, loudness
• references/fact-check.md – Fictional IP verification method and "reader paraphrase" annotation rules (including available Bilibili API)
• references/cover-and-publish.md – Three cover version specs + publishing template + 24-hour judgment
• examples/End_of_Ten_Days_Two_Episodes.md – Structure, data, and decision records for two real cases

Part 5: Capability Boundaries (Be Honest with Users)

• This pipeline does not generate voiceover; for narration, you must separately create TTS and re-align subtitles.
• In network-restricted environments (many domains blocked by DNS), details of fictional IP rules may only be available through second-hand sources – in that case, the fact-check list in Step 8 must be used; do not fabricate.
• Rendering is compute-intensive: ~35 minutes for a 89-second 1080p video on a 4-core laptop; don't promise "renders in minutes."
• Copyright: BGM extracted from Douyin videos is for personal reference only; when publishing, please use the platform's in-app music library or original sound card.