Pixel Cinema MV v2.0 — Software Pixel Framebuffer & Whisper-Synced MV Workflow
Create cinematic, HUD-free 16-bit/32-bit indie pixel-art narrative music videos synced to exact Whisper vocal onsets and audio beats, rendered via a deterministic Uint32Array software pixel framebuffer (270×480 → ×4 nearest-neighbor 1080×1920, 30 fps, BT.709).
Core Engineering & Aesthetic Rules (Non-Negotiable)
-
Pure
Uint32ArraySoftware Pixel Framebuffer (Zero Mixels / Zero CSS Blur)- Draw every background, light cone, weather particle, prop, and character onto an integer logical framebuffer (
270×480for 9:16 portrait,480×270for 16:9 landscape) viatemplates/pixel_fb_engine/src/engine/PixelFB.ts. - Upscale strictly
4×viaimageRendering: "pixelated"+--image-format=pnglossless intermediate frames. - Use ordered
4×4/8×8Bayer dithering for night skies, streetlamp cones, and sunrise halos; usebeginSpr()/endSpr()for automatic 1px selective-hue pixel outlines andpunch(2)for crisp2×integer close-ups.
- Draw every background, light cone, weather particle, prop, and character onto an integer logical framebuffer (
-
Whisper Exact Vocal-Onset Sync (Never Trust Unverified Internet LRC)
- Always run
scripts/transcribe_whisper_sync.pydirectly on the user's local audio track (ffmpegfloat32 PCM decode → localfaster-whisperwithword_timestamps=True). - Map every lyric cue and every individual Chinese character's
attimestamp directly to the real sung onset in the extracted clip soZpixcharacter bounce highlights fire on the exact syllable.
- Always run
-
Causal Physical Storytelling & Zero Game HUD
- Every shot follows a strict causal chain (
Trigger → Action → Contact/Feedback → Result → Transition), never static slideshow/Ken Burns panning. - Zero HUD clutter (
STAGE,BPM,SCORE,MEMORY LOGare strictly forbidden). - Keep main action in the vertical
14%–72%safe zone and placeZpixbinary-edged pixel subtitles in74%–84%(clearing Douyin/TikTok bottom caption & right icon zones).
- Every shot follows a strict causal chain (
-
Anti-Dead-Gap Integer Camera Rule
- Because
camAt()snaps to integer logical pixels (Math.round), never leave a slow scene camera static or stalled at a sine extremum for> 1.0s. - Combine steady linear camera drift (
>= 0.8 px/s) with continuous ambient motion (window rain streaks, swaying curtains, lamp dust motes, drifting leaves/snow) sofilm_gate.pyclearsdead_gap <= 1.0sandmoving_share >= 0.90.
- Because
-
Standardized Naming & BT.709 Delivery
- Tag final H.264 MP4 with
-color_primaries bt709 -color_trc bt709 -colorspace bt709. - Deliver final video named
<歌曲名字>-<时长>s.mp4(e.g.,我落泪情绪零碎-96s.mp4) alongside.srt,.lrc,_contact_sheet.png, and_qa_report.json.
- Tag final H.264 MP4 with
7-Phase Production Pipeline
Phase 1: Audio Inspection & Whisper Exact Sync
- Check
audio/for the user's licensed track (if missing, synthesize a tempo-matched temporary OST and providescripts/swap_audio.py). - Slice the target emotional highlight (
60s–96s) withffmpeg(-ar 44100 -ac 2, smooth fade-in/out). - Run
scripts/transcribe_whisper_sync.pyon the exact clip to extract word/character timestamps:python scripts/transcribe_whisper_sync.py \ --audio audio/song.mp3 --start 43.0 --duration 96.0 \ --model small --out docs/whisper_exact.json - Extract BPM, beats, and RMS energy via
song_facts.py.
Phase 2: Storyboard, Palette Constitution & Timeline
- Author
TASTE.md(5-act color palettes, character canon, anti-slop rules) andSTORYBOARD.md(4–6 fixed main scenes + causal shot table). - Generate
src/data/lyrics_timeline.json,.srt, and.lrcwith exact Whisper characterattimestamps and shot boundaries aligned to real vocal transitions.
Phase 3: Visual Design & Still Inspection
- Scaffold from
templates/pixel_fb_engine/(PixelFB.ts,Palettes.ts,CastEngine.ts,Scenes.ts,fonts.ts). - Run
npm run typecheckand renderCharacterDesignSheet+ key scene stills (remotion still) and visually inspect character proportions, bangs/eyes clarity, and subtitle safe zones.
Phase 4: 10-Second Pilot Sample Check
Render frames=0-299 (10s) first to verify Zpix font loading, audio sync, and integer pixel crispness before committing to a full render:
npx remotion render src/index.ts LuoleiPixelMV-9x16 qc/pilot_10s.mp4 \
--frames=0-299 --image-format=png --codec=h264 --audio-codec=aac --crf=16 --overwrite
Phase 5: Full Render & BT.709 Packaging
npx remotion render src/index.ts LuoleiPixelMV-9x16 output/raw.mp4 \
--image-format=png --codec=h264 --audio-codec=aac --crf=16 --overwrite
ffmpeg -y -i output/raw.mp4 -c:v copy -c:a copy \
-color_primaries bt709 -color_trc bt709 -colorspace bt709 \
"output/<歌曲名字>-<时长>s.mp4"
Phase 6: Automated QA Gate & Contact Sheet Sign-Off
Run scripts/verify_mv_v2.py to enforce:
dead_gap <= 1.0s(zero frozen spans)flicker(0luminance spikes)slideshow(moving_share >= 0.60, target1.0)delivery(bt709=True, CFR30fps,av_offset_ms=0.0)- Visually inspect
qc/final_contact_sheet.pngacross 12 sampled keyframes before delivery.