Pixel Character Beat-synced Music Video

vvai-pixel-mv · Pixel Character Beat-synced MV

Order is fixed: Storyline → Storyboard (user reviews, revises, can go back and forth multiple times) → User approval → Render. Without user approval of the storyboard, no rendering occurs.

A complete example is in assets/examples/: a song's storyline and story.json, 7 segments, 3 characters, with desktop window.

Preparation

• PY: Python 3.10+ with librosa, faster-whisper, numpy, Pillow installed (see INSTALL.md). Use the Python path from the user's CLAUDE.md or memory; if not written, first confirm that python can import these packages, otherwise ask the user.
• MV: PY -X utf8 <this skill>/scripts/pixel_mv.py.
• Requires ffmpeg, Node. Rendering defaults to npx --yes [email protected]; to use a different CLI, set the environment variable HF_CLI, separate multiple parameters with |.

Workflow

1. Collect materials. Must have audio, timed lyrics (srt or lrc), and song title.
   • Optional: start/end cut times, color grade (base / neon / classic / retro), intensity (dj / standard / soft), character image(s).
   • Also ask about the story: does the user have a plot in mind? If not, propose an idea in step 3.
   • If no lyrics file is provided, ask the user for one; do not use transcription to guess. If transcription cannot voice-activate a section, ask the user first; do not treat it as an intro. In Episode 1, the verse was incorrectly judged as an intro.
2. Create project: MV new <project-dir> --audio <audio> --lyrics <srt/lrc> --title <song-title> [--start sec --end sec] [--look base] [--character image].
   • This step trims the audio, detects beats and energy climax points, and aligns lyrics word by word.
   • Also generates an initial draft story.json with keyword-based scene changes, and a storyline.md template.
   • Place the project in the directory specified by the user; if the user has a fixed series directory (in CLAUDE.md or memory), place it there.
3. Write the storyline. Fill in storyline.md following Storytelling Method: one-sentence story, characters, motif, acts, line-by-line. Send to user for approval.
4. Write story.json. Translate the storyline into story.json.
   • See Format for fields, available actions, lyric styles, transitions in Interaction Reference.
   • It's fastest to follow assets/examples/bad-girl.story.json.
   • After writing, run MV check <project> until there are no errors.
5. Generate storyboard for user review.
   • Run MV board <project> --frames: capture 1–3 images per segment, generate storyboard.md with overview, per-segment characters, story actions, lyric sentence numbers and appearance style, transitions, camera, and beat points.
   • Send the storyboard.md and contact sheets from storyboard/frames/ to the user using SendUserFile.
   • Tell the user what can be changed.
6. Revise until satisfied. User requests changes, modify story.json, then run board again.
   • Version number auto-increments by 1, old versions saved in storyboard/history/.
   • Can go back and forth multiple times; each time send the new storyboard to the user.
7. Render after approval.
   • Only when the user explicitly says "render", run MV approve <project>, then MV render <project> --sample 12.
   • Vague replies ("okay", "up to you") do not count as approval.
   • render checks story.json fingerprint; if file was modified after approval, rendering is refused.
   • If output exceeds 30 MB, compress an additional _preview.mp4 for mobile.
8. Delivery. Self-check by viewing a 1 frame/second montage: story understandable, scenes, lyrics, transitions, title all present, duration matches original audio, and audio track is present.
   • If user wants changes after delivery, go back to step 6: modify story.json → generate new storyboard → re-approve → re-render.

Key Rules

• Story before effects. Every line must have at least one story action or one lyric interaction; story actions must land on the keyword's time; transitions must have a narrative reason. See Storytelling Method.
• Scene changes: When no story dictates, default to "desktop main + imagery inserts", see Storyboard Rules. When there is a story, follow the acts; realization/flashback sections use the desktop window perspective.
• Characters:
  • Default Claude character as "me".
  • Other characters declared with cast (girl, rival, friend, ghost).
  • When the user provides an image, run MV character <project> <image> to convert it into a 16×16 pixel character.
• Beat-syncing:
  • Character lands exactly on each beat, squashes on landing, water ripples spread from under their feet.
  • Each beat has camera shake and color flash; strong beats add RGB split.
  • Full-screen white flash only used on transitions and drops, no more than 3 times per second.
• Startle reaction pack: max 2 times per video, only on real twists.
• Lyrics body:
  • Appears only in the scene and project files; chat replies and storyboard only write sentence numbers.
  • Song copyright is the user's responsibility; remind them before publishing.

Command Overview

Command	Purpose
new	Create project: analyze + align + draft + story
story	Generate storyline.md template (--force to rebuild)
check	Check names, times, windows in story.json are recognized by the engine
board [--frames]	Generate storyboard (version N), save history
approve	Record the user-approved version; subsequent render will allow it
render [--sample 12] [--name filename]	Render; --force skips approval check, only use when user explicitly says so
sync	Only sync story.json → assets/story.js (debug use; board and render auto-sync)
upgrade	Upgrade old project to current template engine, old files backed up with .before-upgrade
character	Character image → pixel character
analyze / align / draft	Steps of new, run individually to redo

Customization Beyond Templates

Template engine v2 already covers the following, see Interaction Reference:

• 27 scenes, 16 basic actions;
• 23 story actions;
• 7 lyric styles, 6 entrances, 9 exits;
• 7 transitions;
• Desktop window.

If still insufficient, add in that project's assets/engine.js; do not modify the templates in this skill. Reference Episode 1 Techniques. To incorporate new segments into the template, first confirm with the user. When adding, update all these files:

• Template engine.js;
• BEAT_NAMES list in pixel_mv.py, along with Chinese names ZH_* used in storyboard;
• interactions.md.

Scene Library

• Data in assets/scene-library/scenes.json.
• To add a scene, update both: register keywords and interactions in scenes.json, and write the drawing method in the template engine.js's SCENES.