What does Lyric-Driven Code MV do?
Turn the emotion of a song into a watchable game performance: lyrics decide what to draw, music decides when to emphasize. Express meaning through character goals, actions, feedback, and outcomes; progress bars, items, and transitions serve this causal chain.
From Song Background to Characters and Story
When creating new or rewriting the narrative, if either characters or plot are unspecified, proactively use the current assistant's web search tool—don't ask the user to make up a story first. If only characters are specified, keep the characters and fill in the plot; if only the plot is specified, keep the plot and fill in the characters. If both are already specified, or if you're only modifying subtitles, remixing, or exporting on an existing project, skip this step.
- Identify the work. Cross-reference the song title, lyricist, composer, performer, and recording version, then search for "song title + creation story / interview" and keywords in the original language. Distinguish between the original work's background and the cover artist's interpretation; only ask about the version if different works share the same name and would change the narrative. Only search publicly available identifiers; do not upload private recordings or unpublished lyrics. Do not search if the user has disabled internet access.
- Find verifiable facts. Open creator interviews, official album liner notes, or contemporaneous reports; do not treat search summaries as sources. In the project production notes, record the title, link, supported facts, and the shot that uses them. If no source links a real-life experience to the work, do not present it as creative motivation; the "I" in the lyrics does not automatically equal the singer.
- Let background drive action. First, summarize the protagonist's goal, obstacle, choice, and outcome in one sentence, then arrange visible actions and consequences as the actual song sections unfold. By default, you may use original characters to translate real-life backgrounds, clearly distinguishing fact from adaptation. For example, if only the creation's connection to relocation is confirmed, let the fictional protagonist pack old items in the verse, try to mail back the keys in the chorus, and decide to leave the keys behind in the outro. The relocation has a source; the keys are a prop of the screenwriter, not a real person's anecdote.
- Deliver the final video. Write characters into
actors, and write actions and consequences into each shot'sobjective,action,feedback,outcome, and actually render them in the drawing method corresponding toscene.id. Repeated choruses should show changes resulting from previous choices. Continue with the following steps for preview, rendering, and acceptance; deliver at the length requested by the user, and don't stop at a background summary. When unable to connect or find reliable materials, explain the limitation, create based on lyrics or instrumental emotion, and label it "original interpretation, not based on real experience."
Detailed templates for character settings, research records, and story arcs can be found in From a Song to Continuous Storyboards. For song analysis, step-by-step prompts, action previews, and delivery, continue reading Pixel MV Phased Production. If the finished video has issues with characters, climax, clarity, or transitions, first read Advanced Review and Targeted Optimization, make targeted modifications and re-evaluate using the same scale.
Workflow
Do dependency pre-check first. Pillow is only used by Python scripts under scripts/ and is declared in requirements.txt of the skill directory. First run python -c "import PIL" to check and report if missing; if installation is needed, first explain to the user about Pillow, its version, purpose, and installation command; execute only after getting consent. Do not include pip install in automatic pre-checks.
Node side has zero third-party packages: .mjs files under scripts/, assets/scenes/, assets/templates/starter/ only import node: built-in modules; there is no package.json inside the skill directory; no npm install needed, works offline—the same applies to validate-storyboard.mjs and hud-density.mjs. FFmpeg/FFprobe are system binaries; check them in step 4.
- Fix inputs and evidence. Confirm the song file and its version, proofread lyrics, crop range, aspect ratio, character settings, and delivery scope. When finding the song yourself, first read Audio and Rights, check the usage conditions of the work and its recording. When analyzing reference videos, separately record observable facts, audio verification results, and implementation inferences; only claim to have "seen the entire video" after completing a full time-span check. Mark as unknown when the author's tools have no source code or declarative evidence.
- Establish two timelines. Mark phrase starts and ends, sections, important accents, and pauses; the semantic table records character relationships, imagery, and actions. Read Timing and Storyboard, unify the final video coordinates with the original track offset. When providing original narrative subtitles for instrumental pieces, explain the role of the text; don't treat it as a transcribed vocal. If FFmpeg is already available, first use
scripts/analyze-audio.mjsto generate offline energy, pauses, onsets, and BPM/phase candidates; when needing to check shots and subtitles coaxially, read Sync Candidate Chart. Do not treat candidates as lyric alignment; a draft of transparent annotations can be delivered even without being able to listen to the audio. - Check character distinctiveness first, then choose devices and scenes. For three or more main characters, first create character cards and equal-scale pure black silhouettes for comparison; after removing names, colors, and small props, each person must still be distinguishable by at least two of: head/outfit silhouette, body proportions, or posture. The minimum actual display size for group shots and close-ups for single characters must both pass; if they fail, redraw the characters first—do not mask the issue with color changes, small accessories, or filters.
- Choose devices first, then choose scenes. First determine the chat, game system, timeline, counter, chapter card, perspective, or material stage; then for each shot, write Goal → Action → Feedback → Outcome. Use
chapters[]to record title cards, relationship stages, and settlement; usescene.hud[]to express numerical values, status, zeroing, damage, or failure as emotional feedback; when the same composition changes seasons, materials, or eras, writematerialVariant. Shot length is determined by information density, with no fixed maximum; it is recommended to have at least one readable change every 8-12 seconds, though long takes can also be used for waiting and pauses. Refer to Devices and Chapters and HUD and Subtitles; when needing a ready-made skeleton, pick one of the five JSONs (private message, load save, inventory, settlement, dual-axis slider) from Device Templates and rewrite it—don't start from scratch. Arrange recurring imagery with its first appearance, variation, and reappearance; repeated choruses add information or consequences. Preserve character identity, costumes, and the provenance of props. For pixel art, camera, subtitles, and two-character narratives, read Pixel World and Performance. - Choose an implementation route. Follow the framework specified by the user and the existing project. If no framework is specified and a minimal runnable project is needed, copy the entire
assets/templates/starter/; it already contains the shared core and runtime files; after copying, you can independently runnode export.mjs --help. HTML animation projects can use HyperFrames; React projects should prefer Remotion. Before implementation, read Dependencies and Delivery, list required, existing, and optional items; before installing packages, downloading browsers/models/fonts, using APIs, or paid services, first explain and continue with existing licenses. - Complete a representative segment. For new works, first deliver character cards/silhouette sheets, full-size action strips, and about 10-15 seconds of preview footage; if there are three or more main characters, the preview must include both a single-character identification shot and a group shot, confirming that a single character close-up is still recognizable when cut back to a group shot. The preview must also include lyric imagery tasks, feedback changes, and transitions; if the user specifies a different length or is modifying an existing project, follow their scope. The storyboard JSON serves as the single source of truth for duration, lyrics, scene boundaries, transitions, and music status; the code provides drawing methods per
scene.id. Each frame is evaluated directly from time; particles use fixed seeds. The outgoing transition occupies the tail of the previous shot; the next shot has a fixedlocal=0pre-roll, not increasing the total duration. Execute structural validation andscripts/qa.mjsfor pixel comparisons of duplicates, inversions, and out-of-order. - Expand and verify. First preview the representative segment, check action peaks, sentence ends, transitions, lyric panels, and camera edges; then expand to the authorized scope. When referencing pixel lyric videos, use
captionStyle.mode:"scene-panel"at the storyboard top level, so that the browser preview, ASS export, and chapter cards all share the same warm white/stroke/pixel panel parameters; if not provided, keep the default bottom subtitle for compatibility with older drafts. After export, execute FFprobe, full decoding, contact sheet, and keyframe checks; when playback is available, play through completely, recording each actual verification item item by step. If listening is not possible, deliver asdraft, clearly indicating pending alignment points—do not claim audio-visual sync has been verified.
Music Evidence and Delivery Stages
music.status | Condition | Delivery Obligation |
|---|---|---|
illustrative | Self-timed points, placeholder track, or original demo | Display "demo timing" at the beginning, record presentationNotice:true; explain it is not an alignment result corresponding to a real song |
draft | Audio available, machine analysis or annotations without listening | Write the estimation method in evidence.method, error sources in limitations; record specific time intervals and unconfirmed content in pendingChecks[]; allow delivery of draft final video |
verified | Manual check of relevant audio, lyrics, and key alignment points | Record evidence.reviewer, a complete timestamp with timezone reviewedAt (e.g., 2026-11-17T08:00:00Z); write the verification scope in the delivery note |
Automatic analysis cannot be upgraded to verified. When word-level timings are evenly divided by sentence length, note the subtitles as visual fill; half/double-speed ambiguity of BPM, accompaniment onsets, sustained notes, and source offsets can all cause errors. When audio is missing, first deliver illustrative storyboards or demos; do not wait indefinitely because listening is not possible. Status and licensing are two independent conditions; a draft status cannot replace a release license.
Drawing Conventions
Draw pixel scenes in the order: background → decorations/glows → subjects → foreground occlusion. Decorations must be drawn before subjects to avoid glows, reflections, or particles covering the subject's outline; in the starter's FB, use fb.layer({ glow: layer => ..., subject: layer => ... }) to enforce this order; fb.glow(x, y, radius, color) only draws the underlying glow. The composition should check subject proportion, overall brightness, and single-color area; scripts/qa.mjs will flag when analyzable frames have subject proportion estimated below 0.5%, average brightness below 30/255, or a single color exceeding 60%. These are screening hints, not substitutes for frame-by-frame review.
For example, a neon sign shot: first paint the night scene, then the glow and the sign subject, and finally the foreground occlusion. The two callbacks of layer both receive the same FB instance; glow executes first, then subject:
fb.rect(0, 0, 384, 216, nightSky); // background
fb.layer({
glow: layer => layer.glow(258, 72, 32, [255, 62, 164], { alpha: 0.65 }),
subject: layer => {
layer.rect(231, 52, 54, 38, [28, 26, 42]); // sign board
layer.rect(237, 59, 42, 4, [255, 110, 194]); // sign text
layer.rect(250, 92, 16, 42, coat); // main character
},
});
fb.rect(0, 190, 384, 26, foreground); // foreground occlusion
Templates and Commands
Copy original 12-second storyboard and replace content. See starter README for the minimal project, which comes with a paper boat, lights on both banks, an outro transition, high-resolution subtitles, and a locally generated original rhythm. export.mjs --storyboard reads the storyboard configuration; for real songs, first update project.audio and evidence; --audio can only relocate the same material, adjust per scene.id or register drawing methods. Export saves the actual audio track hash and source record; when an external audio track has a credit, generate a short end-credits attribution; still verify the full attribution obligations before release. Subtitles and music status prompts are generated from JSON.
The following commands run from the video project root; replace <SKILL_DIR> with the absolute path of the skill installation directory; storyboard/material inputs also use the corresponding paths; all work/ output is relative to the video project root. Do not run file-writing commands from the Skill installation directory:
node "<SKILL_DIR>/scripts/analyze-audio.mjs" path/to/song.wav --start 37.2 --duration 12 --out work/audio-analysis.json
node "<SKILL_DIR>/scripts/validate-storyboard.mjs" "<SKILL_DIR>/assets/storyboard-example.json"
node "<SKILL_DIR>/scripts/validate-storyboard.mjs" path/to/storyboard.json --delivery
node "<SKILL_DIR>/scripts/visualize-sync.mjs" path/to/storyboard.json work/audio-analysis.json --out work/sync-review.svg
node "<SKILL_DIR>/scripts/qa.mjs" "<SKILL_DIR>/assets/templates/starter/frame.mjs" --duration 12 --fps 30 --storyboard "<SKILL_DIR>/assets/storyboard-example.json" --out work/qa.json
node "<SKILL_DIR>/scripts/verify-sfx.mjs" --audio work/mixed.wav --storyboard work/storyboard.json --window-ms 50 --out work/sfx-evidence.json
node "<SKILL_DIR>/scripts/lrc.mjs" --lrc work/song.lrc --offset 12 --storyboard work/storyboard.json --out work/lyrics.json
node "<SKILL_DIR>/scripts/new-storyboard.mjs" --preset narrative-12 --duration 90 --lyrics work/lyrics.json --out work/storyboard.json
node "<SKILL_DIR>/assets/templates/starter/export.mjs" --storyboard "<SKILL_DIR>/assets/storyboard-example.json" --out work/demo.mp4
node "<SKILL_DIR>/assets/templates/starter/export.mjs" --frame-time 5.55 --frame-out work/world-frame.png
node "<SKILL_DIR>/assets/templates/starter/export.mjs" --storyboard path/to/storyboard.json --audio path/to/song.wav --out work/mv.mp4
Measure rhythm, check audio-visual sync, inspect safe zones, and create pixel title cards:
node "<SKILL_DIR>/scripts/hud-density.mjs" path/to/storyboard.json --max-gap 12 # readable change density and blank windows
python "<SKILL_DIR>/scripts/sync-audit.py" path/to/storyboard.json \
--analysis work/audio-analysis.json --video work/mv.mp4 # one command to check all timing issues before export
python "<SKILL_DIR>/scripts/layout-check.py" path/to/storyboard.json \
--hud "Archive:56,44,596,200" --hud "DivineFavor:1256,44,1794,200" \
--log "56,700,640,880" --subtitle-band "360,930,1560,988" \
--safe "right=124,bottom=92" --scenes "s09-ledger:full-ui"
python "<SKILL_DIR>/scripts/timeline-sync.py" song.wav --storyboard storyboard.json -o work/timeline.png
python "<SKILL_DIR>/scripts/safe-area.py" --video work/mv.mp4 --at 3 10 -o work/check.png
python "<SKILL_DIR>/scripts/film-sheet.py" shots/<project-name> -o sheet.jpg --cols 3
python "<SKILL_DIR>/scripts/pixel-font.py" --sample work/title-sample.png
python "<SKILL_DIR>/scripts/minimal-renderer.py" path/to/storyboard.json --probe 0 120 300 --out work/probe
node "<SKILL_DIR>/scripts/sfx.mjs" --from-storyboard path/to/storyboard.json --out work/sfx.wav --report work/sfx-report.json
node "<SKILL_DIR>/scripts/sfx.mjs" --list
node "<SKILL_DIR>/assets/templates/starter/export.mjs" --storyboard path/to/storyboard.json --audio path/to/song.wav --sfx --out work/mv.mp4
scripts/lrc.mjs adds LRC time points by --offset seconds to the segment clock, inferring end line by line; when --storyboard is provided, it supplements allowSceneSplit and its reason for cross-shot lyrics. scripts/new-storyboard.mjs generates a 12-shot skeleton with placeholders; lyrics can be imported directly from JSON. 12 drawing method assets are located at assets/scenes; each factory is configurable for subject positioning, color palette, and shot duration. --sfx-script PATH allows a copied-out starter to use a custom sound effect module without modifying source code. By default, the final video does not burn in a draft watermark: draft also does not overlay text on the image—the review progress is recorded via musicStatus/draftMark in the export asset manifest and music.pendingChecks; if a visible hint inside the final video is truly needed, explicitly write project.draftMark:true (with text project.draftMarkText).
hud-density.mjs spreads six types of changes—shot cuts, materials, HUD, events, sound effects, subtitles—onto the timeline, reporting the longest blank window and internal blank windows within a single shot; with --strict, exits non-zero if any blank window exists. sync-audit.py is the master gate before export, one command checks five types of timing issues: whether the storyboard grid and the analysis grid are the same BPM, whether cuts land on beats, whether scene events with sync:"beat" land on beats, whether lyrics land on strong onsets, and whether chapters and sections share boundaries and are both on cuts; when --video is given, it additionally measures the real pixels from the subtitle bottom to the canvas bottom and right in the rendered video. Must pass --analysis: only checking "cut points vs storyboard grid" will yield a perfect 0ms result—because the cut points are snapped to that grid; if the grid itself is wrong, the cuts being aligned means nothing (an old storyboard's evidence says "98.8, 144.1 are another strong candidate not used"—exactly this self-referential loop). Without --analysis, lyrics are only hinted based on beat points, with weaker criteria and also no error judgment. The deviation between lyrics and strong onsets also only yields hints, not errors: onset detection finds energy transients (drum hits, plosives, consonant starts); vocal passages singing smoothly into a long phrase have no transient to find—a "not on strong onset" verdict there is a detector blind spot, not an alignment error—in one test, 9 out of 13 lines differed from the nearest strong onset by 100–800ms, but the phase difference between the final audio track and the LRC start was only 1.2ms; the criterion, not the timeline, was wrong. LRC is manually annotated and more credible than heuristic detection on this matter, so lines falling outside tolerance are only listed as candidates (including opposite-direction candidates), left to human ears to decide. timeline-sync.py extracts the energy envelope and estimated beats from the audio track, draws them together with cuts and subtitles on one chart; BPMs marked speculative can only be used to discover whole-section misstepping, not to claim sync; --align beats prioritizes the storyboard's own music.beats as reference points, falling back to self-estimated beats only if absent. safe-area.py overlays platform safe areas (top, bottom 8.5%, right 6.5%, 4:3, 9:16), both subtitle bottom edge and HUD must stay clear; the bottom edge minimum for 1080p is 92 px. layout-check.py checks for overlaps between HUD elements, between HUD and the subtitle band, and rectangles that go outside the safe area, using pixels from the final video; rectangles are written as name:x,y,x1,y1 (top-left + bottom-right, both absolute pixels, not width/height); --subtitle-band passes the outermost rectangle including the bottom bar and stroke; scenes like --scenes "s09-ledger:full-ui" can exempt that scene's HUD check, though the subtitle band is still constrained by the safe area. minimal-renderer.py is a minimal renderer for placeholder and layout validation (title card + numeric HUD + bilingual subtitles); it does not draw scenes and is not a final video renderer; for visual completeness, use the starter or the user's own project. sfx.mjs synthesizes eight sound effects (tick/type/click/whoosh/confirm/error/riser/chime), the length strictly equal to project.duration, for mixing use; the two-layer alignment relationship is described in SFX Layer and Beat Sync—video hits the beat point, SFX follows the same frame as the video, SFX does not hit the beat point.
The validator retains validateStoryboard(data) → errors[], and provides analyzeStoryboard(data,{beatToleranceMs,delivery}) → {errors,warnings}. Normal hints do not change the exit code; --strict only treats warnings from structural checks as failures; passing does not prove it is publishable or verified. --delivery checks license records, music evidence, and draft pending checklists; field rules are in the Timing and Storyboard reference. Cross-shot lyrics, explicit beat-sync events, HUD support, device/chapter coverage, and missing license items all have locatable hints; the tools do not prove semantic correctness, license validity, or actual audio-visual sync.
Completion Criteria
- Without subtitles, the actions still convey the meaning of the section; HUD changes are traceable to actions; items are acquired before use.
- Main characters are distinguishable across pure black silhouettes, minimum group size, and single close-ups, not relying solely on color, nameplates, or small props; close-ups use independent detail assets, not simply enlarged low-res group sprites.
- Repeated sections show development; waiting, failure, completion, and pauses are proportionate to the song's mood.
- No duplicate frames or out-of-order evaluation; no state drift from start to end, between shots, or at transitions; camera moves don't show black edges; subtitles are legible.
- Subtitle bottom edge and HUD are within platform safe areas: verify with one frame overlaid by
safe-area.py, don't just look at the final video. - The longest blank window reported by
hud-density.mjsdoes not exceed 12 seconds; if it does, first add changes, not shots. - Sound effects land on visual events (not on beat points);
project.sfx[]/scene.sfx[]pass validation; no peak clipping after mixing. - The final video's audio and video streams, duration, aspect ratio, and frame rate are correct; music evidence status and actual verification scope are clearly written.
- Deliver editable source files, storyboard and analysis data, resource sources/licenses/attributions, chosen dependency lock file (if dependencies are used), the final video, and a brief acceptance record. When only a report, storyboard, or draft is requested, deliver the corresponding stage; do not unilaterally extend to the full song.