hongliangzheng11/lyric-video-skill

Lyric Video Standard Pipeline

A fully validated, fixed-style pipeline. The user only needs to provide the audio file and artist name; everything else follows this standard without redesign. Official lyrics are optional: if provided, take the 3A fast path (zero text guessing); if not, take the 3B reconstruction path. Both paths require actual measured timestamps.

Input / Output

Input	Description
Audio	Required. Any MP3/WAV/FLAC, any duration
Artist name	Required. Used for intro, persistent corner logo, and outro credit
Official lyrics (optional)	If the user has official lyrics, they take priority: zero text guessing, but all timestamps come from measured re-interpretation (see 3A). If not provided, use Whisper reconstruction (3B)
Song title (optional)	Defaults to the filename; Artist - Title.mp3 automatically strips the artist prefix
Tagline (optional)	A one-line Chinese subtitle under the intro title; not displayed by default

Output: <project>/renders/video.mp4 + renders/contact-sheet.jpg + transcript.json + NOTES.md.

Fixed Specifications (Do Not Change)

• Vertical 1080 × 1920, 30 fps, total duration equal to the actual audio duration (no cropping, no stretching, no tempo change).
• Audio uses the original file; the framework applies a unified gain during rendering (data-volume="0.9").
• Background: pure black to deep blue-purple, two groups of misaligned light spots slowly cross-fading; no full-screen CSS filters (they leave patch seams).
• Main element: only one transparent glass object, with 8 rounded geometric shapes (sphere/horizontal ellipse/vertical ellipse/rounded rectangle/diamond/triangle/pentagon/hexagon), compressing 11% along the normal when hitting the screen edge → deforming into another shape within 0.94 s → bouncing back and drifting. A single timeline for the entire song, does not reset on lyric changes.
• Bottom: small animal band (cat/bear/penguin/fox) with a seamless 4.5417 s loop, anchored to the bottom 1080×526, layer above background spots and below the glass object, mix-blend-mode: screen.
• Subtitles: the entire block is optically centered at the vertical center y≈960; English/original text on top (Inter, font size auto-scaled by line width), Chinese below (SimSun, warm brown #DDA97D), same group fade in/out.
• Credits use the artist name only; no VERSE / CHORUS / BRIDGE section labels appear on screen.

Detailed parameters, colors, font sizes, and pitfalls are in references/style-spec.md.

Workflow

Let HF = HyperFrames installation root (default G:\hyperframes), PY = Python with faster-whisper (default C:\Users\Admin\.workbuddy-ai\binaries\python\envs\hf\Scripts\python.exe), SK = this skill directory.

1. Create Project

"$PY" "$SK/scripts/new_project.py" "<audio_path>" "<artist_name>" [--title "song title"] [--tagline "one line Chinese"] [--root G:/hyperframes/videos]

Outputs the absolute project path; be sure to note it. It will: create a unique directory → copy the audio → convert to 16 kHz mono WAV → run the official analyze-beatgrid.py to generate audiomap.json → copy band sprites and GSAP → write project.json.

Never overwrite an existing project directory; for modifications, create a new project.

2. Transcribe

"$PY" "$SK/scripts/transcribe.py" "<project>" --model medium.en --lang en

For a 3–4 minute song, CPU int8 takes about 3–6 minutes. For non-English songs, replace --lang with the corresponding language and --model with a non-.en model. Outputs whisper_raw.json (segment + word-level timestamps).

Do not use transcribe.py for mixed-language songs (English+Japanese, English+Korean, etc.): --lang locks the language, and the other language will be forcibly transcribed as the first — the result reads fluently but is entirely wrong, which is more dangerous than failing to decode. Instead, use per-window automatic language detection:

"$PY" "$SK/scripts/transcribe_mixed.py" "<project>" --model "C:/Users/Admin/.cache/whisper-manual/large-v3-turbo"

It decodes in 14 s windows with 10 s steps, each window detecting its own language, keeping only the core words of each window (overlapping areas are discarded, no duplication), prints a language map, and writes whisper_mixed.json. Check the language map first: language flip points are usually segment boundaries. Note that .en models (medium.en / small.en) are pure English models and physically cannot output Japanese; mixed-language songs must use a multilingual model.

For the 3A path (official lyrics), this step is optional: timestamps come from prompt-guided re-interpretation, but for mixed-language songs, it's still recommended to run transcribe_mixed.py first to see the language map, to determine which windows should use --lang en vs --lang ja.

3. Write Lyric Source File (This Step Must Be Done Manually)

First decide which path to take, then write <project>/lyrics_source.json.

3A User provided official lyrics (preferred path)

• Use the official lyrics verbatim: the en field contains the original official text (English sentences in English, Japanese sentences in Japanese, kanji/kana as is), zh is a sentence-by-sentence translation.
• Timestamps do not come from full-track transcription: use the official lyrics as initial_prompt for partitioned re-interpretation to get word-level timestamps (see 3.5), write explicit start/end for each line, and align_lyrics.py will use the manual channel to anchor each line.
• If the official lyrics don't match what the re-interpretation hears → always adopt the official lyrics, list the differences in NOTES.md for user confirmation. Common three types: standard vs typo (不甲斐ない / 不甲埋ない), similar pronunciation (束の間の / 明日の間の), unvoiced consonant mishearing (smiling / falling).
• If two lines in the official lyrics that are sung in one breath are split into lines under 1 second → merge the subtitles into one line (text unchanged, just displayed together), write the merge log in NOTES.md.
• Onomatopoeic or improvisational phrases not in the official lyrics ("Yeah," humming, whistling) → do not add subtitles; do not invent them.
• Structure as follows:

{
  "audio": "Artist - Title.mp3",
  "models": ["large-v3-turbo — prompted window re-scans, official lyrics as initial_prompt"],
  "lines": [
    {"en": "Oh my god, they're coming again", "zh": "Oh tian na, ta men you lai le", "start": 6.36, "end": 9.10, "lang": "en", "conf": "high"}
  ]
}

3B No official lyrics (reconstruction path)

Read whisper_raw.json (or whisper_mixed.json for mixed-language songs) and write the lyric source:

{
  "audio": "Artist - Title.mp3",
  "models": ["faster-whisper medium.en — full track, word timestamps"],
  "lines": [
    {"en": "Sometimes I lay under the moon", "zh": "You shi wo tang wo zai yue guang zhi xia"},
    {"en": "And thank God I'm breathing", "zh": "Gan xie shang tian rang wo reng zai hu xi"}
  ]
}

Common requirements for both paths:

• Lines = actual sung phrases, broken according to official phrasing; Whisper's overly long sentences should be split.
• English spelling should follow official lyrics (preserve apostrophes in 'Cause / I've), do not copy Whisper's typos.
• Chinese is a sentence-by-sentence translation, matching the song's tone, no missing sentences, no merging, no placeholder text.
• Repeated choruses should be written each time (do not use "same as above").
• If a vocal phrase is masked by accompaniment and auto-alignment clearly drifts, add explicit timestamps (listen to the audio first): {"en": "...", "zh": "...", "start": 13.22, "end": 16.80}.
• Chinese songs / instrumental music: for Chinese songs, fill en with the original text and zh with an empty string; the display shows only one line. This skill is not applicable for instrumental music.
• Mixed-language songs: the en field holds the original text of that line's actual language (English sentences in English, Japanese sentences in Japanese, kanji/kana as is), zh holds the Chinese translation. The display is inherently "original + Chinese" two lines; mixed layout does not break the format; the template's English line already includes Yu Gothic UI / Yu Gothic / Meiryo fallback, so kana and kanji display correctly.

3.5 Prompt-Guided Re-Interpretation: The Core Means of Temporal Precision

Full-track transcription is often inaccurate for fast songs, mixed vocals, or rapid-fire lyrics. Using known lyrics as initial_prompt for partitioned re-interpretation is the highest-precision method — prompt quality directly determines re-interpretation accuracy: official lyrics > guessed lyrics > no prompt. You can write a rescan_all.sh to batch process the entire song:

"$PY" "$SK/scripts/rescan.py" "<project>" --from 6.0 --to 20.0 \
    --model "C:/Users/Admin/.cache/whisper-manual/large-v3-turbo" \
    --lang en --prompt "<official lyrics for this window>"

Key points:

• Divide the entire song into 20–26 s windows with ~0.5 s overlap, re-interpreting each window; large-v3-turbo CPU int8 takes about 5–10 s per window.
• For mixed-language songs, run both en/ja paths for each window: only use word-level timestamps that agree across both paths; for words appearing in only one path, cross-check with the other path.
• For rushed lines or lines with unclear starting points, add a narrow 2 s window to lock in (e.g., --from 37.2 --to 39.2 to lock a single line's start); window start points are easily biased by the tail of the previous phrase.
• Hallucination handling: re-interpretation occasionally outputs the same entire line 15+ times (e.g., "永遠の行方はどこにある?" ×15) → discard that window, switch to the other language channel or narrow the window and re-interpret; only use the clean single timestamp; do not write hallucinated timestamps into the lyric source.
• Each re-interpretation saves rescan_<from>-<to>.json and prints each word's start/end within the window. This is a second decode, not an estimate; the numbers go directly into lyrics_source.json's start/end.
• For phrases masked by accompaniment, processed as ooh, merged into one large block, or swallowed repeated phrases, use this technique to write override values; changing the model (small.en / medium.en) can also help.

4. Align + Generate

"$PY" "$SK/scripts/align_lyrics.py" "<project>"
"$PY" "$SK/scripts/motion.py" "<project>"
"$PY" "$SK/scripts/gen_storyboard.py" "<project>"
"$PY" "$SK/scripts/build_frame.py" "<project>"
"$PY" "$SK/scripts/make_index.py" "<project>"

align_lyrics.py prints a line-by-line timeline and performs two hard checks:

• If any sentences at the beginning or end completely fail to align → immediately exit with an error, listing which sentences (do not cover up with guessed timestamps).
• If any sentence in the middle fails to align, or two sentences have the same start time, or a sentence is compressed below 0.5 s → print REVIEW and exit with code 2.

If either occurs, return to step 3.5 to add override values until the output reads OK: every line anchored to a real sung timestamp. Do not skip this step and render. Lines in the 3A path all have explicit start/end (manual channel), so they should all be anchored with 0 gap-fills; the 3B path relies on word matching; for unaligned lines, add timestamps per step 3.5.

motion.py‘s random seed is determined by "artist name + song title" — for the same song, every rerun produces exactly the same path; different songs produce different paths.

5. Verification

cd "<project>"
node "$HF/packages/cli/bin/hyperframes.mjs" check .
node "$HF/packages/cli/bin/hyperframes.mjs" snapshot . --at "3,20,60,120" --no-end -o build/shots

Must achieve: lint / runtime / layout / motion 0 errors, contrast all pass WCAG AA. Then actually view the screenshots: Check the glass object is intact, lyrics are centered, the band is at the bottom, and there are no extra bubbles or section labels. composition_file_too_large is a style hint for single-frame files exceeding 300 lines; it is acceptable (inevitable for single-timeline designs).

6. Render + Acceptance

node "$HF/packages/cli/bin/hyperframes.mjs" render . --skill=music-to-video -q delivery -o renders/video.mp4 --fps 30
"$PY" "$SK/scripts/verify.py" "<project>"

verify.py checks resolution/frame rate/duration deviation (reports error if off by more than one frame) and generates a contact sheet. You must view the contact sheet, confirming that the intro, verses, chorus, repeated sections, and outro all have frames, and subtitles have no gaps or misalignments.

Finally, write NOTES.md (specifications, timestamp sources, reproduction commands, verification log), and deliver the final video and contact sheet to the user.

Hard Rules

• Do not use BPM/beat grids to calculate lyric timings; timestamps can only come from Whisper measurements (including prompt-guided re-interpretation) or manual listening overrides.
• Official lyrics provided by the user take priority over any model reconstruction results; if they conflict, adopt the official lyrics and list the differences in NOTES.md for user confirmation.
• Do not change resolution, frame rate, duration, credits, or subtitle position just because "it looks better."
• Do not claim audio-video synchronization without verification; each song must be spot-checked with frames.
• Missing assets, unavailable tools, or specification conflicts → report clearly; do not use estimated data to fake results.
• Delivered projects and their source files are considered read-only; modifications must create a new directory.

Environment Dependencies

• HyperFrames: node <HF>/packages/cli/bin/hyperframes.mjs (includes skills/music-to-video/scripts/analyze-beatgrid.py).
• Python: faster-whisper, librosa, soundfile, numpy, scipy, pillow. If missing: python -m venv <env> && <env>/Scripts/pip install faster-whisper librosa soundfile numpy scipy pillow.
• Whisper models: Passing the model name directly will auto-download; can also point to a local directory (commonly used locally: C:\Users\Admin\.cache\whisper-manual\medium.en; for prompt-guided re-interpretation, preferred: C:\Users\Admin\.cache\whisper-manual\large-v3-turbo, CPU int8 full song about 48 s, single window 5–10 s).
• ffmpeg / ffprobe must be in PATH.

Known Pitfalls

• Do not use <video> for the band layer: On Windows, hyperframes render fails to read extracted frame directories after linking them into the build directory (ENOENT ... __hyperframes_video_frames/*/frame_00001.jpg). Use a sprite sheet.
• Full-screen CSS filter: hue-rotate() leaves vertical seams in the render; use two layers of light spots cross-fading instead.
• HTML templates with data-composition-id cannot be placed in the project root (triggers multiple_root_compositions); place them in build/.
• Chinese fonts must explicitly declare @font-face { src: local("SimSun"); }, otherwise lint reports font_family_without_font_face.
• Re-interpretation hallucinations: Some windows output the same entire line 15+ times (e.g., "永遠の行方はどこにある?" ×15) → discard that window, switch to the cross-checking en channel or narrow the window and re-interpret; do not write hallucinated timestamps into the lyric source.
• Official lyrics vs. re-interpretation mismatch: Adopt the official lyrics and list the comparison table in NOTES.md for user confirmation; do not arbitrarily change the official text.
• Rapid-fire merged lines: In the official lyrics, two lines sung in one breath that are split into lines under 1 second should have the subtitles merged into one line (text unchanged, just displayed together); write the merge log in NOTES.md.
• Headless font loading failure (Yu Gothic UI FAILED during frame extraction): No need to panic: src: local() fonts that are installed on the system can usually be found by the browser; the next fallback font will also cover it. Check the screenshots first before deciding whether to address it.