muyunxue/dance-mmd-subtitle-kit

MMD Dance Video Subtitles: End-to-End Pipeline

0. One-Line Principle

After writing/rendering, you MUST grab a frame to check: ffprobe duration match ≠ subtitle actually appears; incorrect ASS timestamp or Style color format can result in "succeeded render but no text". This project has had multiple lessons of "thought it was fixed, but actually it wasn't burned in" — acceptance is always based on frame-grab screenshots.

1. Assets and Paths (This Workspace)

Item	Path
Subtitle project	D:\yingyong2\论项目\字幕\
LRC lyrics	字幕\[by维尔日记].txt — KISS OF LIFE《Who is she (Japanese Ver.)》, 36 lines
Old MMD export	字幕\0001-4956 - 副本.mkv (2560×1080 / 30fps; file incomplete, nominal 165s but only decodes ~137s)
New MMD export	D:\mmd.mp4\0002-4956.mkv (2560×1080 / 30fps / 165.166s, complete)
Fonts	字幕\build\fonts\ (Anton, ArchivoBlack, BebasNeue, Bungee, Monoton, PermanentMarker, RubikGlitch, Silkscreen, PirataOne, UnifrakturCook) + system Georgia / Microsoft YaHei
Generation scripts	字幕\build\*.py
Output directory	字幕\输出\<任务名>\ (one folder per task, deliverable name includes readable Chinese name)
ffmpeg	C:\Users\35530\AppData\Local\Microsoft\WinGet\Packages\Gyan.FFmpeg_Microsoft.Winget.Source_8wekyb3d8bbwe\ffmpeg-9.0-full_build\bin\ffmpeg.exe
python	C:\Users\35530\.dsh\dsh-runtimes\dsh-primary-runtime\dependencies\python\python.exe (PIL 12.3 / numpy 2.3, no cv2)

2. Seven-Step Main Workflow

① Probe material   ffprobe resolution/frame rate/duration/color; decode test to check for truncation
② Select segment      song section vs dance action; sample 5s / 30s / full film
③ Align lyrics     LRC absolute time → subtract segment start = ASS relative time
④ Set style     Reference S1–S6 (see style-frames-compare); first produce static frames, then dynamic
⑤ Write ASS     Python generation; follow the format hard rules of ass-render-playbook
⑥ Render       ffmpeg burn (or transparent overlay); first output 5–10s verification, then full segment
⑦ Verify       Grab frames (at least 1 per line) + compose key frame table; check composition, no Chinese, no face blocking, fade out at end

① Probe Material

$ff='...\ffmpeg.exe'; $fp='...\ffprobe.exe'
& $fp -v error -select_streams v:0 -show_entries stream=width,height,r_frame_rate,nb_frames,pix_fmt,color_space -show_entries format=duration -of default=nw=1 $v
& $ff -v error -i $v -map 0:v:0 -c copy -f null NUL   # reports "File ended prematurely" = truncated

• All material in this project is 2560×1080 / 30fps / h264 yuv420p / bt709, ultra-wide 64:27.
• 165s video ≈ 4956 frames; 30s export = 900 frames.

② Select Segment

• LRC chorus first section: 36.892 / 40.366 / 44.003 / 47.577 / 51.179 / 54.698 / 58.309 / 61.926, ends around ~66s.
• 5s comparison sample commonly uses 0:58.2–1:03.2; 30s sample commonly uses 0:36.5–1:06.5; full film 0–154.6s (old file only reaches 137s).
• Trim: -ss 36.5 -t 30 -i src -vf "eq=brightness=-0.05:contrast=1.04:saturation=1.02" -c:v libx264 -crf 16 -pix_fmt yuv420p (Slight darkening to give white/cream text contrast; full color grading see subject-color-grade)

③ Align Lyrics

LRC is absolute time; ASS uses time relative to segment start. Example: if segment starts at 36.5s, then 36.892 → 0.392s. One cue per line; further split into word-level events within a line (see below). End events should be terminated 0.14–0.4s early to avoid cross-line adhesion.

④ Set Style

• First read the style-frames-compare skill, select S1–S6 based on target mood, first output static frames for user review.
• Important constraint: when user explicitly says "no Chinese", keep only original lyric English/Japanese; character in the center, large text placed in left/right safe zones and upper/lower corners.
• Existing reusable generators: styles_anim.py (S2/S4/S6 5s animation), flare_wide.py (explosion), neon_wide.py (neon), avoid_side2.py (full avoidance), kpop_styles.py (static draft), art_frames.py (artistic text static draft).

⑤ Write ASS

• Use Python generation (more reliable than manual): define Ass class + word()/rect() helpers, then write().
• Hard rules, templates, and pitfalls in ass-render-playbook (time format H:MM:SS.cc, Style color values without trailing &, relative fontsdir paths, etc.).

⑥ Render

• Working directory = output directory; ASS uses ASCII filenames; font directory relative:

cd "字幕\输出\S4S6_30s"
& $ff -v error -y -i _base_30s.mp4 -vf "ass=_s4.ass:fontsdir=fonts" `
  -c:v libx264 -preset medium -crf 16 -pix_fmt yuv420p -movflags +faststart S4_30s.mp4

• Transparent overlay (for post-production): -f lavfi -i "[email protected]:s=2560x1080:r=30,format=rgba" -vf "ass=x.ass:alpha=1" + VP9 yuva420p (details in ass-render-playbook).

⑦ Verification Checklist

• ffprobe duration matches design (5.000 / 30.000 / 154.6)
• Grab frames: at least 1 per line, confirm text actually appears, position correct, no garbled characters
• User requested no Chinese → grab 2–3 frames to confirm only English/Japanese
• Person's face/body not obscured (especially check close-up/medium shots)
• Last line fades out before end (e.g., before 29.86s), no lingering text
• Compose 关键帧.jpg (keyframe.jpg) (2×2 or 2×3 grid, each cell 1280×540, with labels) for user convenience
• Readable filename + explanatory md delivered together

3. Task Types → Deliverables

Task	Deliverable
Static frame proposal	N 2560×1080 jpg + 总览.jpg (overview.jpg) + explanatory md
5s sample	1–3 5s mp4 + comparison video + key frame table
30s sample	1–2 30s mp4 (900 frames) + comparison version + key frame table + explanatory md
Paired delivery (common)	2 videos per style: X_有背景_30s.mp4 (Background, H.264 burned) + X_纯字幕_透明_30s.webm (Subtitles Only, Transparent, VP9 alpha); N styles × 2 videos, plus 纯字幕_透明预览.jpg (Transparent Subtitle Preview, checkerboard composite preview)
Full film	Subtitle ASS + burned mp4 / transparent webm + explanation
Color grading	Color graded mp4 + comparison frame + parameter explanation / LUT

Transparent version rendering (same ASS as background version, frame-to-frame identical):

& $ff -v error -y -f lavfi -i "[email protected]:s=2560x1080:r=30,format=rgba" `
  -vf "ass=_s4.ass:alpha=1:fontsdir=fonts" -t 30 -c:v libvpx-vp9 -pix_fmt yuva420p `
  -b:v 0 -crf 30 -row-mt 1 -auto-alt-ref 0 -metadata:s:v:0 alpha_mode=1 OUT.webm

Verify alpha: ffmpeg -c:v libvpx-vp9 -ss 12.4 -i OUT.webm -frames:v 1 -pix_fmt rgba f.png, PIL check alpha min=0 / max=255, coverage > 0; frame grab must explicitly specify -c:v libvpx-vp9 otherwise alpha is lost and everything becomes 255.

4. Parallel (Sub-Agent) Convention

Standard practice when the user says "do separately with sub-agents":

1. The main agent first prepares the shared base footage (trimmed, uniformly darkened) and a copy of fonts/, cue sheet (relative time table);
2. Each style gets a sub-agent, with hard-coded prompt: input path, output filename (S4_30s.mp4 etc.), ASS hard rules, timeline table, verification requirements;
3. Each sub-agent only writes its own prefix files, no overwriting conflicts;
4. The main agent collects all results, creates comparison video/key frame table/explanation, and performs unified verification frame-grab. Measured: a single 30s style (2560×1080, 8 lines, ~40–60 Dialogues) takes about 10–20 minutes from design to export; two sub-agents in parallel ≈ time of the slowest one.