muyunxue/dance-mmd-subtitle-kit

MMD Subject-Aware Color Grading (Emphasize the Dancer)

0. Conclusion First

• For MMD stage videos wanting to "make the dancer stand out," there are two paths:
  1. Light Zone Shaping (Recommended, Clean): A soft elliptical/radial gradient centered on the dancer—inside the ellipse, lift/brighten/warm/sharpen/bloom the dancer; outside, dim/cool/desaturate/slightly blur the background; the transition is wide, almost invisible.
  2. Hard Matte Extraction (Higher Ceiling but Messy): Obtain a character mask from frame-differencing a background plate, then grade the character and background separately and composite. The core difficulty is that the character's cast shadow moves with them and is connected to them, so the mask often includes large areas of shadow → resulting in bright blobs; aggressive erosion eats away wings and feathers.
• Only use method 1 for final delivery; use method 2 when you can re-render a character layer with alpha from Blender/MMD (highest quality).

1. Assets & Tools

Item	Path
Source video	D:\mmd.mp4\0002-4956.mkv (full 165.166s) / 字幕\0001-4956 - 副本.mkv (truncated to ~137s)
Grading scripts	字幕\build\grade_ffmpeg.py (light zone / soft light), grade_final5s.py (5s final parameters), color_grade.py (legacy frame-diff matte), gothic_art.py (static-frame gothic grading)
Validation scripts	avoid_layout.py (occupancy analysis), avoid_side2.py (occupancy + bbox)
ffmpeg	C:\Users\35530\AppData\Local\Microsoft\WinGet\Packages\Gyan.FFmpeg_Microsoft.Winget.Source_8wekyb3d8bbwe\ffmpeg-9.0-full_build\bin\ffmpeg.exe
Python	C:\Users\35530\.dsh\dsh-runtimes\dsh-primary-runtime\dependencies\python\python.exe (PIL/numpy, no cv2/deep learning)

2. Route A: Light Zone Shaping (Default Approach for This Project)

2.1 Generate a Soft-Light Mask (Single PNG)

import numpy as np
from PIL import Image
W,H = 2560,1080
yy,xx = np.mgrid[0:H,0:W].astype(np.float32)
cx,cy = 0.515,0.565          # Dancer center (approximately for 2560x1080 vertical composition; sample a frame to confirm if different)
rx,ry = 0.245,0.47           # Half-width / half-height (as ratios); too small reveals the circle, too large loses separation
q = ((xx-W*cx)/(W*rx))**2 + ((yy-H*cy)/(H*ry))**2
mask = np.clip(1.0-q,0,1)**0.55          # 0.5–0.7 gives a softer transition
Image.fromarray((mask*255).astype(np.uint8)).save("_mask_final.png")

Reference outputs: 字幕\输出\调色测试\_mask_final.png, 字幕\输出\字幕小样\_mask_final.png.

2.2 ffmpeg Composite (Lifted Subject + Dimmed Background)

Idea: Create a graded subject version + a graded background version, then merge them with the mask as an alpha:

[0:v]split=2[sv][bv];
[sv]eq=brightness=0.050:contrast=1.06:saturation=1.15,
    colorbalance=rs=0.02:bs=-0.02:rm=0.05:bm=-0.06:rh=0.02:bh=-0.02,
    split=2[sm][sh];[sh]gblur=sigma=14[blur];
    [sm][blur]blend=all_mode=screen:all_opacity=0.34[sub];
    [sub]format=yuva420p[suba];
[1:v]format=gray[gm];[suba][gm]alphamerge[subj];
[bv]eq=brightness=-0.045:contrast=1.07:saturation=0.87,
    colorbalance=rm=-0.03:bm=0.05,gblur=sigma=0.6[bg];
[bg][subj]overlay=0:0,format=yuv420p[v]

Command:

& $ff -v error -y -ss 36.5 -t 30 -i src.mkv -loop 1 -i _mask_final.png `
  -filter_complex "<graph above>" -map "[v]" -t 30 `
  -c:v libx264 -preset medium -crf 16 -pix_fmt yuv420p OUT.mp4

Note: When using -loop 1 -i mask.png, the mask is a static stream; remember to limit the duration with -t after overlay. Parameter meanings: bg brightness negative = dim, saturation<1 = desaturate, bm>0 = bluish background (cool), gblur = slight blur; in subject, brightness/saturation/warm tones/blend screen achieve the bloom effect.

2.3 Measured Parameter Sets (2560×1080, Purple Stage MMD)

Set	bg brightness	bg sat	subj gain	subj sat	bloom	vignette	Effect
A Natural	−0.035	0.82	+0.030	1.08	0.20	0.30	Lightly lifts the dancer, almost no change to atmosphere
B Enhanced	−0.075	0.70	+0.055	1.16	0.34	0.42	Clear separation, light zone edge slightly visible
Final Direction	−0.085	0.68	+0.050	1.15	0.30	0.38	Very soft transition, recommended as the base grade for the full video

Test parameters on a 5s clip, then apply directly to a 30s/full-length video (ffmpeg runs in one pass, much faster than per-frame Python).

3. Route B: Hard Matte Extraction (When a Cleaner Mask is Needed)

3.1 Background Plate

• Sample every 1/2 second from the full video, scale to 1280×540, compute the per-pixel median = background plate bg_half.npy.
• Command / review: 字幕\build\color_grade.py's build_background().
• Known issue: When the dancer stands still in the center for a long time, the median becomes "contaminated" (certain pixels on the dancer's body are often the dancer) → the frame difference is too small at those body parts → holes in the mask. Mitigate with fill_holes or a bounding box fallback.

3.2 Frame-Difference Mask (Fastest at 1/4 Resolution)

m = |frame_q - bg_q|.max(axis=2) > 38
m = MaxFilter(5) (dilate) → largest connected component → optional hole filling / feathering

• Used for occupancy analysis / avoidance (see avoid-subtitles); when used for grading, you need to add a "character similarity" constraint, otherwise shadows and light spots bleed in.

3.3 Known Failure Modes (Measured in This Project)

• Cast Shadow Contamination: The character's shadow moves with them, forming a single connected region with the character → after grading, bright blobs/color patches appear around the dancer. Solution: do not use frame-difference as a precise matte; either accept "lifting the shadow along with the dancer" (looks weird) or revert to light zone shaping.
• Median Background Pollution: Mask has holes where the dancer stands still → parts of the body are not graded, causing mottling. Solution: fill holes / use bbox.
• 8-bit Banding: Aggressive color pushes create banding in skies/gradients → add 1–3 levels of grain (ffmpeg noise=alls=6:allf=t+u) or output 10bit/HEVC.
• Subtitle Contrast: Dimmed background makes white text more readable; if dimmed too aggressively, the dancer also becomes dark, requiring further lift or added bloom.

4. Gothic / Neon Stylized Grading (For Static Frames)

• Gothic Red-Black (rejected by the user, kept for reference): 字幕\build\gothic_art.py's gothic_grade(): dim by 0.42–0.52, desaturate by 0.30–0.42, midtones purple/red, strong vignette, haze.
• Theme Color Version (what the user ultimately wanted): White/off-white base + lavender purple/magenta pink bloom, background dimmed to deep purple-black; see grading constants in gothic_art.py (BLOOD/LAV/PINK/VIOLET).
• Neon Version: neon_wide.py (draws neon directly with ASS, no grading needed).
• Quick Color Toning: The trio of eq + colorbalance + vignette can push a purple stage to any hue:
  • Cool Cyan: colorbalance=bs=0.10:bm=0.12:bm... (blue shift), desaturate;
  • Warm Ember Fire: colorbalance=rs=0.10:rm=0.08, eq=saturation=1.1.

5. Validation Checklist

• Sample 3–5 frames: is the separation between dancer brightness and background natural? Are there obvious circles at the light zone edge?
• Close-ups / Face shots: is the face overexposed? Is skin tone off?
• Are shadows crushed black / showing banding (8-bit)?
• When viewed with subtitles: is the white text clear enough?
• Does ffprobe output the correct duration / framerate? For 10bit output use -pix_fmt yuv420p10le -c:v libx265

6. Resources

• 5s final parameters & comparisons: 字幕\输出\调色测试\ (A Natural / B Enhanced / Final Direction + comparison videos)
• 30s base grade: 字幕\输出\S4S6_30s\_base_30s.mp4 (only slightly dimmed, not fully graded)
• Previously attempted but rejected directions: Gothic (字幕\输出\字幕设计\G1…G3), FLARE burst, Neon (字幕\输出\字幕小样)——kept for parameter reference.