muyunxue/dance-mmd-subtitle-kit

Character-avoiding subtitle layout for MMD dance videos — measures per-frame bounding boxes, computes per-cue union bounds, places text in safe side/top columns, and verifies zero overlap on every frame. Use when asked for subtitles that never block the character.

What is dance-mmd-subtitle-kit?

This skill provides a verified, zero-overlap subtitle layout algorithm for MMD dance videos. It operates in two modes: "strictly avoid" (zero pixel overlap with the character's mask on every frame) and "mostly avoid" (allows overlap with wing edges/background but keeps face and body clear). The core algorithm works at 1/4 resolution (640×270) for speed, computing a per-frame bounding box via frame differencing against a background plate, thresholding at 38, applying MaxFilter dilation, and extracting the largest connected component. Bounding boxes are unioned across each lyric cue to produce safe side/top column positions. The layout is then verified frame-by-frame: each character glyph box is mapped back to the mask and checked for overlap; if any hit is found, font size is reduced globally and verification repeats until zero overlap. The skill includes a ready-to-use implementation (avoid_side2.py) that handles rendering and keyframe output, and provides detailed validation checklists and known pitfalls such as background light artifacts and projected shadows. It is designed for MMD dance videos with typical resolutions up to 2560px wide.

Works with~Claude Code~Codex CLI~Cursor

npx skills add https://github.com/muyunxue/dance-mmd-subtitle-kit/tree/HEAD/skills/avoid-subtitles

Documentation

Character-Avoiding Subtitles (Completely / Mostly Avoid Blocking)

0. Distinguish Two Goals

"Completely avoid": The intersection of every character box with the character mask on every frame = 0. The cost is that text can only be placed in narrow side/top columns, and font size is constrained.
"Mostly avoid": Allows overlapping wing edges/background, but the face and body must remain clean. Larger font styles (S4/S6, etc.) can be used, and only close-up frames are checked.

By default, this skill does "completely avoid" and provides verifiable data; if the user says "explosive/large font/artistic", it is likely the latter.

1. Core Algorithm (Fast Version: Bounding Box Only)

No longer requires pixel-accurate matting; only the character's bounding box is needed. Computed at 1/4 resolution, very fast:

Background plate bg_half.npy (1280×540, see subject-color-grade 2.1/3.1).
Downscale each frame to 640×270, compute difference: d = |frame-bg|.max(axis=2), threshold 38 (lower thresholds would include LOVE light signs/glare).
MaxFilter(5) dilation → largest connected component (BFS, fast on 640×270) → multiply bbox ×4 to restore full resolution.
Unite per-frame bboxes for each lyric cue window (character moves within a cue, so use the widest range of the entire cue) to get per-cue bounds = (left, right, top, bottom).
Layout: text columns placed to the left of left-50 / right of right+50 / top / bottom; font size shrinks according to available width.
Frame-by-frame verification: Map each character box back to 640×270, compute intersection with per-frame mask, report worst hit; if non-zero, shrink font size globally and retry until 0.

2. Ready-to-Reuse Implementation

字幕\build\avoid_side2.py (recommended, full version: pass1 bbox → layout → verify → render → keyframes):

$py='C:\Users\35530\.dsh\dsh-runtimes\dsh-primary-runtime\dependencies\python\python.exe'
& $py '字幕\build\avoid_side2.py'

It works with a fixed old clip (0001-4956), 15.4s segment; to change clip/segment modify these constants:

SRC / OUT / T0,T1
CUES (per-cue start/end, relative to segment)
Font size/position candidates in EL (columns L/R, y, fs)
LEFT_MARGIN / right available width logic Outputs: NEON_avoid_15s.ass + rendered video + keyframe table. Old versions (synthesis demos): avoid_demo.py, avoid_debug.py, avoid_layout.py (with occupancy heatmap 分析_占位与选位.jpg).

3. Measured Data from This Project (Old Clip 0:51–1:06.4)

Fixed bottom layout: 6/6 cues overlap character, character region occlusion 7%–12%.
Side column avoidance: 0/6 cues overlap character, per-cue overlap ≤0.2%; frame-by-frame verification 0 pixels.
Available side columns per cue (old clip, full width 2560): left column approx 0–500px, right column approx 2100–2560px, varies per cue, must compute on the fly.
Character in center, wings spread occupies ~48%–82% width; close-up/extreme close-up nearly fills frame → close-up frames require text in top/bottom corners.

4. Layout Strategy (Attractive Even When Avoiding Character)

Main lyrics in left column, decorative words in right column to create a "left-right duet" feel;
Long lines at the top (e.g. MAKING EVERYBODY GO), bottom only for small text/monospaced info;
Each cue has only one "visual hero", rest set to small font, avoid filling the entire line with large text that cannot fit in side columns;
Character box estimation should be conservative: ASCII width ≈ fs×0.66, CJK ≈ fs×1.06; reserve 10–14px padding.

5. Verification Checklist

Does the per-cue bbox miss wing tips/top of head? (Check 分析_占用区域.jpg)
Frame-by-frame worst hit = 0 (script will print)
Check 3–4 frames: text touching/cropped by edges, overlapping small fingers/wings
Spot-check close-up/extreme close-up frames
If "mostly avoid" is allowed, record which frames overlap wings (acceptable)

6. Known Pitfalls

Background LOL light signs/glare may be misidentified by frame difference → threshold 38 + largest connected component usually removes them; if not, add spatial prior (character centered).
Character shadows may be mistaken for the character (bbox becomes larger) → this is safe for avoidance (you will avoid the shadow), but narrows side columns.
MaxFilter on 640×270 ≈ expands 8px at full resolution; do not add extra padding during verification.
If switching to a new video or camera angle, all bounds must be recomputed; old numbers cannot be reused.