muyunxue/dance-mmd-subtitle-kit

Character-Avoiding Subtitles (Completely / Mostly Avoid Blocking)

0. Distinguish Two Goals

• "Completely avoid": The intersection of every character box with the character mask on every frame = 0. The cost is that text can only be placed in narrow side/top columns, and font size is constrained.
• "Mostly avoid": Allows overlapping wing edges/background, but the face and body must remain clean. Larger font styles (S4/S6, etc.) can be used, and only close-up frames are checked.

By default, this skill does "completely avoid" and provides verifiable data; if the user says "explosive/large font/artistic", it is likely the latter.

1. Core Algorithm (Fast Version: Bounding Box Only)

No longer requires pixel-accurate matting; only the character's bounding box is needed. Computed at 1/4 resolution, very fast:

1. Background plate bg_half.npy (1280×540, see subject-color-grade 2.1/3.1).
2. Downscale each frame to 640×270, compute difference: d = |frame-bg|.max(axis=2), threshold 38 (lower thresholds would include LOVE light signs/glare).
3. MaxFilter(5) dilation → largest connected component (BFS, fast on 640×270) → multiply bbox ×4 to restore full resolution.
4. Unite per-frame bboxes for each lyric cue window (character moves within a cue, so use the widest range of the entire cue) to get per-cue bounds = (left, right, top, bottom).
5. Layout: text columns placed to the left of left-50 / right of right+50 / top / bottom; font size shrinks according to available width.
6. Frame-by-frame verification: Map each character box back to 640×270, compute intersection with per-frame mask, report worst hit; if non-zero, shrink font size globally and retry until 0.

2. Ready-to-Reuse Implementation

字幕\build\avoid_side2.py (recommended, full version: pass1 bbox → layout → verify → render → keyframes):

$py='C:\Users\35530\.dsh\dsh-runtimes\dsh-primary-runtime\dependencies\python\python.exe'
& $py '字幕\build\avoid_side2.py'

It works with a fixed old clip (0001-4956), 15.4s segment; to change clip/segment modify these constants:

• SRC / OUT / T0,T1
• CUES (per-cue start/end, relative to segment)
• Font size/position candidates in EL (columns L/R, y, fs)
• LEFT_MARGIN / right available width logic Outputs: NEON_avoid_15s.ass + rendered video + keyframe table. Old versions (synthesis demos): avoid_demo.py, avoid_debug.py, avoid_layout.py (with occupancy heatmap 分析_占位与选位.jpg).

3. Measured Data from This Project (Old Clip 0:51–1:06.4)

• Fixed bottom layout: 6/6 cues overlap character, character region occlusion 7%–12%.
• Side column avoidance: 0/6 cues overlap character, per-cue overlap ≤0.2%; frame-by-frame verification 0 pixels.
• Available side columns per cue (old clip, full width 2560): left column approx 0–500px, right column approx 2100–2560px, varies per cue, must compute on the fly.
• Character in center, wings spread occupies ~48%–82% width; close-up/extreme close-up nearly fills frame → close-up frames require text in top/bottom corners.

4. Layout Strategy (Attractive Even When Avoiding Character)

• Main lyrics in left column, decorative words in right column to create a "left-right duet" feel;
• Long lines at the top (e.g. MAKING EVERYBODY GO), bottom only for small text/monospaced info;
• Each cue has only one "visual hero", rest set to small font, avoid filling the entire line with large text that cannot fit in side columns;
• Character box estimation should be conservative: ASCII width ≈ fs×0.66, CJK ≈ fs×1.06; reserve 10–14px padding.

5. Verification Checklist

• Does the per-cue bbox miss wing tips/top of head? (Check 分析_占用区域.jpg)
• Frame-by-frame worst hit = 0 (script will print)
• Check 3–4 frames: text touching/cropped by edges, overlapping small fingers/wings
• Spot-check close-up/extreme close-up frames
• If "mostly avoid" is allowed, record which frames overlap wings (acceptable)

6. Known Pitfalls

• Background LOL light signs/glare may be misidentified by frame difference → threshold 38 + largest connected component usually removes them; if not, add spatial prior (character centered).
• Character shadows may be mistaken for the character (bbox becomes larger) → this is safe for avoidance (you will avoid the shadow), but narrows side columns.
• MaxFilter on 640×270 ≈ expands 8px at full resolution; do not add extra padding during verification.
• If switching to a new video or camera angle, all bounds must be recomputed; old numbers cannot be reused.