Japanese Lyric Subtitles: Phrasing, Onset Timing, Orthography,…

Japanese Lyric Subtitles: Phrasing, Onset Timing, Orthography, and Typography

0. What This Skill Solves

The four most common errors in Japanese lyric subtitles — all of which look fine to a non-native but are instantly wrong to a Japanese viewer:

1. Equal time-slicing by character count — A 5.7-second line with 16 syllables gets 0.36 seconds each. Actual sung durations vary by 3×.
2. Breaking in the middle of a phrase — 君 / を見るたび. A particle should never start a new segment.
3. All-hiragana — Lyrics provided for TTS/generation models are often all-kana. Shown on screen, they read like a children's book.
4. Snapping onsets to pitch-change points — The phrase-ending note fall gets mistaken for the next phrase's start, pushing the whole phrase half a second early.

---

1. Eight Hard Rules

#	Rule	Counterexample
R1	Break only at bunsetsu (文節) boundaries. Noun + particle, verb + auxiliary verb must not be split	✗ 君／を見るたび　✗ 見／てるのに
R2	Each segment's onset = the measured sung start of its first syllable. ⛔ No equal time-slicing	✗ (end-start)/n × k
R3	Max 3 segments per line, prefer 2	✗ 君を／見る／たび
R4	No punctuation; use full-width space 　 between segments	✗ 君を、見るたび
R5	Prohibited line-start characters (JIS X 4051): small kana ゃゅょっ, long vowel ー, geminate っ, closing brackets	✗ Line starts with っ
R6	Single line ≤ 16 characters (Netflix Japanese subtitle spec)	—
R7	After appearing, text does not disappear (cumulative display), otherwise the viewer loses context	✗ Erasing the first segment when the second appears
R8	Background large text must contain all characters of the line; character-by-character missing must be empty	✗ Background shows only the second half

Why R1 uses bunsetsu instead of "words": Japanese has no spaces. Machine line breaks happen at the box width. Humans read in bunsetsu breaths — 君を／見るたび is exactly two breaths. Test: if you say the segment aloud and it feels like one breath, it's correct.

---

2. Measured Onset Timing (Core — Don't Estimate)

2.1 Prerequisites

Requires a vocal separation stem (vocals.wav from Demucs / htdemucs). The mixed track has drums and bass, and energy peaks are mostly drum hits.

python scripts/measure_lyric_onsets.py --vocal vocals.wav --lines lines.json --out timing.json

lines.json looks like this (kana is for syllable count to determine expected position; disp is the display orthography):

[{"id": 9, "start": 42.26, "end": 48.00,
  "kana": ["きみを", "みるたび", "しんぞうがとまるの"],
  "disp": ["君を", "見るたび", "心臓が止まるの"]}]

2.2 Algorithm (Three Steps — Missing One Causes Errors)

1. Extract F0: librosa.pyin(fmin=150, fmax=900, hop_length=128), then median-filter to remove vibrato.
2. Get candidate boundaries: Pitch jumps ≥0.8 semitones ∪ librosa.onset.onset_detect(delta=0.30, wait=3).
3. ⛔ Threshold filter (most critical step): Candidate must simultaneously satisfy:
   • Energy in the next 40ms is ≥55% of the previous 40ms (not a sudden drop)
   • Within 180ms after, F0 non-NaN frames ≥60% (someone is still singing)

Then: for each bunsetsu, calculate expected relative position as cumulative_syllables / total_syllables, and snap to the nearest valid boundary.

2.3 Why Step 3 Cannot Be Skipped (Real Counterexample)

For 「君を見るたび 心臓が止まるの」, the び at 44.87s ends with a falling tone (F#2 → end), 0.33s of silence, then しんぞう starts at 45.24s.

• Only step 2 → boundary at 44.81 (the fall of び), pushing 「心臓が止まるの」 0.43s early.
• Step 3, only the first condition (40ms energy ratio) → Still wrong. Energy hasn't dropped at 44.81; the 40ms window is too short.
• Adding "still voicing within 180ms" → 45.17, within 0.07s of manual check. ✅

⇒ "Is someone still singing after this point?" distinguishes character change from sentence end better than "is the energy high right now?"

2.4 When This Rule Also Fails

• English lyrics: No pitch plateau; pyin can't detect syllable boundaries. Only option is proportional time-slicing, with explicit "estimated" annotation.
• Melisma (one syllable, many notes): Number of pitch plateaus > number of syllables; snapping will drift. Requires manual listening.
• Sentences with snap deviation >0.45s: Always label UNSURE for manual check; never treat as accurate.

2.5 ⛔⛔ First Validate the Line Window Itself, Then Intra-Line Phrasing

This is a more upstream pitfall than §2.3, and the most easily overlooked.

The vast majority of lyric timing files (lyrics.ts / SRT / LRC) have start/end values that are linearly divided by line count or hand-estimated, not per-line listened. Using these for subtitles means even if intra-line timing is perfect, the whole line is misaligned.

Symptom: Subtitles are 1–4 seconds early or late relative to the vocal, and drift worsens over time (because of linear stretching). Users don't notice in the chorus (whose window is usually correct), but the first few verses of the main melody will be desynchronized.

Fix: Re-anchor each line using breath intervals

On the vocal stem, there is always silence between lines. This is the hardest boundary, requiring no model:

rms = librosa.feature.rms(y=vocal, frame_length=1024, hop_length=128)[0]
e = rms / rms.max()
quiet = e < 0.055                      # relative threshold, empirically sufficient
# consecutive quiet >= 0.16s counts as a breath interval

Then: The voiced segments between adjacent intervals are the phrases.

Real example (same song):

interval 0.00-12.12 (intro, no singing)
voiced 12.12-13.73  ← ドアのとこで
interval 13.73-14.67
voiced 14.67-16.59  ← 立ち止まってる
interval 16.59-17.07
voiced 17.07-17.98  ← 本当は

But the lyric sheet says "Line 1: 12.00–15.75" "Line 3: starts at 19.50" — Line 3 actually starts at 23.63, 4.1 seconds early.

⭐ Check: Compare the number of detected voiced segments to the number of bunsetsu in this section of lyrics.

• Equal ⇒ The singer breathes once per bunsetsu; direct 1:1 match, millisecond precision, no guessing.
• Fewer voiced segments ⇒ melisma; use §2.2's pitch method for intra-line boundaries.
• More voiced segments ⇒ threshold too loose; increase 0.055 or lengthen 0.16.

⛔ Don't use "energy peak count" instead of breath detection: vibrato in a long note produces multiple peaks. A 16-syllable line was measured to have 33 energy peaks — completely unusable.

⛔ Don't use "rolling median normalization + shallow threshold" for detecting weak breaths: in the main verse, 23 false intervals appeared (true value: 19), noise and true breaths indistinguishable. Better to anchor only those phrases with hard silence, and explicitly mark the rest as "remapped/estimated".

---

3. Orthography: Kanji-Kana-Majiri

Lyric sheets (especially AI-generated song lyrics) often write in full hiragana. Must convert to standard kanji-kana-majiri for display.

Full hiragana (lyric sheet)	Display notation
きみをみるたび	君を見るたび
ほんとは / きみのほう	本当は / 君のほう
なかに	中に
においだけで	匂いだけで
いちばんとおい	一番遠い
こわれちゃいそう / ちかづくと	壊れちゃいそう / 近づくと
たちどまってる	立ち止まってる
わたしのあかし	私の証

⛔ Don't over-kanji-ize: すきだよって should not be written as 好きだよって… actually it should (好き). But for colloquial adverbs/final particles like なっちゃう, ずるいよね, なんてかな, keep hiragana — changing everything to kanji feels unnatural. Test: Would this word normally be written in kanji in Japanese novels/lyric typesetting? If unsure, keep hiragana.

---

4. Typography Parameters (Sourced — Don't Invent Your Own)

The following values come from quantitative reverse-engineering of four commercial anime-style MVs (1280×720 base scaled to 1080p):

Item	1080p Value
Main lyric character height	97~108px
Narration small text	53px
Chorus emphasis line	124px
Giant background text	270~670px
Bottom baseline	y ≈ 950~1004
Single line max width	≤1747px (≤91% screen width)

⭐ Most important finding: Commercial MV main lyrics generally have NO background plate — white text + 2~3px dark outline. Only emphasis lines use a full-width white bar with black text. Adding a rounded rectangle background to every line is the primary sign of an amateur product.

Color palette should not change mid-song: a single project uses one hue; the chorus only increases saturation, not changes hue.

4.1 Vertical Multi-Column = Visualizing Phrasing

In horizontal layout, full-width spaces separate segments, but visually it feels like "no phrasing." In vertical layout, align each bunsetsu to one column, read from right to left, and the phrasing becomes the layout itself:

君を ｜ 見るたび ｜ 息が止まるの
right col  mid col    left col

Column position is fixed by bunsetsu index (k-th segment is always in the k-th column). When a new segment appears, existing columns do not move — otherwise the entire block shifts horizontally with each reveal.

4.2 Physical Limit of Vertical Layout (Calculate Before Choosing Font Size)

For a line of n characters and line-height ratio 1.46: required height = n × font-size × 1.46. 1080p screen height, usable 940px ⇒ max font size = 940 / (n × 1.46).

• 8 characters (心臓が止まるの) → 79px, below the 97px minimum
• 6 characters → 107px ✅

⇒ For lines longer than 7 characters, vertical layout cannot fit the main lyric size range; use horizontal two-line instead. Calculate first, don't discover after completing the layout.

---

5. Self-Check (Run Every Time — If You Skip, Delivery Will Fail)

python scripts/check_lyric_coverage.py --timing timing.json --disp disp.json

Output must simultaneously satisfy:

COVER line='君を見るたび' missing=[] OK=True
ORDER sub_a='君を' sub_b='見るたび' concat_equals_full=True

If missing is not empty ⇒ this version must not be delivered. ORDER checks that concatenating the segments equals the full line (prevents missing/repeated characters).

---

6. Must-Do Before Starting

Grep the project documentation for subtitle specs. Previously determined font sizes, color palettes, and whether to use background plates are typically in 02_ドキュメント/ or docs/ in some analysis/decision document:

grep -rn -E "subtitle|font|typography|geometry|char height|outline|background" docs/ 02_ドキュメント/ | head -40

⛔ Don't decide on your own because "I think it looks better" — that will inevitably conflict with already-determined specs, and rework cost is re-rendering the entire video. Section 4 of this skill was created because of this lesson.

---

7. Real Measurement Log (For Calibrating Expectations)

A Japanese song at 163.994 BPM, first chorus line:

Syllable	き	み	を	み	る	た	び
Onset (s)	42.31	42.76	43.09	43.41	43.61	43.98	44.18
Pitch	B3	E4	F#4	G#4	F#4	F4	F#4

7 syllables ↔ 7 pitch plateaus, one-to-one ⇒ This chorus is syllabic (one note per syllable). For a song with more melisma, the number of plateaus will be fewer than syllables, and snapping must be marked UNSURE.

---

8. ⛔ BPM Is Not a Grid (Real Drift)

A song's bpm field is only a label; never use it to calculate cut points. Real example: a song at 128.28 → 132.39 BPM (beat 468ms → 453ms). Fixed grid at the 30th bar is off by a full beat.

• Use beats[] (already snapped to onset) + bars[].start to get actual times
• beats is for impact points (cuts/flashes/text hits); beats_smooth is for continuous motion (camera speed/loops)
• ⛔ Never write t0 + n * 60 / bpm
• Bar number ≠ array index: When a pickup bar exists, bars[].index differs from position; query by index

9. ⛔ ASR Word Timestamps Are Too Early — Cannot Be Used as Onsets

Measured deviation: 0.04–2.0 seconds, median 0.62 seconds; worst after long notes. ⇒ ASR can only provide approximate line start (±1s). Precise onsets require forced alignment (MMS CTC / torchaudio). Check: line onset error ≤ 0.15 seconds counts as accurate.

10. ⛔ Whisper Hallucination List (Delete If Present)

In silence, breaths, or vocalise, Whisper generates:

• 「ご視聴ありがとうございました」(once covering 0.6–103 seconds)
• 「チャンネル登録…」
• 「作詞・作曲・編曲 初音ミク」
• Repeated short words, ラララ…

⇒ Rule: Use official lyrics as ground truth; only use transcription if no official lyrics exist, and compare line-by-line with the official sheet (simple diff misses "whole line missing"). ⇒ Disable VAD (Silero misclassifies sung vowels as non-voice); separated vocal stem is better than any DSP enhancement.

11. LRC Markers (Mini-Language for Lyric Animation)

Marker	Effect
phraseA/phraseB	Force line break, two halves become different beats
*word*	Emphasis: larger + impact component + emphasis color
Line-end !	Kime line (strongest treatment, can include flash/shake) — 1–2 lines per chorus section
[instrumental]	Instrumental line: prevents previous line from remaining on screen

⛔ Don't double-apply negative offset: Lyric engines usually have a built-in "0.2s before vocal" offset. If your timeline already subtracted 0.2s, applying it again will make everything 0.4s early.

12. Chorus Ending Stop Bars

The few bars at the end of a chorus where the band drops out and it's nearly a cappella are the most important visual position in the entire video — not a time to cut away. Don't treat it as "there are no drums here, so put anything."

---

13. ⭐ Comparison of Five Forced Alignment Tools on Japanese Lyrics

Source: third-party benchmark (20 segments, including Chinese/English/Japanese). The table below shows the Japanese column.

Tool	Onset Error (Median)	Offset Error (Median)	Coverage	Verdict
MMS_FA (torchaudio)	0.08 s	0.11 s	8/8	✅ Only one that meets spec
whisper-timestamped	—	0.16 s	8/8	⚠️ Second choice
WhisperX	—	1.67 s (p90 2.81)	8/8	❌ Full coverage but boundary drift
NeMo NFA	—	—	0/8	❌ No Japanese pretrained CTC
aeneas	2.70 s	3.66 s	8/8	❌ Broken

⛔ aeneas is not suitable for singing: its own documentation states "Audio is assumed to be spoken: not suitable for song captioning." Measured 2.7–3.7s error on Japanese — worse than equal time-slicing.

⇒ To achieve ≤0.15s, only MMS_FA works. All others fail the spec.

⚠️⚠️ But MMS_FA's license is CC-BY-NC (non-commercial). If we go commercial on Steam, we must make separate licensing arrangements or use it only for internal alignment without distributing model weights. This must be confirmed before commercialization.

⚠️ MMS_FA relies on uroman romanization for non-Latin scripts. For mixed kanji-kana Japanese text, p90 can drift to 2.53s for long segments ⇒ must check line-by-line.

14. The Two Offset Values Are Not the Same Thing

Value	Purpose	Source
0.15–0.25 s	Allows the viewer to read the text (reading anticipation)	Lyric engine default: "text 0.2s before audio"
0.03–0.05 s	Allows the viewer to sing along (karaoke alignment feel)	Another source's lyric video practice

⇒ Choose based on whether this video is "text to read" or "text to sing along with." We are making "text to read" ⇒ use 0.15–0.2 s. ⛔ Do not stack both.

---

15. Actual Rules for Vertical Typesetting (Not Just "Rotate the Text")

Rule	Implementation
Punctuation marks	Occupies one cell, written in the upper right of the cell (horizontal: lower left)
Small kana (っ ゃ ゅ ょ)	Occupies one cell, written in the upper right of the cell
You-on (きゃ)	Treated as two kana, each occupying one cell
Long vowel mark 「ー」	Occupies one cell
Paragraph start	One cell indent
⛔ Prohibition	Punctuation marks must not appear at the start of a line
Line-end punctuation	If a sentence-ending mark falls at the end of a line with no cell available, write it outside the cell or share a cell with the last character
Numbers	Use kanji numerals for vertical writing; Arabic numerals (1–2 per cell) are acceptable for horizontal

⇒ Our current vertical layout puts "each character in one line downwards," but does not handle punctuation mark positioning, small kana offset, or paragraph indentation. Adding these three will make it instantly "correct" to Japanese readers.

16. Reading Speed Standard: 4 cps is the Broadcast Subtitle Standard, Not the Lyric Standard (Including Real Project Measurements)

⚠️ Confidence distribution: Numbers: B-grade (AVTpro JA style guide for Japanese subtitling certification, two subtitle companies' public specs, three-way cross-consistent); "How to apply to lyrics" is my own inference, D-grade — explicitly annotated, do not use as evidence.

16.1 Hard Numbers from Three Sources (Mutually Consistent Parts)

Item	AVTpro JA (Certification Standard)	GoLocalise	DMPJ	Adopted
Max reading speed	4.0 cps	12 char/s (≈ upper limit, different metric)	~4 char/s	4.0–5.0 cps
Characters per line (horizontal)	15 char	16 char	13–16 char	≤15
Characters per line (vertical)	11 char	—	—	≤11
Lines on screen	2	2	2	2
Minimum display time	20 frames (0.8 s @25fps)	1 s	1.5 s	≥0.8 s
Maximum display time	6 s (lyrics: 7 s)	7 s	—	6 s (lyrics: 7 s)
Punctuation handling	。→ full-width space; 、→ half-width space	—	—	Same as AVTpro
Font	Gothic	—	Gothic preferred (readability)	Gothic

⭐ Two items confirm our existing decisions:

1. "Don't use punctuation" is correct — AVTpro's official practice is to replace punctuation with spaces (。→ full-width, 、→ half-width). Our approach of no punctuation at all is equivalent and cleaner ⇒ R7 does not need changes.
2. Gothic (sans-serif) font is correct — We use Yu Gothic Bold, which is a Gothic font; the direction is right (though font licensing is a problem; see §17).

16.2 ⛔ But 4 cps Cannot Be Directly Applied to Lyrics (My Inference, D-Grade)

The reason for 4 cps: Dialogue not understood must be re-read; subtitles are the only information channel. But lyrics are sung simultaneously — the viewer sees the word at the same moment they hear it; no need to "read before comprehension."

⇒ Lyric subtitles should not be designed for "reading completion," but for "singing along." (AVTpro itself makes an exception for lyrics: max display time relaxed from 6s to 7s — the industry acknowledges lyrics are a different category.)

⚠️ But this does not mean "lyrics can be arbitrarily fast." Beyond ~6 cps, even "singing along" fails because the eye can't find the current character. ⇒ Practical check: peak cps ≤ 5.0 acceptable, 5.0–6.0 needs improvement, >6.0 requires layout changes (line break / two lines simultaneous / shorten bunsetsu).

16.3 Real Project Measurements (lyric_timing_v2.json, 22 lines / 52 bunsetsu)

Scope	Result
Per whole line (current WHOLE_LINE v7)	22 lines: 5 lines >4.0 cps, worst L5 = 5.30 cps (16 char / 3.02 s)
Per bunsetsu (if reverting to per-bunsetsu reveal)	52 bunsetsu: 11 >4.0 cps (21%), worst L3「ポケットのなかに」6.90 cps
Vertical column character count	Longest 16 char → split into two columns of 8 each, does not exceed 11, OK

⇒ Two layers of conclusion, don't mix:

• By strict 4.0 cps standard: 5 lines fail, but by §16.2's lyric standard (≤5.0), only L5's 5.30 needs attention.
• ⛔ If reverting to per-bunsetsu reveal, 21% of bunsetsu will exceed 5–7 cps ⇒ This is the quantitative basis for the WHOLE_LINE decision (originally chosen intuitively as "per-bunsetsu would trigger viewer to check sync," now we have numbers).

16.4 ⭐ Incidental Discovery of a Real Defect: L3–L7 Timeline Is Equally Divided

Symptom	L3–L7 five consecutive lines, all exactly 3.02 seconds, perfectly contiguous (23.63→38.72 = 15.09 s ÷ 5)
Root cause	Breath interval detection found no gaps in this legato section (notes connected between syllables), REMAP degraded to equal division, and issued no warning
Real counter-evidence	Sweeping 4 thresholds × 2 minimum lengths in the 22–44s range, true silence exists at only 7 positions (22.63 / 23.01 / 24.24 / 25.60 / 28.29 / 31.43 / 41.55), none at 26.65 / 29.67 / 32.69 / 35.70 / 38.72 ⇒ these five boundaries were not measured
Why hard to detect	source field only says REMAP, not distinguishing "anchored remap" from "equal-division fallback"
Fix (not implemented, belongs to the program side)	①Split REMAP into REMAP_ANCHORED / EQUAL_DIVIDE; the latter must print a warning ②When using equal-division fallback, also output cps; if >5.0, mark red directly ③For legato sections, use pyin pitch continuity instead of silence to determine boundaries

⛔ This is another case where "all indicators look green but the data is fake": missing=[] only means "every line has timing data," not "the timing was measured."

17. ⛔ Font Licensing: We Currently Use Windows System Fonts — Commercial Risk (B-Grade)

⚠️ Confidence level: B (official licensing pages / FAQ from each font vendor, read line by line — not secondhand)

17.1 Four License Types, Worlds Apart

Font	Burned-in video subtitles	Commercial use	Notes
SIL OFL 1.1 (Noto/Source Han, M PLUS, Zen Maru, Kosugi, Shippori, Shinshu Mincho…)	✅	✅ No attribution required	Only rule: ⛔ Cannot sell the font file alone; derivative works must use same license
IPA Font License 1.0 (IPAex, Kokoro Mincho…)	✅ Explicitly allows "visual content" use	✅	⛔ Derivative font names must not contain "IPA"; redistribution must include license
Hiragino (macOS built-in)	✅ Output can be sold/distributed	✅	⛔ But explicitly prohibits "creating data in a form that functions as a font in digital content"; also tied to macOS — using it on Windows violates terms
Nihon Shogi Kenkyujo and other commercial foundries	⚠️ Must purchase	Requires commercial license	Video typing: based on capital, ¥22,000–55,000
Kotori Moji Font Pro	✅ (including YouTube monetization)	✅	⛔ But "embedding/webfont" is generally prohibited ⇒ burning into video ≠ embedding a font; these two acts are in different clauses

⭐ Key distinction (most commonly confused): "Burning text into video frames" and "embedding a font file into a document/webpage" are two different actions, governed by different clauses. The former produces pixels, treated as "output material"; the latter transfers the font file itself. ⛔ Don't read Kotori's "embedding prohibited" as "making video prohibited" — its own FAQ states "use in videos" is clearly OK.

17.2 Our Current Status

Item	Value
Current font	C:\Windows\Fonts\YuGothB.ttc (Yu Gothic Bold, Windows built-in)
Glyph style	Gothic ✅ (matches §16.1's readability recommendation)
License status	⚠️ Unchecked. Windows built-in fonts are tied to the OS license; burning into a commercial product (Steam DLC) is a gray area, not explicitly permitted

⇒ Recommend switching to OFL fonts (zero cost, explicitly commercial-use-permitted, no attribution required). Candidates by vibe:

Candidate	License	Vibe	Suitability
Zen Maru Gothic	OFL	Rounded, soft	⭐ Best match for our pink/purple cute aesthetic
Shippori Mincho	OFL	Emotionally rich mincho	Suitable for lyrical lines, but has tension with "cute"
M PLUS 1p	OFL	Geometric, modern	Neutral, safe
Noto Sans JP	OFL	Most neutral	Fallback, can't go wrong

⚠️ Changing font = changing visual style; must be approved by Ronny after visual inspection. ⛔ I am not authorized to change fonts unilaterally (this is a "visual layer" change; per the ironclad rule, must be reviewed by him).