rocketmandrey/conversational-video-subtitles

Conversational video subtitles

Overview

Captions are their own layer: they carry every spoken word and never depend on what is drawn in the frame. Labels of things in the frame (part names, pros/cons, arrows) are a different layer with a different visual language (callouts next to the object); never cut a word from the captions because "it is already on screen".

Pipeline: words.json + forbid.json (boxes captions must not cover) + config.json + profile → plan → manifest.json (canonical data) → burn (baked preview = visual truth) / export srt|ass|vtt → validate.

Format roles (do not mix them up):

format	carries	does NOT carry
manifest.json	words, timings, layout, placement, accepted overlaps, lifts with reasons, render checks: source of truth for data	pixels
.srt	text + timing	weight, colour, outline, active word, position. Never promise style in SRT
.ass	family + Bold/\b700, size converted to libass metrics, outline, backing box, active word per time slice, position	fades; needs the font: subtitles=x.ass:fontsdir=fonts
.vtt	::cue CSS (family, weight, outline via text-shadow, backing) + classes .active_<spk> / .em_<spk> / .dim / .u per time slice	exact position; the page must load the font with @font-face
baked preview.mp4	everything: the only source of visual truth	—

When to use

• Dialogue skits, interviews, explainers with a narrator: vertical or square, one or two lines max.
• You have word timings (or audio: optional cloud STT via env keys) and know where faces / hands / key props are (boxes; for 3D scenes render a flat-colour matte and box it).
• Not for lyric videos (bouncy, scaling words are deliberately not supported).

Algorithm

Segmentation (chunk in the profile): hard break on pause > pause_s (0.35 s), sentence end, speaker change; soft break on a comma once a chunk has 2+ words; cap by max_words, pixel width (max_width_frac of the safe zone) and max_dur_s (~1.2 s of speech reads well). A chunk never ends on a function word (не, в, а, and, the…): it moves to the next chunk. A lone word joins its neighbour of the same sentence if it fits. Word durations are clamped to 2.5× the median so a word does not hang into a pause. Timing: a unit appears ≤ 50 ms before its first word, never cuts the previous word, stays hold_s after its last word, at least min_show_s.

Placement — placement: {mode: fixed_bottom_center, collision_lift: false} (default for dialogue).

1. One baseline for the whole video: baseline = safe.bottom - baseline_margin_px (or set it explicitly). Every unit is bottom-aligned on it (y + h = baseline) and centred on the frame (x + w/2 = W/2). Nothing moves: no lift, no sideways shift, no per-speaker x. Line width is capped so the centred line stays inside the safe zone.
2. A line that crosses a forbidden box at the baseline stays put: the unit gets overlap_accepted: [classes] and the QA sheet draws a yellow frame "overlap accepted (fixed baseline)". Information, not a failure. Crossing the platform UI is still an error (shrink the font or split the phrase).
3. Option collision_lift: true (mode baseline_lift): x under the speaker (align: speaker) or centred with left / right fallbacks; if every x collides, lift by whole lift_step_px steps until free. The unit gets lift = {px, reason: [classes hit at the baseline], t0, t1}, the manifest lists all lifts in collision_lifts. No reason → no lift. Use it only when the brief asks to keep faces clear at any cost.
4. Never drop a word.

Look. Outline (outline_px) + soft shadow + a light backing whose alpha is raised per unit only as much as needed to keep the darkest lit colour ≥ 4.5:1 (WCAG) against the 90th-percentile pixel behind it. Speakers differ by the active-word colour and, in hybrid/cards, a 4 px bar in the speaker colour; no speech bubbles, no heavy name plates. Calm motion: alpha fades only, no scale pops, no vertical slides.

QA (scripts/validate.py, also run by demo):

• schema: the manifest against templates/manifest.schema.json;
• lines ≤ 2;
• placement: in fixed_bottom_center the spread of y-baseline and of x-centre over all units is 0 (0.5 px tolerance) and there are no lifts; in baseline_lift every lift is on the grid, has a reason, and the unit really collides at the baseline;
• zero overlaps with the platform UI on every frame; forbidden boxes: 0 in baseline_lift, reported as overlap_accepted (info) in fixed_bottom_center;
• every word exactly once, in order, visible while spoken;
• contrast ≥ 4.5 and no bounce, both measured per frame by burn;
• fonts resolve: file present, Pillow family and style, OS/2 weight, fc-scan, and the libass fontselect picks the declared file and weight;
• pixels: on the baked frame AND on the libass render of the ASS, the active colour is present, the glyph bottom is within 3 px of the layout, the stem width is within 1 px of a reference raster of the declared weight, and the outline is within 1 px of outline_px.

Profiles (profiles/*.json)

profile	what	best for
karaoke	2–4 words at once, spoken word in speaker colour, rest dimmed	most stable; fixed lanes, explainers
reveal	words fade in on speech; layout fixed up front; ≤ 2 lines	calm narration
cards	one thought per card (≤ 8 words, 2 lines), dense panel, keyword coloured	slow, quotable lines; reads ahead of speech
kinetic	one token at a time, larger type	punchlines only; little context
hybrid (default)	≤ 3 words, words fade in, underline under the spoken word, emphasis colour, speaker bar	dialogue

Options from the R&D systems as parameters: tempo = chunk.max_dur_s: 1.2 + timing.min_show_s: 0.83 (on in hybrid); stress = emphasis: "energy" (word loudness z-score per speaker from --audio), or "keyword" with a keywords list.

Commands

python -m subkit demo examples/fixture                     # 5 profiles + hybrid_lift -> previews, srt/ass/vtt, validation_report.{md,json}
python -m subkit plan --words W --forbid F --config C [--profile hybrid] [--audio A] -o manifest.json
python -m subkit burn manifest.json --forbid F --video in.mp4 [--audio A] -o preview.mp4 --qa qa.jpg
python -m subkit export manifest.json --format ass -o captions.ass     # srt | ass | vtt
python scripts/check_clean.py                                          # before a commit: no media/keys/abs paths
ELEVENLABS_API_KEY=... python -m subkit stt audio.wav -o words.json     # optional; or --provider groq with GROQ_API_KEY

Pitfalls

• libass ≠ Pillow font size. libass sizes by OS/2 winAscent+winDescent and places the baseline winAscent below the line top; export converts both, or ASS text comes out ~35 % smaller, thinner and lower.
• Variable fonts in ASS fall back to the default instance (usually Regular). Ship a static instance of the weight you use (here Onest-Bold.ttf, 700) and pass fontsdir.
• ffmpeg -ss before -i resets timestamps: the subtitles filter then renders t=0. Seek after -i when burning ASS.
• \k/\kf karaoke is in centiseconds and colours every spoken word permanently; for "only the current word" emit time slices (as export does).
• SRT has no style: if a deliverable needs weight or highlight, ship ASS/VTT or the baked video.
• A floating line (per-speaker x, hysteresis, "looks nicer" lifts) reads as jitter in dialogue. Keep the default fixed mode; in baseline_lift the validator re-checks that each lift really collides.
• Contrast is judged on the lit colour against the brightest 10 % of the pixels behind it, not the average: fire, snow and highlights fail averages.
• Don't drop words already drawn on screen. Move the drawn label to an annotation layer instead.
• Safe zones in templates/safe_zones.json are approximate; platform UIs change, so re-check them before publishing.

Checklist

• words.json covers the whole speech (cloud STT or a forced aligner); speakers labelled
• forbid.json: faces, hands, fire/props per shot (+ speakers boxes for align: speaker)
• config: size, fps, safe_zone, speaker colours (different hues, both ≥ 4.5:1 on the backing)
• plan → manifest; read overlap_accepted (fixed) or collision_lifts (lift mode, each with a real reason)
• burn → look at the QA sheet (cyan = baseline, yellow = overlap accepted, orange = lift with reason) and at real frames
• validate all PASS; ship baked video (+ ASS/VTT if a player renders them; SRT only as plain text)