TimeCraker/auto-subtitle

auto-subtitle — Automatically Transcribe Audio/Video to SubtitleCue

One command for subtitles: Feed in an audio/video file, get <same-name>.cues.json + <same-name>.srt. The data contract is byte-aligned with video-motion's SubtitleCue: {start, end, text}, seconds with 3 decimal places, single-line text (long sentences already split). The subtitles array can be directly plugged into the video-motion timeline.

Step 1 · Confirm Source and Language

• Supports mp4 / wav / flac (decoded directly by PyAV). If decoding errors occur → transcode first: ffmpeg -i in.mp4 -vn -ac 1 -ar 16000 out.wav
• Confirm the language and pass --lang: zh (default) / en and other whisper language codes
• Don't run on pure music/sound effect sources: VAD will filter out empty or very short output (this is a feature, not a bug)
• If the source has known text (manuscript / script / original text), archive it as UTF-8 first — this serves as the baseline for Step 3 L2 comparison, preventing baseline drift

Step 2 · Run Transcription

The venv is not committed (284 MB build artifact, already covered by .gitignore for .venv/). This repository follows the convention that the skill template does not carry build artifacts, similar to templates without node_modules. Therefore, the venv is created outside the repository and built once for reuse across all projects:

python -m venv "$env:USERPROFILE\.cache\vibe-studio\asr-venv"
& "$env:USERPROFILE\.cache\vibe-studio\asr-venv\Scripts\python.exe" -m pip install -r skills/auto-subtitle/templates/asr/requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

cd skills/auto-subtitle/templates/asr
& "$env:USERPROFILE\.cache\vibe-studio\asr-venv\Scripts\python.exe" transcribe.py <input.mp4|wav> --outdir <output directory>

• Default model is large-v3-turbo (≈1.6GB first auto-download, good Chinese quality baseline); for quick verification add --model small (≈484MB). Model cache is in ~/.cache/huggingface, downloaded once for global reuse
• If model download fails, set HF_ENDPOINT to change the source: default direct connection (tested working as of 2026-08); fallback e.g. $env:HF_ENDPOINT="https://hf-mirror.com" (note: under hub 1.x, this mirror may have 308 redirect compatibility issues; prefer official first)
• stdout prints a one-line English summary: segments=N cues=M total_audio=... model=... lang=... out=...; exit code 0 for success, 1 for failure

Step 3 · Three-Level Verification (Mandatory, Cannot Skip)

L1 Structural: JSON must be loadable by json.load; subtitles is an array and each item has exactly start/end/text fields; start < end and monotonic non-decreasing across items; values to 3 decimal places; last item's end ≤ ffprobe duration + 1s. SRT timestamp lines must match HH:MM:SS,mmm --> HH:MM:SS,mmm, sequence numbers starting from 1 consecutively.

L2 Accuracy: If known text exists → use difflib.SequenceMatcher for comparison (strip punctuation and whitespace on both sides), overall similarity ≥0.85, per-sentence ≥0.70 (use max over concatenated windows of consecutive cues to cover splits/merges); if no known text → spot-check the beginning / middle / end three sections.

L2.5 Timeline (routine check): Subtitle windows must align with word-level speech windows; no subtitles during inter-sentence silence — audit the gap between adjacent cues: inter-sentence gaps should show >1s of real silence (intra-sentence split gap=0 is normal); no flash fragments <0.3s; first cue start aligns with speech onset, last cue end does not swallow final silence. The script internally uses word-level timestamps (first word start / last word end) to set windows; this step is a product-level verification.

L3 Conclusion line: All passed = ALL GREEN; if there are red items → fix the split / VAD parameters, rerun and recheck; if 2 consecutive rounds of red → stop and report, do not blindly tune.

Step 4 · Delivery

• Report: absolute path of cues.json + number of cues + audio duration + model/language + three-level verification conclusion
• video-motion integration: The subtitles array can be directly pasted into the video-motion project's src/cues.ts as cues.subtitles, with zero field conversion
• Explicitly not done (future consideration): speaker diarization, word-level timestamp export, translation, TTS dubbing

Step 5 · Document Re-run Command

Append the full re-run command at the end of the delivery report (input path / --lang / --outdir / --model), ensuring the next session or another person can exactly reproduce the run; the output (cues.json / srt) goes into the output area (in this repo, products/; cross-repo, use project root output/), not committed to git — the command is the asset.

Boundaries and Pitfalls

• GBK console: script prints only in English; file writes always use explicit encoding="utf-8", JSON ensure_ascii=False
• Split rules: split single lines >20 characters at punctuation marks (。！？；，、); if no punctuation in window, hard split with a minimum of 3 characters for the tail segment (to prevent 1-character fragments); fragments <0.3s merge into the previous line, not exceeding the 20-character cap
• Word-level timestamps are the only correct source for subtitle windows — segment-level timestamps will swallow inter-sentence silence, causing subtitles to appear about one silence segment early (already fixed; do not revert word_timestamps=False)
• CPU int8 inference: ≈28s audio outputs in under 1 minute; for longer audio, estimate proportionally; first run of large-v3-turbo includes download, so allocate sufficient timeout
• In Chinese recognition, numbers often become Arabic numerals ("two hundred milliseconds" → "200 milliseconds"); this is a normal difference during L2 comparison, not a recognition error

Delivery Wiring

Archive the final SRT/cues together with the original audio track source and model version; do not keep only the burned-in subtitle mp4. When delivering videos, use media-qc to check subtitle windows against actual duration; accuracy is still verified by known manuscript or spot-checking. Default model download and local inference do not replace manual dubbing.