Auto Subtitle
Convert any audio or video into subtitles. Default output is bilingual (original + Chinese). Entirely local inference, no API key required.
Trigger
Handle both cases:
- Zero instruction: User just drops an audio/video file without saying anything → they want subtitles. Just run, don't ask.
- One sentence: "Convert to subtitles", "Generate subtitles", "Make subtitles", "Add subtitles", "Translate this audio" + file.
Only one case requires a prior question: if the user explicitly says Chinese only (no original line), add --mono. Otherwise, always run with default bilingual output.
Environment
Three dependencies, all accessible in PATH:
| Dependency | Purpose | Verification Command |
|---|---|---|
| Python 3.10+ | Runs the skill's scripts | python --version |
| babelscribe | Transcription engine (whisper.cpp wrapper) | babelscribe devices |
| ffmpeg | Audio transcoding, --burn also uses it | ffmpeg -version |
To install engine: pip install babelscribe.
Models are not downloaded manually: On first transcription, babelscribe automatically downloads ggml-large-v3-turbo.bin (approximately 1.6 GB) to ~/.babelscribe/models/. Additional models may be downloaded on first run for a certain language or when using --accurate. This is normal, not a hang.
If you have an NVIDIA GPU, it automatically selects the Vulkan backend – no need to compile CUDA yourself. babelscribe devices lists all available devices and shows which one auto would pick. In practice, on an RTX 3050 Ti Laptop (4 GB VRAM), a 48-second audio takes about 20 seconds to transcribe, a 51-minute live stream replay takes about 13 minutes.
The skill's scripts are located in the scripts/ directory alongside the SKILL.md.
On Windows, always run with $env:PYTHONIOENCODING='utf-8', otherwise Chinese output may be garbled in the console.
Procedure
Step 1 · Transcription + Re‑segmentation
$env:PYTHONIOENCODING='utf-8'
python scripts/make_subtitles.py "<audio/video path or URL>" --outdir "<output directory>"
Read the KEY=VALUE lines at the end of stdout:
| Key | Meaning |
|---|---|
SOURCE_LANG | Auto-detected language, e.g., en, zh, ja |
STEM | Base name of the output files; used for subsequent steps |
CUES_JSON | Package for translation, input for Step 3 |
SOURCE_SRT | Original language subtitles (already re‑segmented, not the raw blob) |
NEEDS_TRANSLATION | yes = needs translation to Chinese; no = original is already Chinese |
PUNCTUATION_LOST | yes = segmentation quality is insufficient; proceed to Step 2 |
TRANSCRIPT_TXT | Only appears when PUNCTUATION_LOST=yes; input for Step 2 |
Why re‑segment: Whisper’s native segments are often 7‑8 seconds long with hundreds of characters, which overcrowds the screen. The script re‑segments word‑level timestamps by punctuation and pauses into short cues of ≤20 characters (Chinese) / ≤48 characters (English).
Step 2 · Add Punctuation to Chinese/Japanese Original (only when PUNCTUATION_LOST=yes)
Whisper outputs only sporadic punctuation for Chinese and Japanese, relying mostly on pauses for segmentation. For Chinese, it may cut words like "转成文 / 字". For Japanese, it's worse – there are no spaces between words, and whisper's segments are contiguous, making it impossible to even locate pauses, resulting in each subtitle cut exactly at 20 characters, producing things like "申 / し訳ない" or "くま / るさん". The fix is to place the subtitle breaks at actual punctuation points:
- Read
TRANSCRIPT_TXT(a single long text without punctuation). - Only add punctuation, without changing a single character. Keep words that are obviously misrecognized by the ASR – if it heard "击翻" instead of "机翻", or "自己" instead of "词级", or "おやすい" instead of "おやすみ", preserve them unchanged. Changing words will cause re‑alignment to fail, and punctuation cannot be re‑inserted.
- Write to
<STEM>.punct.txt(UTF‑8). - Re‑run Step 1 with
--punct-file(it will automatically reuse<STEM>.raw.json, completing in seconds without re‑transcribing):
python scripts/make_subtitles.py "<input>" --outdir "<output directory>" --punct-file "<STEM>.punct.txt"
The density of punctuation is the only truly important metric. Subtitle limit is 20 characters, so punctuation must be denser than in written language: One punctuation mark (、 or 。) every 8‑18 characters. If density is insufficient, subtitles will still be cut arbitrarily – as verified with a 51‑minute Japanese audio:
| Punctuation Density | Number of Cues | Ending on Punctuation |
|---|---|---|
| Not added | 821 | 25% |
| ~32 chars/punct (too sparse) | 1005 | 76% |
| 8‑18 chars/punct | 1081 | 98% |
Use 、 at phrase boundaries, after particles, at conversational pauses – it serves subtitle line‑breaking, which is more frequent than in written Japanese and is not a corruption of the text. Also, only 。 、 ! ? … are allowed: parentheses like 「」 are not in the script's punctuation table and will be discarded outright.
After adding punctuation, each subtitle in SOURCE_SRT will end with a punctuation mark. If the output shows WARN: ... alignment failure rate NN%, it means the original text was changed – go back, only add punctuation, and retry.
How to add punctuation for long audio: A 51‑minute audio has over 14,000 characters, which cannot be done in one go. Split into chunks of ~1500 characters, give each chunk to a parallel agent (with 200 characters of context before and after, but output only for this chunk), then verify with a script that 'removing all punctuation results in character‑by‑character equality with the original' before concatenating. This verification step is mandatory and cannot be replaced by the agent's own report – in practice, 2 out of 10 chunks reported 'passed character‑by‑character verification' while actually missing or changing characters.
Re‑running with different density is cheap: Because raw.json is reused, the cost of trying different densities is nearly zero. When the first try achieves only 76% ending on punctuation, just redo the punctuation addition – don't settle.
Step 3 · Translation (only when NEEDS_TRANSLATION=yes)
Read CUES_JSON, which contains {"cues": [{"id":1,"start":...,"end":...,"text":"..."}]}.
- First, read the entire chunk and understand the full sentences, then assign Chinese text to each cue. Translating each cue independently will break a single sentence that spans multiple cues (e.g., "replacing people." isolated would be translated as "取代人们" while losing context).
- Limit each Chinese cue to 16 characters, and ensure it fits the duration of that cue (Chinese reading speed is approximately 4‑5 characters per second; a 1.2‑second cue cannot display 18 characters).
- Write the translation as
<STEM>.zh.json,{"1": "first translation", "2": "..."}, where IDs must match those inCUES_JSON. - Maintain consistent terminology; preserve proper nouns in their original form or use commonly accepted translations.
Step 4 · Generate Final Output
python scripts/merge_bilingual.py "<STEM>.cues.json" "<STEM>.zh.json"
Optional parameters:
--mono– Only keep the translated text, no original (when user wants pure Chinese subtitles)--order zh-first– Chinese on top (default: original on top)--burn "<video path>"– Burn bilingual subtitles into the video, outputting<STEM>.bilingual.burned.mp4. Only burn when the user provides a video and really needs hard‑burned subtitles – burning is slow and irreversible.
Output:
<STEM>.bilingual.srt– Bilingual SRT, compatible with any player/platform<STEM>.bilingual.ass– Chinese larger, original smaller – suitable for burning or professional players
Step 5 · Report
Provide a one‑sentence conclusion with key numbers (language, duration, number of cues, output paths). Paste 2‑3 example subtitle lines so the user can see quality quickly. Do not paste large logs.
Common Switches
| Scenario | Parameters |
|---|---|
| Noisy audio, heavy accent, technical terms | --accurate (uses large‑v3 + beam search, ~2‑3× slower but more accurate) |
| Auto‑detected language is wrong | --lang zh (or en, ja, th, …) |
| Want shorter/longer subtitles | --max-chars 16 (Chinese counts in characters) |
| Video URL | Pass the URL directly; script uses yt‑dlp internally |
| Re‑run with different punctuation/segmentation | --punct-file <file> (automatically reuses <STEM>.raw.json, seconds) or explicit --from-raw <STEM>.raw.json |
Gotchas
- Do not use babelscribe's built‑in
.srtoutput as the final product – each line is a single whisper segment, without length control. The skill's script already re‑segments. - Do not modify the transcribed text just to 'fix typos'. If you need to change something, do it at the translation layer.
- Pure music / silent files will return
ERROR: Transcription result is empty. This is not a bug; inform the user accordingly. - For long audio (>30 minutes), the
CUES_JSONin Step 3 may contain many cues. Translate them in batches, keeping ID correspondence, without missing any. A parallel agent reportingokdoes not mean the file was actually written – in practice, 1 out of 20 translation batches returnedstatus: okbut didn't write a file, and 1 returned null. Before finishing, always verify with a script that 'every cue ID has a corresponding translation' – don't trust the agent's self‑report. - Do not use
python -c "..."in PowerShell to run scripts with quotes – quotes will be eaten, causingSyntaxError. Write scripts as.pyfiles. - First run for a certain language / using
--accuratewill download additional models – this is normal.
Self‑Test
After modifying the script, run regression with the fixed materials under examples/:
powershell -NoProfile -ExecutionPolicy Bypass -File "examples\gen_fixtures.ps1"
python scripts/make_subtitles.py "examples\audio\zh.wav" --outdir "examples\out"
python scripts/make_subtitles.py "examples\audio\en.wav" --outdir "examples\out"
examples\zh.txt / examples\en.txt are spoken by Windows SAPI into examples\audio\zh.wav / en.wav (requires the Chinese voice Microsoft Huihui Desktop, which comes with Chinese Windows). Expected results:
- Chinese (zh): Without punctuation, 9 cues, of which about 6 end with no punctuation (
PUNCTUATION_LOST=yes); after adding punctuation as described, 10 cues, all ending with punctuation - English (en): 19 cues, longest ≤48 characters