Communitygithub.com

dsh-auto-subtitle

Convert any audio or video file into subtitles fully offline on this machine — auto-detect the spoken language among 99 languages, transcribe with word-level timestamps, re-segment into readable cues, translate into Chinese, and emit original+Chinese bilingual SRT/ASS, optionally burned into the video.

Qu'est-ce que dsh-auto-subtitle ?

This skill, dsh-auto-subtitle, is a fully offline AI agent skill that converts any audio or video file (local or URL) into subtitles. The primary output is a bilingual subtitle file with original language and Chinese. It works by detecting the spoken language among 99 languages, transcribing with word-level timestamps using babelscribe (a whisper.cpp wrapper), re-segmenting into readable short cues, translating into Chinese, and merging into final SRT or ASS format. The skill handles both zero-instruction scenarios (user drops a file) and explicit requests like 'generate subtitles'. It is designed for the terminal and uses Python, ffmpeg, and babelscribe as dependencies. Key features include optional monochrome output, accuracy modes for noisy audio, and a bulletproof punctuation repair step for Chinese and Japanese. The skill is ideal for developers or users needing to quickly generate high-quality bilingual subtitles without cloud dependencies.

Compatible avec✓Claude Code~Codex CLI~Cursor
npx skills add hangzhouspark/dsh-auto-subtitle

Demander à votre IA préférée

Ouvre une nouvelle conversation avec cette compétence d'agent déjà préchargée.

Documentation

Auto Subtitle

Convert any audio or video into subtitles. Default output is bilingual (original + Chinese). Entirely local inference, no API key required.

Trigger

Handle both cases:

  1. Zero instruction: User just drops an audio/video file without saying anything → they want subtitles. Just run, don't ask.
  2. One sentence: "Convert to subtitles", "Generate subtitles", "Make subtitles", "Add subtitles", "Translate this audio" + file.

Only one case requires a prior question: if the user explicitly says Chinese only (no original line), add --mono. Otherwise, always run with default bilingual output.

Environment

Three dependencies, all accessible in PATH:

DependencyPurposeVerification Command
Python 3.10+Runs the skill's scriptspython --version
babelscribeTranscription engine (whisper.cpp wrapper)babelscribe devices
ffmpegAudio transcoding, --burn also uses itffmpeg -version

To install engine: pip install babelscribe.

Models are not downloaded manually: On first transcription, babelscribe automatically downloads ggml-large-v3-turbo.bin (approximately 1.6 GB) to ~/.babelscribe/models/. Additional models may be downloaded on first run for a certain language or when using --accurate. This is normal, not a hang.

If you have an NVIDIA GPU, it automatically selects the Vulkan backend – no need to compile CUDA yourself. babelscribe devices lists all available devices and shows which one auto would pick. In practice, on an RTX 3050 Ti Laptop (4 GB VRAM), a 48-second audio takes about 20 seconds to transcribe, a 51-minute live stream replay takes about 13 minutes.

The skill's scripts are located in the scripts/ directory alongside the SKILL.md.

On Windows, always run with $env:PYTHONIOENCODING='utf-8', otherwise Chinese output may be garbled in the console.

Procedure

Step 1 · Transcription + Re‑segmentation

$env:PYTHONIOENCODING='utf-8'
python scripts/make_subtitles.py "<audio/video path or URL>" --outdir "<output directory>"

Read the KEY=VALUE lines at the end of stdout:

KeyMeaning
SOURCE_LANGAuto-detected language, e.g., en, zh, ja
STEMBase name of the output files; used for subsequent steps
CUES_JSONPackage for translation, input for Step 3
SOURCE_SRTOriginal language subtitles (already re‑segmented, not the raw blob)
NEEDS_TRANSLATIONyes = needs translation to Chinese; no = original is already Chinese
PUNCTUATION_LOSTyes = segmentation quality is insufficient; proceed to Step 2
TRANSCRIPT_TXTOnly appears when PUNCTUATION_LOST=yes; input for Step 2

Why re‑segment: Whisper’s native segments are often 7‑8 seconds long with hundreds of characters, which overcrowds the screen. The script re‑segments word‑level timestamps by punctuation and pauses into short cues of ≤20 characters (Chinese) / ≤48 characters (English).

Step 2 · Add Punctuation to Chinese/Japanese Original (only when PUNCTUATION_LOST=yes)

Whisper outputs only sporadic punctuation for Chinese and Japanese, relying mostly on pauses for segmentation. For Chinese, it may cut words like "转成文 / 字". For Japanese, it's worse – there are no spaces between words, and whisper's segments are contiguous, making it impossible to even locate pauses, resulting in each subtitle cut exactly at 20 characters, producing things like "申 / し訳ない" or "くま / るさん". The fix is to place the subtitle breaks at actual punctuation points:

  1. Read TRANSCRIPT_TXT (a single long text without punctuation).
  2. Only add punctuation, without changing a single character. Keep words that are obviously misrecognized by the ASR – if it heard "击翻" instead of "机翻", or "自己" instead of "词级", or "おやすい" instead of "おやすみ", preserve them unchanged. Changing words will cause re‑alignment to fail, and punctuation cannot be re‑inserted.
  3. Write to <STEM>.punct.txt (UTF‑8).
  4. Re‑run Step 1 with --punct-file (it will automatically reuse <STEM>.raw.json, completing in seconds without re‑transcribing):
python scripts/make_subtitles.py "<input>" --outdir "<output directory>" --punct-file "<STEM>.punct.txt"

The density of punctuation is the only truly important metric. Subtitle limit is 20 characters, so punctuation must be denser than in written language: One punctuation mark (、 or 。) every 8‑18 characters. If density is insufficient, subtitles will still be cut arbitrarily – as verified with a 51‑minute Japanese audio:

Punctuation DensityNumber of CuesEnding on Punctuation
Not added82125%
~32 chars/punct (too sparse)100576%
8‑18 chars/punct108198%

Use 、 at phrase boundaries, after particles, at conversational pauses – it serves subtitle line‑breaking, which is more frequent than in written Japanese and is not a corruption of the text. Also, only 。 、 ! ? … are allowed: parentheses like 「」 are not in the script's punctuation table and will be discarded outright.

After adding punctuation, each subtitle in SOURCE_SRT will end with a punctuation mark. If the output shows WARN: ... alignment failure rate NN%, it means the original text was changed – go back, only add punctuation, and retry.

How to add punctuation for long audio: A 51‑minute audio has over 14,000 characters, which cannot be done in one go. Split into chunks of ~1500 characters, give each chunk to a parallel agent (with 200 characters of context before and after, but output only for this chunk), then verify with a script that 'removing all punctuation results in character‑by‑character equality with the original' before concatenating. This verification step is mandatory and cannot be replaced by the agent's own report – in practice, 2 out of 10 chunks reported 'passed character‑by‑character verification' while actually missing or changing characters.

Re‑running with different density is cheap: Because raw.json is reused, the cost of trying different densities is nearly zero. When the first try achieves only 76% ending on punctuation, just redo the punctuation addition – don't settle.

Step 3 · Translation (only when NEEDS_TRANSLATION=yes)

Read CUES_JSON, which contains {"cues": [{"id":1,"start":...,"end":...,"text":"..."}]}.

  • First, read the entire chunk and understand the full sentences, then assign Chinese text to each cue. Translating each cue independently will break a single sentence that spans multiple cues (e.g., "replacing people." isolated would be translated as "取代人们" while losing context).
  • Limit each Chinese cue to 16 characters, and ensure it fits the duration of that cue (Chinese reading speed is approximately 4‑5 characters per second; a 1.2‑second cue cannot display 18 characters).
  • Write the translation as <STEM>.zh.json, {"1": "first translation", "2": "..."}, where IDs must match those in CUES_JSON.
  • Maintain consistent terminology; preserve proper nouns in their original form or use commonly accepted translations.

Step 4 · Generate Final Output

python scripts/merge_bilingual.py "<STEM>.cues.json" "<STEM>.zh.json"

Optional parameters:

  • --mono – Only keep the translated text, no original (when user wants pure Chinese subtitles)
  • --order zh-first – Chinese on top (default: original on top)
  • --burn "<video path>" – Burn bilingual subtitles into the video, outputting <STEM>.bilingual.burned.mp4. Only burn when the user provides a video and really needs hard‑burned subtitles – burning is slow and irreversible.

Output:

  • <STEM>.bilingual.srt – Bilingual SRT, compatible with any player/platform
  • <STEM>.bilingual.ass – Chinese larger, original smaller – suitable for burning or professional players

Step 5 · Report

Provide a one‑sentence conclusion with key numbers (language, duration, number of cues, output paths). Paste 2‑3 example subtitle lines so the user can see quality quickly. Do not paste large logs.

Common Switches

ScenarioParameters
Noisy audio, heavy accent, technical terms--accurate (uses large‑v3 + beam search, ~2‑3× slower but more accurate)
Auto‑detected language is wrong--lang zh (or en, ja, th, …)
Want shorter/longer subtitles--max-chars 16 (Chinese counts in characters)
Video URLPass the URL directly; script uses yt‑dlp internally
Re‑run with different punctuation/segmentation--punct-file <file> (automatically reuses <STEM>.raw.json, seconds) or explicit --from-raw <STEM>.raw.json

Gotchas

  • Do not use babelscribe's built‑in .srt output as the final product – each line is a single whisper segment, without length control. The skill's script already re‑segments.
  • Do not modify the transcribed text just to 'fix typos'. If you need to change something, do it at the translation layer.
  • Pure music / silent files will return ERROR: Transcription result is empty. This is not a bug; inform the user accordingly.
  • For long audio (>30 minutes), the CUES_JSON in Step 3 may contain many cues. Translate them in batches, keeping ID correspondence, without missing any. A parallel agent reporting ok does not mean the file was actually written – in practice, 1 out of 20 translation batches returned status: ok but didn't write a file, and 1 returned null. Before finishing, always verify with a script that 'every cue ID has a corresponding translation' – don't trust the agent's self‑report.
  • Do not use python -c "..." in PowerShell to run scripts with quotes – quotes will be eaten, causing SyntaxError. Write scripts as .py files.
  • First run for a certain language / using --accurate will download additional models – this is normal.

Self‑Test

After modifying the script, run regression with the fixed materials under examples/:

powershell -NoProfile -ExecutionPolicy Bypass -File "examples\gen_fixtures.ps1"
python scripts/make_subtitles.py "examples\audio\zh.wav" --outdir "examples\out"
python scripts/make_subtitles.py "examples\audio\en.wav" --outdir "examples\out"

examples\zh.txt / examples\en.txt are spoken by Windows SAPI into examples\audio\zh.wav / en.wav (requires the Chinese voice Microsoft Huihui Desktop, which comes with Chinese Windows). Expected results:

  • Chinese (zh): Without punctuation, 9 cues, of which about 6 end with no punctuation (PUNCTUATION_LOST=yes); after adding punctuation as described, 10 cues, all ending with punctuation
  • English (en): 19 cues, longest ≤48 characters

Skills associés