Communitygithub.com

hangzhouspark/dsh-auto-subtitle

Turn any audio or video file into subtitles fully offline on this machine — auto-detect the spoken language among 99 languages, transcribe with word-level timestamps, re-segment into readable cues, translate into Chinese, and emit original+Chinese bilingual SRT/ASS, optionally burned into the video. Use when the user uploads, drops, attaches, or names an audio/video file (mp3, wav, m4a, aac, flac, ogg, opus, wma, mp4, mkv, mov, webm, avi, ts, flv) and wants subtitles, captions, a transcript, 字幕, 听写, or 转写; or says "转字幕", "生成字幕", "做个字幕", "加字幕", "翻译这段音频", or asks what was said in a recording — including when they give no instruction beyond the file itself.

dsh-auto-subtitle 是什么?

dsh-auto-subtitle is a Claude Code agent skill that turn any audio or video file into subtitles fully offline on this machine — auto-detect the spoken language among 99 languages, transcribe with word-level timestamps, re-segment into readable cues, translate into Chinese, and emit original+Chinese bilingual SRT/ASS, optionally burned into the video. Use when the user uploads, drops, attaches, or names an audio/video file (mp3, wav, m4a, aac, flac, ogg, opus, wma, mp4, mkv, mov, webm, avi, ts, flv) and wants subtitles, captions, a transcript, 字幕, 听写, or 转写; or says "转字幕", "生成字幕", "做个字幕", "加字幕", "翻译这段音频", or asks what was said in a recording — including when they give no instruction beyond the file itself.

兼容平台✓Claude Code~Codex CLI~Cursor
npx skills add hangzhouspark/dsh-auto-subtitle

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

Auto Subtitle

Turn any audio or video file into subtitles. Default output: bilingual subtitles in the original language + Chinese. Fully local inference, no API keys required.

Trigger

Handle both scenarios:

  1. Zero instruction: The user just drops an audio/video file without saying what to do → they want subtitles. Start immediately, don't ask back.
  2. One phrase: "转字幕", "生成字幕", "做个字幕", "加个字幕", "翻译这段音频" + file.

Only one case requires asking first: if the user explicitly says only Chinese (no original language line), then add --mono. For everything else, run with the bilingual default.

Environment

Three dependencies, all need to be in PATH:

DependencyPurposeVerification Command
Python 3.10+Runs the skill's scriptspython --version
babelscribeTranscription engine (whisper.cpp wrapper)babelscribe devices
ffmpegAudio transcoding, also used for --burnffmpeg -version

Install the engine: pip install babelscribe.

No need to manually download models: On first transcription, babelscribe automatically fetches ggml-large-v3-turbo.bin (~1.6 GB) to ~/.babelscribe/models/. It will download additional models when running a new language or using --accurate for the first time; this is normal, not a freeze.

With an NVIDIA GPU, it automatically uses the Vulkan backend—no need to compile CUDA yourself. babelscribe devices lists all available devices and shows which one auto will pick. Tested: on an RTX 3050 Ti Laptop (4 GB VRAM), a 48-second audio file transcribes in ~20 seconds, and a 51-minute live stream replay in ~13 minutes.

The skill's scripts are located in the scripts/ directory at the same level as SKILL.md.

When running on Windows, always include $env:PYTHONIOENCODING='utf-8', otherwise Chinese output may be garbled in the console.

Pipeline

Step 1 · Transcribe + Re-segment

$env:PYTHONIOENCODING='utf-8'
python scripts/make_subtitles.py "<audio/video path or URL>" --outdir "<output directory>"

Read the KEY=VALUE lines at the end of stdout:

KeyMeaning
SOURCE_LANGAuto-detected language, e.g. en, zh, ja
STEMBase filename for outputs, used in subsequent steps
CUES_JSONPackage of cues to be translated, input for Step 3
SOURCE_SRTOriginal language subtitles (already segmented, not the raw one-chunk version)
NEEDS_TRANSLATIONyes = needs translation into Chinese; no = original is already Chinese
PUNCTUATION_LOSTyes = segmentation quality is insufficient, go to Step 2
TRANSCRIPT_TXTAppears only if PUNCTUATION_LOST=yes, input for Step 2

Why re-segment: Whisper's native segments are often 7–8 seconds long with hundreds of characters—too cluttered for subtitles. The script uses word-level timestamps to re-segment into short lines (≤20 characters for Chinese, ≤48 for English) based on punctuation and pauses.

Step 2 · Punctuation restoration for Chinese/Japanese (only if PUNCTUATION_LOST=yes)

Whisper only sparsely outputs punctuation for Chinese and Japanese, relying on pauses for hard breaks. For Chinese, this can produce cuts like "转成文 / 字"; for Japanese, it's worse—Japanese has no spaces between words, and whisper's segments run end-to-end, making even pauses hard to find, resulting in every subtitle being hard-cut at exactly 20 characters, producing fragments like "申 / し訳ない" or "くま / るさん". The fix is to align the subtitles with actual punctuation:

  1. Read TRANSCRIPT_TXT (one long text block without punctuation).
  2. Only insert punctuation, don't change a single character. Even if the ASR has clearly wrong characters—e.g., it hears "machine translation" as "machine transla" or "word-level" as "word-let's"—keep them as is. Changing words will cause realignment to fail, and punctuation won't stick.
  3. Write the output as <STEM>.punct.txt (UTF-8).
  4. Re-run Step 1 with --punct-file (it will reuse <STEM>.raw.json so it's instant, no re-transcription):
python scripts/make_subtitles.py "<input>" --outdir "<output directory>" --punct-file "<STEM>.punct.txt"

Punctuation density is the only true metric for this step. Subtitles have a 20-character limit, so punctuation must be denser than in written prose: every 8–18 characters should have a comma or period. Insufficient density means subtitles still get hard-cut. Test with the same 51-minute Japanese audio:

Punctuation densityCue countCues ending on punctuation
None added82125%
~32 chars/punct (too sparse)100576%
8–18 chars/punct108198%

Use commas (、) at phrase boundaries, after particles for small pauses, and at natural breath points in speech—this is for subtitle line breaks, so it's okay to use them more frequently than in written Japanese; it's not damaging the text. Only use 。 、 ! ? …: brackets like 「」 are not in the script's punctuation table and will be dropped.

After adding punctuation, every cue in SOURCE_SRT should end on a punctuation mark. If you see WARN: ... alignment failure rate NN%, it means you changed the original text. Go back and only insert punctuation.

How to handle long audio: A 51-minute audio file has 14,000+ characters—too many to handle in one go. Chunk it into ~1,500-character blocks, give each block to a parallel agent (with 200 characters of context before and after, but only output the current block), and finally verify with a script that stripping all punctuation from the output results in an exact character-by-character match with the original before concatenating. This verification is not optional, and don't trust the agent's self-report—tested with 10 chunks, 2 of which said "verification passed" but actually lost or changed characters; only the script caught it.

Re-running with different density is cheap: Because raw.json is reused, the cost of experimenting with density is nearly zero. If after the first attempt only 76% of cues end on punctuation, just redo it—don't settle.

Step 3 · Translation (only if NEEDS_TRANSLATION=yes)

Read CUES_JSON, which is {"cues": [{"id":1,"start":...,"end":...,"text":"..."}]}.

  • First, read the entire segment and understand it as whole sentences, then distribute the Chinese translation across the individual cues. Translating each cue independently will break sentences that span multiple cues (e.g., "replacing people" in isolation might be translated as "取代人们" while losing the broader context).
  • Keep each Chinese line within 16 characters and match the cue's duration (Chinese reading speed ~4–5 characters/second; an 18-character line for a 1.2-second cue is too long for viewers to finish).
  • Write the translation as <STEM>.zh.json, {"1": "first line translation", "2": "..."}, with IDs matching the cues.
  • Keep terminology consistent; preserve proper nouns or use standard translations.

Step 4 · Merge outputs

python scripts/merge_bilingual.py "<STEM>.cues.json" "<STEM>.zh.json"

Optional parameters:

  • --mono — Keep only the translation, don't overlay the original (when user wants pure Chinese subtitles)
  • --order zh-first — Put Chinese on top (default: original on top)
  • --burn "<video path>" — Burn the bilingual subtitles into the video, output <STEM>.bilingual.burned.mp4. Only burn when the user has given a video file and explicitly needs hard subtitles; burning is slow and irreversible.

Outputs:

  • <STEM>.bilingual.srt — Bilingual SRT, compatible with any player/platform
  • <STEM>.bilingual.ass — Chinese large, original small script, suitable for burning or professional players

Step 5 · Report

After completion, give a one-sentence summary with key numbers (language, duration, cue count, output paths). Show 2–3 sample subtitle lines for the user to see quality. Don't paste large logs.

Common switches

ScenarioParameter
Noisy audio, heavy accent, lots of jargon--accurate (switches to large-v3 + beam search, ~2–3x slower but more accurate)
Auto language detection wrong--lang zh (or en, ja, th, …)
Want shorter/longer subtitles--max-chars 16 (counted in "characters" for Chinese)
Video URLPass the URL directly; the script uses yt-dlp internally
Re-run with new punctuation or segmentation strategy--punct-file <file> (auto-reuses <STEM>.raw.json, instant) or explicit --from-raw <STEM>.raw.json

Pitfalls

  • Don't use babelscribe's built-in .srt output as the final product—each line is one whisper segment without length control. The skill's scripts already re-segment correctly.
  • Don't "fix ASR errors" by modifying the transcript text. If you need to correct something, do it at the translation layer.
  • Pure music/silence files will return ERROR: transcription result is empty; this is not a bug, just tell the user.
  • For long audio (>30 minutes), Step 3 will have many cues. Translate in batches but keep IDs matching—don't miss any cues. A parallel agent reporting ok doesn't mean the file was actually written—tested with 20 translation batches: 1 returned status: ok without writing the file, 1 returned null. Before finishing, use a script to verify that every cue ID has a translation—don't trust the agent's self-report.
  • Don't use python -c "..." in PowerShell for scripts with quotes; the quotes will be eaten, causing SyntaxError. Write a .py file and run that.
  • Running a new language or switching to --accurate will download additional models for the first time; this is normal and takes time.

Self-test

After modifying the script, run regression tests with the fixed assets in examples/:

powershell -NoProfile -ExecutionPolicy Bypass -File "examples\gen_fixtures.ps1"
python scripts/make_subtitles.py "examples\audio\zh.wav" --outdir "examples\out"
python scripts/make_subtitles.py "examples\audio\en.wav" --outdir "examples\out"

examples\zh.txt / examples\en.txt are synthesized into examples\audio\zh.wav / en.wav by Windows SAPI (requires the Microsoft Huihui Desktop Chinese voice, which comes with Chinese Windows). Expected results:

  • Chinese: 9 cues without punctuation, ~6 of which have no punctuation at the end (PUNCTUATION_LOST=yes); after adding punctuation as described, 10 cues, all ending with punctuation
  • English: 19 cues, maximum ≤48 characters

相关技能