ZJU-REAL/Easel

Automatic Subtitles (Speech-to-Text)

Recognizes human speech in audio/video and generates subtitles. Based on the shared script skills/shared/scripts/asr.py (faster-whisper wrapper), with deterministic parameters and reproducible results, featuring Chinese-friendly sentence segmentation. Optionally burn subtitles into video (reuses skills/shared/scripts/video_ops.py / FFmpeg subtitles filter).

This skill handles only "speech → subtitle files (+ optional burning)". For general video editing, see video-editing; for pure noise reduction, see audio-denoise; for smart clipping of long videos with burned subtitles, see clipify.

Input

Field	Required	Description
input_file	Yes	Path to audio or video file (video will auto-extract audio track)
format	No	srt (default) / ass / txt / json
language	No	auto (default) or zh/en etc. ISO 639-1 code
model	No	tiny/base (default)/small/medium/large-v3; larger = more accurate but slower
burn	No	Whether to burn subtitles into the video (requires video input)

Supports: audio formats mp3/wav/m4a/aac/flac; video formats mp4/mkv/mov/webm.

Output

• Subtitle files placed in outputs/topic-name/ (SRT/ASS/TXT/JSON)
• If burning subtitles: video with hardcoded subtitles (*-sub.mp4)
• Report: detected language, number of subtitle segments, model used, output paths

Prerequisites

• On first run, models are downloaded from HuggingFace; requires an external network proxy. The script reads EASEL_PROXY or http(s)_proxy environment variables as the proxy; if neither is set, it connects directly. You can also pre-set with export https_proxy=... http_proxy=....
• For CPU environments, use the defaults --device cpu --compute-type int8.

Execution Steps

Script paths (relative to project root): skills/shared/scripts/asr.py, skills/shared/scripts/video_ops.py.

1. Generate subtitle file

# Video → SRT (auto-extract audio track + auto-detect language)
python skills/shared/scripts/asr.py transcribe \
  -i input.mp4 -o outputs/topic-name/input.srt --language zh

# Video → ASS (with styling, **font size and margins auto-adapt to video portrait/landscape**): video input auto-detects dimensions
python skills/shared/scripts/asr.py transcribe \
  -i input.mp4 -o outputs/topic-name/input.ass --format ass --model small
# Audio only → ASS: cannot detect dimensions, defaults to portrait 1080x1920; for landscape add --res 1920x1080
python skills/shared/scripts/asr.py transcribe \
  -i voice.mp3 -o outputs/topic-name/voice.ass --format ass --res 1920x1080

• ASS styles auto-adapt to target video dimensions: font size based on the short side (min(w,h)*0.05, suitable for both portrait and landscape, no longer oversized on landscape), bottom margin based on height, left/right margins based on width, PlayRes = actual width/height. Video input auto-detects; for audio only, specify with --res widthxheight. For hardcoded subtitles with styling, prefer burning this ASS file (next section) — more reliable and auto-adaptive than bare SRT with manual force_style.
• Chinese defaults to ~18 characters per line; long lines auto-break/split; adjust with --max-line-chars.
• If -o is omitted, a project directory is created from the input filename, e.g., talk.mp4 outputs to outputs/talk/talk.srt; for existing projects, explicitly use -o outputs/topic-name/<filename>.<format>.
• View available models/languages: python skills/shared/scripts/asr.py info.

2. (Optional) Burn subtitles into video

When the user wants "hardcoded subtitles / burn into video", use FFmpeg's subtitles (for SRT) or ass (for ASS) filter:

# Burn SRT (customizable style)
ffmpeg -y -i input.mp4 \
  -vf "subtitles=outputs/topic-name/input.srt:force_style='FontName=Noto Sans CJK SC,FontSize=20,PrimaryColour=&H00FFFFFF,OutlineColour=&H80000000,BorderStyle=1,Outline=2'" \
  -c:a copy outputs/topic-name/input-sub.mp4

# Burn ASS (styles already in ASS file, faithful reproduction)
ffmpeg -y -i input.mp4 \
  -vf "ass=outputs/topic-name/input.ass" \
  -c:a copy outputs/topic-name/input-sub.mp4

• If paths contain special characters, wrap the subtitles= value in single quotes and escape colons.
• For soft subtitles (toggleable) instead of burning: ffmpeg -i in.mp4 -i sub.srt -c copy -c:s mov_text out.mp4.

3. Output and Report

• Subtitle files and burned video are written to outputs/topic-name/.
• Report to user: detected language, number of subtitle segments, model, file paths; suggest rerunning with --model/--max-line-chars if needed.

Rules

1. Video: extract audio track first — the script automatically uses FFmpeg to extract a 16kHz mono WAV, leaving the original video unchanged.
2. Chinese-friendly sentence segmentation — break lines by punctuation/length, avoid overly long lines.
3. Never delete original files — only produce new files in outputs/.
4. Model selection — use base for speed, small/medium for accuracy; CPU use int8.
5. Burning requires video input — audio-only input produces subtitle files only.
6. No profile dependency — transcription does not require user profiling.

References

• SYSTRAN/faster-whisper — CTranslate2-accelerated Whisper inference
• openai/whisper — Original Whisper model and ASR method
• FFmpeg subtitles / ass filter documentation (subtitle burning)