bilingual-video-subtitles

Bilingual Video Subtitles 培训视频双语字幕

The complete pipeline is in scripts/pipeline.py (single entry point with caching support for resume after interruption).

Host environment note: This skill is adapted for cross-environment operation; the transcription backend is automatically selected (original sandbox Omni gateway / OpenAI-compatible API / whisper bridge). If no Omni gateway is available and no API backend is configured, it falls back to agent mode — the transcribe/translate stages pause with exit code 3 and write out a manifest. After completing the corresponding transcription/translation, rerun build to resume. Read the operational protocol and backend configuration guide at references/host-adaptation.md. Encountering pipeline exit 3 is not a failure; complete the work indicated in the manifest and rerun.

Standard Workflow

1. Obtain input: video file + corresponding regulatory/guidelines documents (.docx/.txt/.md).
2. Read the reference document and create a glossary glossary.txt (one line per entry: Chinese = English, 10–40 entries). The glossary determines the translation consistency across the entire video. Must cover: system/policy names, roles and acronyms (e.g., PM=Project Manager), business forms or form names, internal system names.
3. Execute the build (run in foreground with sufficient timeout; a 50-minute video full workflow takes about 15–25 minutes):

python3 scripts/pipeline.py build \
  --video <video path> --doc <document path> \
  --work <working directory> --out <output directory> \
  --glossary <glossary.txt> \
  --hotwords "安全红线,需求单,变更单,PM,QA,SRE" \
  --preview 90

4. Generation sequence (each step is cached; rerun the same command after a failure to resume): setup (audio segmentation) → transcribe (Qwen3-Omni full manuscript) → timing (whisper timestamps) → align (pinyin alignment) → clean (remove filler words, split sentences) → translate (batch Chinese-to-English) → package (project file + SRT + ASS) → render (preview + full version).
5. Use the preview video output from --preview to confirm the effect with the user (you can use media-read to check frame samples for subtitle rendering), then deliver the full video, SRT, and subtitle project file.

Deliverables

File	Purpose
<name>_中英双语字幕_完整版.mp4	Hard-subtitled final video (Chinese in white top line, English in light blue bottom line, both with a semi-transparent black background bar, readable on any background)
<name>_中英双语字幕.srt	Bilingual subtitle file (can be imported into players/platforms)
<name>_字幕工程.json	Master data: contains all timestamps + bilingual text. Only this file is modified for annual adjustments.
--work directory	Intermediate cache (transcription, whisper timestamps). Keep to avoid re-transcription.

Annual Fine-Tuning (Updating Subtitles After Policy Revisions)

When policy changes affect only a few sentences, do not re-transcribe:

1. Ask the user to point out the timespan/content to modify, or directly edit the zh/en text in the previous year's <name>_字幕工程.json (do not modify start/end timestamps if unchanged).
2. Re-render:

python3 scripts/pipeline.py render --project <project.json> [--video <new video>] [--srt-only]

If the new video has added/removed sentences in corresponding chapters: run build --stage transcribe on the new video to get the new manuscript, manually compare and replace the corresponding entries in the project file, then render.

Splitting Videos by Topic/Role (Microlearning)

Based on the fully subtitled, full-length video (preserving bilingual subtitles).

1. Locate chapter boundaries from the subtitle data: use the role/chapter list from the reference document as keywords to search for chapter transition sentences in zh_lines.json (working directory). Typical patterns: "The Nth role is X", "Above are the responsibilities of X, the requirement is:", "Violations for X are:", etc. Use the start time of the transition sentence as the split point (that sentence belongs to the next clip).
2. Constraints: Each clip ≤10 minutes (hard limit 15 minutes); if a chapter exceeds the limit, split within the chapter at natural boundaries (e.g., between "Duties/Violations/Determination Methods"); do not merge different roles.
3. Write a split plan JSON and execute:

{"base": "video_name", "lead_in": 0.35,
 "chapters": [{"start": 0, "end": 288.4,
               "title": "01_Introduction_Purpose and Overview", "note": "Content description"}]}

python3 scripts/pipeline.py split --plan plan.json \
  --video <subtitled_full_video.mp4> --out <output directory>

4. Output to <output directory>/<video_name>_splits/: automatically named clips + 00_split_manifest.txt (clip name, time range, duration, content description).
5. Spot check: Extract frames (ffmpeg PNG capture) at the beginning of each clip to verify the split point lands exactly on the corresponding topic's opening sentence.

Key Implementation Constraints (Do Not Deviate)

• The qwen-omni backend must inject the full reference document into the prompt as terminology context, with --timeout 900 --max-tokens 8192 (script has this built-in); API backends with dedicated ASR (e.g., mimo-asr) cannot accept text prompts, so terminology correction relies on the clean stage cross-referencing the document.
• whisper is used only for timestamps; its transcribed text is discarded entirely (Chinese text quality is poor, but timestamps are accurate).
• Alignment must use pinyin-based global alignment (homophone errors are automatically resolved at the pinyin level); greedy per-sentence alignment causes global drift.
• Alignment quality: align output qual mean ≥0.75 is normal; if <0.6, first check whisper voice coverage (see references/troubleshooting.md).
• For long-duration CPU-intensive tasks (whisper/render), be aware of the host's background task cleanup strategy; run in foreground if necessary.
• Network restrictions: HuggingFace requires a mirror; the script already has HF_ENDPOINT=hf-mirror.com and HF_HUB_DISABLE_XET=1 set; do not remove them.

Troubleshooting

See references/troubleshooting.md: handling upload timeouts, single long lines in transcription, low alignment quality, font garbling, render failures, etc.