Clip Talking-Head v2
Volcengine Transcription + AI Speech Error Detection + Web Review
Quick Start
User: Help me clip this talking-head video
User: Process this video
Output Directory Structure
output/
└── YYYY-MM-DD_VideoName/
├── ClipTalkingHead/
│ ├── 1_Transcription/
│ │ ├── audio.mp3
│ │ ├── volcengine_result.json
│ │ └── subtitles_words.json
│ ├── 2_Analysis/
│ │ ├── readable.txt
│ │ ├── auto_selected.json
│ │ └── speech_error_analysis.md
│ └── 3_Review/
│ └── review.html
└── Subtitles/
└── …
Rule: If a folder already exists, reuse it; otherwise, create a new one.
Flow
0. Create output directory
↓
1. Extract audio (ffmpeg)
↓
2. Upload to get public URL (uguu.se)
↓
3. Volcengine API transcription
↓
4. Generate word-level subtitles (subtitles_words.json)
↓
5. AI analysis of speech errors/silences -> pre-selected list (auto_selected.json)
↓
6. Generate review webpage (review.html)
↓
7. Start review server, user confirms via web
↓
【Wait for user confirmation】 → click「Execute Clip」on web page or manually /clip
Execution Steps
Step 0: Create Output Directory
# Variable settings (adjust according to actual video)
VIDEO_PATH="/path/to/video.mp4"
VIDEO_NAME=$(basename "$VIDEO_PATH" .mp4)
DATE=$(date +%Y-%m-%d)
BASE_DIR="output/${DATE}_${VIDEO_NAME}/ClipTalkingHead"
# Create subdirectories
mkdir -p "$BASE_DIR/1_Transcription" "$BASE_DIR/2_Analysis" "$BASE_DIR/3_Review"
cd "$BASE_DIR"
Steps 1-3: Transcription
cd 1_Transcription
# 1. Extract audio (if filename has colons, prepend file: prefix)
ffmpeg -i "file:$VIDEO_PATH" -vn -acodec libmp3lame -y audio.mp3
# 2. Upload to get public URL
curl -s -F "files[][email protected]" https://uguu.se/upload
# Returns: {"success":true,"files":[{"url":"https://h.uguu.se/xxx.mp3"}]}
# 3. Invoke Volcengine API
SKILL_DIR="/Users/chengfeng/Desktop/AIos/EditingAgent/.claude/skills/ClipTalkingHead"
"$SKILL_DIR/scripts/volcengine_transcribe.sh" "https://h.uguu.se/xxx.mp3"
# Output: volcengine_result.json
Step 4: Generate Subtitles
node "$SKILL_DIR/scripts/generate_subtitles.js" volcengine_result.json
# Output: subtitles_words.json
cd ..
Step 5: Analyze Speech Errors (Script + AI)
5.1 Generate Readable Format
cd 2_Analysis
node -e "
const data = require('../1_Transcription/subtitles_words.json');
let output = [];
data.forEach((w, i) => {
if (w.isGap) {
const dur = (w.end - w.start).toFixed(2);
if (dur >= 0.5) output.push(i + '|[Sil' + dur + 's]|' + w.start.toFixed(2) + '-' + w.end.toFixed(2));
} else {
output.push(i + '|' + w.text + '|' + w.start.toFixed(2) + '-' + w.end.toFixed(2));
}
});
require('fs').writeFileSync('readable.txt', output.join('\\n'));
"
5.2 Read User Preferences
First, read all rule files in the UserPreferences/ directory.
5.3 Generate Sentence List (Critical Step)
Must split into sentences first, then analyze. Split into sentence list by silence:
node -e "
const data = require('../1_Transcription/subtitles_words.json');
let sentences = [];
let curr = { text: '', startIdx: -1, endIdx: -1 };
data.forEach((w, i) => {
const isLongGap = w.isGap && (w.end - w.start) >= 0.5;
if (isLongGap) {
if (curr.text.length > 0) sentences.push({...curr});
curr = { text: '', startIdx: -1, endIdx: -1 };
} else if (!w.isGap) {
if (curr.startIdx === -1) curr.startIdx = i;
curr.text += w.text;
curr.endIdx = i;
}
});
if (curr.text.length > 0) sentences.push(curr);
sentences.forEach((s, i) => {
console.log(i + '|' + s.startIdx + '-' + s.endIdx + '|' + s.text);
});
" > sentences.txt
5.4 Script Auto-Tagging Silences (Must Execute First)
node -e "
const words = require('../1_Transcription/subtitles_words.json');
const selected = [];
words.forEach((w, i) => {
if (w.isGap && (w.end - w.start) >= 0.5) selected.push(i);
});
require('fs').writeFileSync('auto_selected.json', JSON.stringify(selected, null, 2));
console.log('Number of silences >= 0.5s:', selected.length);
"
→ Output auto_selected.json (only silence indices)
5.5 AI Analysis of Speech Errors (Append to auto_selected.json)
Detection Rules (priority order):
| # | Type | Detection Method | Removal Range |
|---|---|---|---|
| 1 | Repeated sentence | Adjacent sentences share ≥5 starting characters | Shorter entire sentence |
| 2 | One-sentence-apart repetition | When middle is a fragment, compare surrounding sentences | Previous sentence + fragment |
| 3 | Fragment | Half-spoken sentence + silence | Entire fragment |
| 4 | Within-sentence repetition | Pattern A + middle + A | Earlier part |
| 5 | Filler words | 'that that', 'like like' type | Earlier part |
| 6 | Restart/correction | Partial repetition/negation correction | Earlier part |
| 7 | Filler words | 'um', 'ah', 'like' | Mark but do not auto-delete |
Core Principles:
- Split sentences first, then compare: Use sentences.txt to compare adjacent sentences
- Delete entire sentence: Fragments and repeated sentences should be deleted as a whole, not just the anomalous words
Segmented Analysis (loop):
1. Read readable.txt offset=N limit=300
2. Combine with sentences.txt to analyze these 300 lines
3. Append speech error indices to auto_selected.json
4. Record in speech_error_analysis.md
5. N += 300, go back to step 1
🚨 Critical Warning: Line Number ≠ Index
readable.txt format: idx|content|time
↑ Use this value
Line 1500 → "1568|[Sil1.02s]|..." ← idx is 1568, not 1500!
Format of speech_error_analysis.md:
## Segment N (Line range)
| idx | Time | Type | Content | Action |
|-----|------|------|---------|--------|
| 65-75 | 15.80-17.66 | Repeated sentence | "This is a case I edited" | Delete |
Steps 6-7: Review
cd ../3_Review
# 6. Generate review webpage
node "$SKILL_DIR/scripts/generate_review.js" ../1_Transcription/subtitles_words.json ../2_Analysis/auto_selected.json ../1_Transcription/audio.mp3
# Output: review.html
# 7. Start review server
node "$SKILL_DIR/scripts/review_server.js" 8899 "$VIDEO_PATH"
# Open http://localhost:8899
Users on the webpage can:
- Play video clips to confirm
- Check/uncheck deletion items
- Click「Execute Clip」
Data Format
subtitles_words.json
[
{"text": "G", "start": 0.12, "end": 0.2, "isGap": false},
{"text": "", "start": 6.78, "end": 7.48, "isGap": true}
]
auto_selected.json
[72, 85, 120] // Pre-selected indices generated by Claude analysis
Configuration
Volcengine API Key
cd /Users/chengfeng/Desktop/AIos/EditingAgent/.claude/skills
cp .env.example .env
# Edit .env to fill in VOLCENGINE_API_KEY=xxx