flmaximwang/AgentSkill-WatchingVideo

Watch a Video

End-to-end pipeline to "watch" a video: metadata → subtitles → download → keyframe extraction → vision analysis.

When to Use

• "看这个视频" / "总结这条视频" / "这个视频讲了什么"
• "帮我看看 BVxxx" / bilibili.com/video/BVxxx
• a YouTube link (youtube.com/watch, youtu.be, shorts)
• "看这个本地视频" / a path to a local .mp4/.mov/.mkv/.gif
• Any request to understand a video's content (not just its metadata)

Three Sources

Source	Metadata	Subtitles	Download
Bilibili (BV/URL)	bili video BVxxx --yaml	bili video BVxxx --subtitle	yt-dlp
YouTube (URL)	yt-dlp -J <url>	yt-dlp --write-auto-subs	yt-dlp
Local file	— (skip)	sibling .srt/.vtt if present	— (skip)

All three converge on the same keyframe step (Step 4) — everything from there is source-agnostic.

Prerequisites

• yt-dlp: download + YouTube metadata/subtitles — ~/.local/bin/yt-dlp
• ffmpeg: keyframe extraction — $HOME/.hermes/tools/ffmpeg-9.0.1-darwin-arm64
• Python: numpy + Pillow — supply them per-run with env -u PYTHONPATH uv run --with numpy --with Pillow python3 (no interpreter on this machine has them preinstalled; see the Pitfalls entry)
• Bilibili only: bili CLI (~/.local/bin/bili); bili status must show authenticated
• YouTube only: a JS runtime — yt-dlp --js-runtimes node (node is at ~/.local/bin/node; without it YouTube formats are incomplete)

Pipeline

Step 1 — Metadata

# Bilibili
bili video BVxxx --yaml                    # title / author / duration / description / stats

# YouTube
yt-dlp --js-runtimes node -J "https://www.youtube.com/watch?v=XXXX" \
  | jq '{title, uploader, duration, description, subtitles: (.subtitles|keys), auto: (.automatic_captions|keys)}'

# Local file
ffprobe -v error -show_entries format=duration,size -of default=nw=1 <file>   # optional

Step 2 — Subtitles (cheapest source; prefer over downloading frames)

# Bilibili — often unavailable; check `subtitle.available` in Step 1
bili video BVxxx --subtitle              # plain text
bili video BVxxx --subtitle-timeline     # with timestamps

# YouTube
yt-dlp --js-runtimes node --skip-download --write-auto-subs --sub-langs "en.*" \
  --sub-format vtt -o "/tmp/video-watch/v.%(ext)s" "<url>"     # → v.en.vtt

# Local file — read a sibling subtitle if one exists
ls /path/to/*.srt /path/to/*.vtt

If subtitles exist and answer the question, stop there. Download video only when the question needs 画面 (visual content) or subtitles are missing.

Step 3 — Download video

mkdir -p /tmp/video-watch
yt-dlp --js-runtimes node --no-playlist -o "/tmp/video-watch/video.mp4" "<url>"

• --no-playlist avoids downloading a whole season.
• Always output under /tmp/ — never pollute the workspace.
• If format selection fails, drop -f and let yt-dlp pick.

Step 4 — Extract keyframes

export PATH="$HOME/.hermes/tools/ffmpeg-9.0.1-darwin-arm64:$PATH"
env -u PYTHONPATH uv run --quiet --with numpy --with Pillow python3 scripts/extract_keyframes.py \
  /tmp/video-watch/video.mp4 \
  -o /tmp/video-watch/out \
  --cols 3 --sheet-max-cells 12 --sheet-width 1600 \
  --json

The three parameters matter (defaults target human viewing, too big for vision encoders):

• --cols 3 — at most 4 columns
• --sheet-max-cells 12 — at most 12–16 cells per sheet (script auto-splits into more sheets)
• --sheet-width 1600 — safe width (≤2400; >8000 is rejected by vision encoders)

Output:

/tmp/video-watch/out/
├── overview_01.jpg   ← contact sheets, chronological (read these first)
├── overview_02.jpg   ← more sheets when there are many keyframes
├── frames/kf_001_0.000s.jpg   ← individual frames (read for detail)
└── keyframes.json    ← width/height/duration/fps/keyframe_count; keyframes[].{n,time,path,diff}

Step 5 — Vision analysis

Read every overview sheet in order with vision_analyze:

vision_analyze(image_url="/tmp/video-watch/out/overview_01.jpg", question="描述这张关键帧总览图中的画面内容")

• Read all sheets — the script splits when keyframes are many.
• Text too small in a sheet → read the individual frames from frames/.
• keyframes.json gives time + path for locating any specific moment.

Keyframe Tool

scripts/extract_keyframes.py comes from ztough926/video-understanding (GPL-3.0).

Compares consecutive frames pixel-by-pixel and keeps only frames where the picture really changed; adaptive threshold = median diff × sensitivity, clamped to [0.008, 0.08]. Pure image algorithm, no model. Typical: 3-minute video (5000+ frames) → 50–100 keyframes → 4–8 sheets.

Quick Reference

Need	Command
Bilibili info	bili video BVxxx --yaml
Bilibili subtitles	bili video BVxxx --subtitle
Bilibili comments	bili video BVxxx --comments
YouTube info	yt-dlp --js-runtimes node -J <url> | jq '{title,uploader,duration}'
YouTube subtitles	yt-dlp --js-runtimes node --write-auto-subs --sub-langs "en.*" --sub-format vtt <url>
Download	yt-dlp --js-runtimes node --no-playlist -o /tmp/video-watch/video.mp4 <url>
Extract keyframes	env -u PYTHONPATH uv run --quiet --with numpy --with Pillow python3 scripts/extract_keyframes.py <file> -o <out> --cols 3 --sheet-max-cells 12 --sheet-width 1600 --json
Read keyframes	vision_analyze(image_url=...)

Pitfalls

• Always pass --cols 3 --sheet-max-cells 12 --sheet-width 1600. The default --sheet-width 9600 produces images vision encoders reject.
• Read ALL overview sheets, not just the first.
• Subtitle availability is rare on Bilibili (subtitle.available: false is the common case) — YouTube auto-captions are usually there, so try them first for YouTube.
• YouTube needs --js-runtimes node on this machine (deno is absent); without a JS runtime some formats go missing.
• Download to /tmp/ — never into the workspace.
• Interpreter / deps: python3 on this machine has no numpy/Pillow, and the ~/.hermes/tools/python-* build has none either. Use env -u PYTHONPATH uv run --with numpy --with Pillow python3 <script> — it resolves the deps per run, no venv to maintain. The env -u PYTHONPATH is not optional: the agent runtime exports a PYTHONPATH pointing at a python3.14 site-packages while uv's default interpreter is 3.11, so numpy's compiled extension fails to load with No module named 'numpy._core._multiarray_umath' (a message that blames numpy and sends you chasing a broken install that is fine).
• ffmpeg is not on the default PATH — prepend $HOME/.hermes/tools/ffmpeg-9.0.1-darwin-arm64.

Full Example

export PATH="$HOME/.local/bin:$HOME/.hermes/tools/ffmpeg-9.0.1-darwin-arm64:$PATH"

# --- Bilibili ---
bili video BV1bban6dE4C --yaml          # 1. metadata (no subtitle on this one)
mkdir -p /tmp/video-watch
yt-dlp --no-playlist -o "/tmp/video-watch/video.mp4" \
  "https://www.bilibili.com/video/BV1bban6dE4C"                      # 2. download
env -u PYTHONPATH uv run --quiet --with numpy --with Pillow python3 scripts/extract_keyframes.py \
  /tmp/video-watch/video.mp4 \
  -o /tmp/video-watch/out --cols 3 --sheet-max-cells 12 --sheet-width 1600 --json   # 3. keyframes
# 4. vision_analyze(image_url="/tmp/video-watch/out/overview_01.jpg", ...)

# --- YouTube ---
yt-dlp --js-runtimes node --write-auto-subs --sub-langs "en.*" --sub-format vtt \
  -o "/tmp/video-watch/yt.%(ext)s" "<youtube-url>"                    # subs first
# then the same download + keyframe + vision steps