RaganaBloodedge/read-video-skill

read-video — Read a video by fetching bytes on demand

What problem does this skill solve?

When a user shares a video link and wants to understand its content, the naive approach is to download the entire file — which is slow, consumes storage, carries copyright risks, and often only needs a few seconds. This skill works by fetching only the small byte range around the target moment. The audio track is transcribed offline locally (the primary method for answering "what is this about"), and the video track extracts keyframes to pass to a vision model for visual context.

Measured benchmark (249 MB / 9 min 56 sec MP4, not faststart):

Action	Upstream bytes fetched	Percentage of total
Grab frame at 30s	2.0 MB	0.839%
Grab frame at 8 min	1.7 MB	0.707%
Grab 20s audio starting at 480s	6.6 MB	2.784%

When to use / When not to use

Use: The user provides a direct video link, or a web page containing a video (Bilibili and other platforms are automatically resolved by resolve.py — no need for manual API calls).

Do NOT use: Local files (just read them directly), long videos that require frame-by-frame understanding (still costly; ask the user for specific timestamps first).

Hard rules

1. Do not download the entire file. Never use wget/curl -O for the whole file in any step.
2. Probe before acting. Seeking without probing will waste time on servers that don't support Range.
3. -ss must come before -i. Placing it after -i is output-side seeking, which decodes sequentially to the target point — effectively downloading the whole file. Putting it before -i is input-side seeking, where the demuxer jumps directly.
4. Keep measurement on. --measure is enabled by default: it counts upstream bytes via a local streaming proxy, providing the only hard evidence that bytes are truly fetched on demand, and proof that no full file was downloaded.
5. Pass web page URLs to scripts for resolution. A page URL is not a direct link, and direct links with signatures expire. Use resolve.py, or pass the link directly to grab.py/transcribe.py (they will parse it automatically and add the required headers). Do not write site-specific API calls in the conversation — they are never reviewed and always differ.

Workflow

Step 0: Identify input — page URL or direct link

• Page URL (Bilibili, YouTube, Douyin, Weibo, Vimeo… 1700+ sites) → resolve to a direct link. You can also pass it directly to grab.py / transcribe.py — they recognize it, parse it, and automatically include the necessary request headers.
• Direct link (.mp4 / .m3u8 and other media extensions) → skip resolution.

The decision logic in resolvers.is_page_url(): anything with a media extension is treated as a direct link; all other HTTP links go to the resolution layer.

Step 1: Obtain a stream

python scripts/resolve.py "<page URL>"

Output includes video_url (for grabbing frames), audio_url (not the same stream, for transcription), headers (CDN-required request headers), plus title, duration, and a set of recommended sampling timestamps.

The Platform line indicates which backend was used: native (hand-written adapter, currently only Bilibili) or yt-dlp (generic, 1700+ sites). To force a specific backend, use --backend native|yt-dlp|auto.

One command to do it all (resolution and data fetching seamlessly connected — recommended):

python scripts/transcribe.py --url "<page URL>" --seconds 90
python scripts/grab.py "<page URL>" --times 14 29 44 58 73 --prefix shot --outdir ./out

Step 2: Probe (recommended for direct links, can be skipped for page URLs)

python scripts/probe.py <direct-link> --json

If the direct link comes from a CDN, include the source: --header 'Referer: https://www.bilibili.com'.

Look at three fields:

• seekable — if false, stop. The server ignores Range; seeking becomes sequential streaming. Fall back to "decode as it arrives" (-i <url> without -ss, grabbing the earliest frame) or tell the user it's not possible.
• container.moov_offset / faststart — If the index is at the head (faststart), the fixed cost is tens of KB. If at the tail, you must fetch the last part first (~441 KB in testing). This is the fixed cost for "grabbing one frame" and cannot be avoided.
• content_length — denominator for calculating "percentage of total file."

Step 3: Fetch data based on user intent

User provides a specific timestamp → pinpoint extraction:

python scripts/grab.py <url> --time 183.5 --out shot.jpg

User did not provide a timestamp → sample a few frames first:

python scripts/grab.py <url> --times 5 60 180 420 --prefix shot --outdir /tmp/rv

Sampling tips: Take 4–6 frames based on content density, not evenly spread across the entire video (that would decode the whole thing). If uncertain, grab 3 frames first, review them, then decide if a specific segment needs deeper analysis.

To understand what is being said → transcribe first (recommended first step):

python scripts/transcribe.py --url <url> --segments 0:90 320:90 630:97

This single command cuts audio → performs local recognition, outputting timestamped Chinese text. No external API, no cloud inference needed. The first run automatically downloads the SenseVoice model (~240 MB from hf-mirror, one time only).

For long videos, transcribe 3–4 segments of ~90 seconds each to cover the main narrative — no need to transcribe the entire file. This is much faster than frame-by-frame analysis and better suited for answering "what is this about." Transcribe first, then supplement with frames for visual detail — that's the order.

After transcription, check the yield rate. Normal speaking rate is ~4–6 characters/second. The script output includes chars_per_second. If below 1 char/sec, the script will actively prompt. In such cases (music, gameplay, silent demos), do not use the sparse text as content. Switch to denser frame sampling (8–10 frames) for visual analysis — the audio carries no information, only the visuals do.

If you only need the audio file (e.g., to send elsewhere for transcription):

python scripts/grab.py <url> --start 0 --seconds 60 --audio clip.wav

If the video has embedded subtitle tracks, prefer reading subtitles (ffmpeg -i <url> to check for subtitle streams) — they are more accurate and faster than ASR.

Step 4: Pass frames to a vision model

grab.py only fetches pixels locally; it does not understand content. Pass the output jpg path to a vision model with your question ("What is in this frame?" "What is happening here?"). For multiple frames, provide them in chronological order to allow the model to construct a narrative.

How to read the output

grab.py outputs JSON, with each action including a byte accounting:

{
  "ok": true,
  "actions": [{
    "kind": "frame", "time": 30,
    "out_bytes": 48211,
    "upstream_bytes": 2097152,
    "requests": 4,
    "wall_seconds": 2.317
  }],
  "total_upstream_bytes": 2097152
}

• upstream_bytes / content_length gives the "percentage of total file." If it exceeds 5%, stop and review: likely -ss is in the wrong position, or the server doesn't support Range.
• Multiple requests are normal: ffmpeg fetches in separate chunks (moov, from keyframe to target frame, audio headers).
• If ok: false, check err_tail. Common issues are timeouts and 5XX errors — the script has built-in retries; if still failing, try a different timestamp.

transcribe.py output:

{
  "ok": true,
  "text": "This is a land surrounded by great rivers and mountains…",
  "segments": [{"start": 0, "end": 25, "text": "…"}],
  "audio_seconds": 276.7,
  "model": "SenseVoiceSmall-int8"
}

text is the concatenation of all segments; segments include timestamps for queries like "what was said at second X."

Environment

• Interpreter: Python in a local managed venv. On Windows, the path looks like ~/.workbuddy/binaries/python/envs/default/Scripts/python.exe. Note: it's Scripts/, not the Unix-style bin/ (writing bin/python will result in a "file not found" error).
• Dependencies: httpx, imageio-ffmpeg (data fetching), sherpa-onnx, numpy (transcription), yt-dlp (generic platform resolution). All are installed in the same venv — if any are missing, the script will clearly report the error. Without yt-dlp, Bilibili still works (via the native adapter), but other sites will fail to resolve.
• ffmpeg: Provided by imageio-ffmpeg as a static build — no system installation required. To specify a different ffmpeg, use the environment variable RV_FFMPEG or --ffmpeg.
• ASR model cache: ~/.workbuddy/cache/read-video/models (~240 MB, downloaded automatically on first run). If the file exists and the byte count matches, download is skipped.

Known limitations

• CDN origin validation: Video CDNs for Bilibili and similar sites require a Referer header; without it, you get a 403 error. Using resolve.py or --page automatically adds it; when passing a direct link manually, remember --header 'Referer: https://www.bilibili.com' (supported by all three scripts, can be repeated).
• Direct links expire: Site-issued direct links have signatures and deadlines — use them within the same batch of operations. Once you get a 403, re-run resolve.py.
• Platform coverage = yt-dlp coverage (1700+ sites), not maintained separately. If a site fails to resolve, first use --backend yt-dlp to see the raw error. Sites requiring login/Cookies will fail — in those cases, either give up or provide Cookies manually. yt-dlp itself needs regular upgrades (old versions fail after site updates) — run pip install -U yt-dlp.
• "Resolved" ≠ "reachable": The resolution layer only provides the URL. When data fetching fails, first determine whether it's a resolution failure or a network connectivity issue — in testing, the local machine could not reach YouTube/Vimeo (proxy 502), but Douyin/Weibo/Xigua Video were reachable.
• HLS / m3u8: Not a single file; the box dissection in probe.py will fail (with an error field). In this case, grab.py still works — ffmpeg will fetch the required segments by itself. content_length will be empty, so the percentage cannot be calculated.
• DASH audio/video separation: Many sites (including Bilibili) serve video and audio as two separate URLs. Use the video stream for frame grabbing and the audio stream for transcription — using the wrong one will silently return empty results.
• -ss precision is limited by GOP: To grab a frame at a specific second, you must decode from the nearest keyframe. Therefore, the smallest fetch unit is "keyframe to target frame," not "a single byte range."
• ASR homophone errors: SenseVoiceSmall works well for Chinese, but proper nouns (names, places, ancient terms) often have homophone transcription errors (e.g., "阪泉之野" → "版圈之也", "涿鹿" → "涿陆"). Correct proper nouns against context before citing them — do not treat the transcription text as factual.
• Dependencies: httpx + imageio-ffmpeg (data fetching) + sherpa-onnx + numpy (transcription). If any are missing, the script will clearly report the error and suggest which to install.

For more detailed principles, troubleshooting notes, and benchmark data, see references/methodology.md. For the relationship between this skill and the dsh plugin dsh-readvideo, see references/dsh-dual-form.md.