video-sync
Combine shots.json + original video into a single video: the video on one side, the current shot's analysis info on the other. The information updates with every cut, and the shot list auto-scrolls and highlights the current shot.
The layout depends only on the aspect ratio:
| Original Video | Layout | Composition Method |
|---|---|---|
| Landscape (width > height), Square | Video on top, info below | vstack |
| Portrait (height > width) | Video on the left, info on the right | hstack |
Both orientations ensure the video is scaled as-is, without cropping or stretching; the panel fills the remaining space.
Only three screenshots are taken; everything dynamic is handled by ffmpeg. The only things that change on the panel are: scroll position and which row is highlighted. So we measure row positions once, take one static base screenshot, and two long tiled images of the full shot list (one dimmed, one fully lit). Scrolling = cropping a viewport from the dim long image. Highlighting = cropping a single row from the lit long image and overlaying it. Positions are driven by sendcmd expressions at each cut point. For a 53-shot video, screenshots are reduced from 53 to 3 (takes 12 seconds).
The current shot is pinned to the second row: shots 1 and 2 do not scroll (the highlight moves down one row); starting from shot 3, the list scrolls up one row per cut. Motion happens at cut points, pauses within shots — at a cut, it scrolls with --ease (default 0.45 seconds) and stops. For a 16-second long shot, the list stays still (no eye fatigue).
Row height is determined by content — longer descriptions make taller rows. The highlight bar is layered per row height (crop height cannot be changed at runtime).
{baseDir} = the directory containing this file. The script is {baseDir}/scripts/video-sync.mjs, zero dependencies, run directly with node.
Boundaries (what this skill does NOT do): It does not perform shot detection (that's video-shots's job; this skill consumes its shots.json), does not edit, transition, or add music, does not burn subtitles onto the video, and does not do frame-by-frame rendering (motion effects are achieved via ffmpeg time-based cropping).
Step 0 — Obtain shot analysis data first
You need a shots.json (output of video-shots) and the corresponding original video. Having a frames/ directory (with one keyframe per shot) is better — it will be used as thumbnails in the shot table. If absent, the thumbnail slot will be empty.
shots.json must match this video exactly: if meta.durationSeconds does not match the original video's duration, do not composite; go back and correct the shot analysis.
Step 1 — Check geometry first, then process the video
node {baseDir}/scripts/video-sync.mjs plan shots.json --video <video>
This only calculates geometry without compositing — takes seconds. It outputs: layout type, video area, panel area, output dimensions. Confirm these three numbers are reasonable before proceeding — compositing a 3-minute video takes 1–2 minutes; it's not worth redoing if the numbers are wrong.
Adjust if needed: --panel <ratio> (for landscape: panel height ÷ video height, default 0.8; for portrait: panel width ÷ video width, default 1.6), --width / --height (maximum bounds for the video area, default 1920 / 1080).
If the original video is too small, scale up the video area: the panel is calculated relative to the video area. A 640×360 video gives a panel of only 640×288 — four columns crammed together, unreadable. Use --scale 2 to scale the video area to 1280×720 and the panel to 1280×576, preserving aspect ratio. Default is 1 (only downscale, never upscale). plan / panels / compose / export all accept the same --scale.
Step 2 — Render the panel
node {baseDir}/scripts/video-sync.mjs panels shots.json --video <video> \
--frames frames --lang zh|en [--out panels] [--chrome <path>]
Produces: static.png (base panel), list-dim.png / list-lit.png (full-length tiled images of the shot list — one dimmed, one fully lit), layout.json (row positions, used during composition for scrolling), and a preview file panels/panel.html — open it in a browser to preview the panel. Change shots via #S07 to see effects (the scrolling animation in the preview matches the export).
To change the layout, edit {baseDir}/scripts/panel.css and rerun panels. No need to modify the script or rethink the screenshot logic. Style conventions are documented in {baseDir}/references/panel-style.md.
Step 3 — Composite
node {baseDir}/scripts/video-sync.mjs compose shots.json --video <video> \
[--panels panels] [-o out.mp4] [--crf 20] [--ease 0.45]
The audio track is copied directly from the original video (silent videos automatically get no audio track). After compositing, the output dimensions, duration, and layout type are printed to stderr — double-check them.
You can also run both steps in one command:
node {baseDir}/scripts/video-sync.mjs export shots.json --video <video> -o out.mp4 --frames frames
Step 4 — Sample frame verification ⛔ Do not skip
Do not deliver just because compositing succeeded. Extract at least three frames to verify the information matches the video:
for t in 5 30 60; do ffmpeg -v error -y -ss $t -i out.mp4 -frames:v 1 -q:v 3 /tmp/check-$t.jpg; done
Check three things: Which shot does this frame belong to? Is the shot number in the panel the same? Does the highlight row follow correctly? If they don't match, usually the shots.json does not correspond to the same video, or the --video parameter points to the wrong file.
Report in one sentence: output path, dimensions, duration, layout type, number of shots, and which panel images were not rendered (if any).
Boundaries
- Scrolling only occurs at cut points: scrolls with
--easeseconds, then stops. The panel is static between cuts. To make it scroll continuously (e.g., for a constant crawl), modify themotionPlansegments — but long shots will cause eye strain. - Requires a headless browser (Chrome / Chromium / Edge). This is the price for "layout can be freely customized": using ffmpeg's
drawtextfor the panel would mean implementing Chinese line breaks, thumbnails, and highlighting from scratch — changing the layout would require rewriting filter graphs. - The vocabulary (shot scale / type / camera movement / transition) is bundled within this skill — it does not import across directories, so the skill can be fully copied. If
video-shotsadds new terms, they must be added here too; otherwise they will appear as enumeration keys. - Panel rendering always uses exactly three screenshots, independent of shot count: ~12 seconds for 53 shots. Compositing time scales with video duration: ~75 seconds for a 203-second video.
Self-test
node {baseDir}/scripts/selftest.mjs
122 assertions — no ffmpeg, no browser: geometry (landscape/square/portrait, even-side dimensions, upper/lower bounds, sliders), panel page data contracts (vocabulary delivery, rhythm roles, thumbnail presence/absence, escaping), animation commands (each must be short — if an expression is too long, ffmpeg fails configuration), ffmpeg parameters. Run this after modifying any script.