zenstory-ai/video-reference

1. Positioning

This skill is used to deconstruct an already-finished high-quality narrative video on demand, extracting its editing rhythm, sound design division, and storytelling techniques into method cards and quantitative metrics that can be directly referenced in the next production. It runs entirely locally, outputting only measurement data, deconstruction annotations, and a sanitized reference file. It does not call MiMo or score/block the daily production pipeline.

The Agent acts as a deconstruction editor: first faithfully recording "how exactly this finished video was edited and mixed" (source facts), then distilling from it "how a new piece of material can still learn from this" (transferable methods). Facts and methods must be completely separated: facts containing original character names, plot details, and specific dialogue remain only in the local deconstruction directory; the final exported production_reference.json retains only generic methods and actual measured rhythm values.

2. Input

• The finished video file to deconstruct (supports .mp4 / .mov / .mkv / .webm).
• The video understanding output directory U for this specific video (recommended to set ASR_SEGMENT_SECONDS=5 during the understanding phase; the finer the window, the more accurate the narration speech rate estimation). This skill will optionally read asr_result.json, asr_timing_evidence.json, and background_research.json (the character names, aliases, and ASR dialogue registered in these files will all be included in spoiler-leak scanning). When making manual annotations, the Agent should also consult the storyboard collage (storyboard) and vlm_analysis.json.

Even without running video understanding, this skill can still operate independently, but the exported narration speech rate field will be empty, the anti-leak scan vocabulary will only rely on the Agent's hand-written entities, and check will issue corresponding warnings. In this case, use frames --span 0,<duration> --step 2 to extract frames for a collage to assist with video review.

All scripts/... paths below are relative to this skill's directory.

3. Deconstruction Workflow

python3 scripts/reference.py measure <finished_video> --work-dir U        # Run ffmpeg to measure cut points and loudness, cached by file identity
python3 scripts/reference.py frames  <finished_video> --work-dir U --review          # Export frame grids for manual review of suspected cut points
python3 scripts/reference.py frames  <finished_video> --work-dir U --longest 5       # Extract 12 frames evenly from the 5 longest shots to check for missed cut points
python3 scripts/reference.py check   --work-dir U [--json]      # Validate the deconstruction draft and print derived metrics
python3 scripts/reference.py export  --work-dir U --out <next_production_work_dir>/production_reference.json

1. Measure Shots and Loudness (measure): The script generates U/reference_measurements.json (do not manually edit), recording full-video shot cut points, shot duration distribution, cuts per minute, a 10-second sliding window cut density curve, integrated LUFS loudness, loudness range (LRA), true peak, and per-second short-term loudness. This step requires fully decoding the entire video (a 5-minute 720p video takes about 15–70 seconds; a 4K video takes about 7 minutes); as long as the video file has not changed, the cache will be reused directly. Adjusting --soft-score or --hard-score thresholds alone does not require re-decoding.
   • Initial Cut Point Screening Logic: Frames with a scdet scene change score ≥10 are directly classified as cut points. For scores between 4 and 10, a frame is classified as a cut point only if its score is more than twice the highest score of other non-adjacent frames in a ±0.3 second window (i.e., a clear local peak). This filters out most camera shake, fast zooms, and flash frames, placing these uncertain frames into the review window shots.review_windows. Additionally, because the scdet score subtracts the amount of change in the previous frame, when transitioning from a high-motion shot abruptly to a static scene, the scdet score may be suppressed to only about 1 point; therefore, the tool also collects peaks in the inter-frame difference (mafd) into shots.review_windows for manual confirmation, but does not directly count them as automatic cut points.
2. Review Cut Points Manually (frames): Any fixed threshold will inevitably miss some true cuts in real videos (a hard cut in a dim night scene may only score 5–7, while a long shot in a fierce fight might produce 7–8 point peaks every 0.2 seconds). Therefore, visual manual review is essential. Running frames --review creates a frame grid for each review window; running frames --longest 5 evenly extracts 12 frames from the 5 longest shots in the entire video. Images are output to U/reference_frames/, and the terminal prints the time in seconds for each frame. After reviewing, register corrections in labels.cut_fixes: add for true cut points missed by the algorithm (in seconds), remove for false cut points mistakenly detected by the algorithm (matched within ±0.1 seconds of the detected cut point). If all cut points are confirmed accurate with no additions or deletions needed, explicitly write "cut_fixes": {} to indicate the review is complete. Note: if measure is re-run causing the review windows to change, the old cut_fixes are no longer valid, and the Agent must cross-check and update them against the new windows. After review, all subsequent shot statistics, section cut densities, and final exports will be recalculated using the corrected cut points.
3. Annotate Audio and Sections (labels): The Agent fills in labels in U/reference_breakdown.json, including audio track assignments audio_spans, narrative function sections sections, subtitle formatting subtitles, and annotation basis basis.
   • Audio boundaries must align with the actual start/end of sound: The switching points in audio_spans must fall at the moments when the ear actually hears the speaker begin or end speaking, not the appearance or disappearance time of the subtitle line (annotation based on subtitle cut-in frames often results in narration end points being 0.24–0.42 seconds later than the true on-air end). If U/speech_boundary_anchors.json already exists in the directory, directly align boundaries with its acoustic_pauses (start = end of previous utterance, end = start of next utterance); because the switch_on_cut_share rate (audio-visual cut alignment) only tolerates ±0.25 seconds of error around the visual cut point, misaligned boundaries will severely distort the calculated handover rate. Voice-over or internal monologue by original characters still belongs to the original audio track and is uniformly labeled as original_dialogue; if a distinct treatment is needed, write it into source_facts and methods.
   • After filling in labels, run check first to view the derived statistics derived printed in the terminal (duration share of each audio track, cut density per section, narration speech rate, proportion of sound switches that fall on hard visual cuts, relative time-axis position of each narrative function section structure, and the first appearance time of the original sound first_original_at).
4. Extract Facts and Transferable Methods (source_facts and methods): Based on the objective derived data just calculated, in the same reference_breakdown.json, continue to fill in source facts source_facts (must be anchored to a specific time interval or measurement path, and explicitly list the proper names involved entities) and transferable methods methods (including the method rule rule, applicable scenarios applies_when, conditions to avoid avoid_when, the downstream files it applies to applies_to, supporting evidence evidence, and quantitative reference targets targets). Note: targets are only allowed to contain reference paths {"from": "<measurement_path>"}; it is strictly forbidden to write specific numbers in the rule text. When exporting, the script will automatically fill in the actual measured values.
5. Validate and Export (check → export): Run check to confirm zero errors, then execute export to generate production_reference.json; before writing to disk, the exporter performs another full-field anti-spoiler leak scan on the output.

For complete field definitions, enumeration lists, and good/bad method writing examples, see references/reference-schema.md.

4. Validation and Anti-Leak Rules

reference.py check will block with exit code 1 if any of the following errors occur:

Rule ID	Validation Requirement
R1	Top-level, labels, fact, method, target, and cut_fixes must use strictly closed key sets and closed enumerations; fact must not mix method fields, method must not mix fact fields; IDs must follow f1… / m1… and be globally unique; subtitles field type is fixed; reasons for skipped_dimensions must be non-empty strings
R2	audio_spans and sections must be ordered by time ascending, non-overlapping, adjacent gaps ≤0.5 seconds, and fully cover the entire video duration; subtitle evidence timestamps must fall within the video duration; cut_fixes.remove must match a detected cut point, add must not duplicate existing cut points
R3	Each fact must choose between a time anchor t: [a, b] (within the video duration) and a measurement anchor measure: [path] (resolving to a non-string scalar in shots / loudness / derived), and must explicitly fill the entities array
R4	Each method must bind at least one piece of supporting evidence: reference to a defined fact ID, or measure:<path> (also limited to the three root paths above)
R5	target is only allowed to contain the from key, pointing to a numeric leaf node or whitelisted derived object under shots, loudness, or derived; list subscripts are strictly forbidden
R6	In rule, applies_when, avoid_when, and reasons for skipped_dimensions, the following are strictly forbidden: proper names from the original video (ignoring spaces), any 8 consecutive Hanzi characters that exactly match ASR dialogue/background materials/source_facts (spanning punctuation or adjacent ASR windows also blocked) or 5 consecutive English words, absolute time codes, "Nth second" or "N minutes M seconds" (including Chinese numeral forms), and local absolute paths
R7	Each of the five production dimensions (narrative_structure, pacing, shots_editing, narration_subtitles, audio_visual) must have at least one extracted method; if a dimension truly cannot be extracted, a reason must be given in skipped_dimensions
R8	When exporting production_reference.json, run the R6 leak scan again on all key names and string values, and the exported object must not contain the keys source_facts, labels, entities, evidence, statement, from, or path at any level

The anti-leak vocabulary is automatically aggregated from source_facts[].entities, character names and aliases in background_research.json, cultural_notes entries of no more than 5 characters, and short terms enclosed in 《》 or 「」 in the materials. If a conflict arises from an 8-character overlap with source_facts, simply rephrase the generic editing description on one side.

The following only trigger warnings (not blocking export): method missing applies_when, rule containing Arabic numerals directly, ASR precision not being AVAILABLE_COARSE, a video with Chinese dialogue lacking character names in both understanding outputs and materials (relying solely on hand-written entities), empty basis, a mismatch between the file identity recorded in the understanding output and the current measured video, and review windows existing without filled cut_fixes.

Note: R6 can only block literal string overlap; it cannot catch paraphrasing that still recounts the original plot. When writing methods, summarize "what to do when encountering a specific material feature"—never write "how the protagonist entered the door." Additionally, the validator does not check whether the natural language semantics match the data—before writing "which type of section has denser cuts," first cross-check the measured cuts_per_min printed by check; do not write conclusions that contradict the measured data based on memory.

5. Deliverable and Subsequent Use

The final exported production_reference.json contains only: total video duration and canvas size, cut_detection (cut detection parameters and number of manually added/removed cut points), profile (core pacing and loudness metrics with provenance markers: measured = pure algorithm measurement, reviewed = after manual cut_fixes, labeled = depends on manual audio/section annotation), the relative position distribution of each narrative section on the timeline structure, subtitle formatting features subtitles, a list of transferable methods methods, and explanations for skipped dimensions skipped_dimensions.

How to use: Place the exported production_reference.json into the work_dir of the next production task (or bind it via the production_reference template in the resource library). Downstream scriptwriting stages will recognize the file's existence and actively read it, recording the adoption decision (adopt / adapt / skip) for each method in the optional field reference_methods of recap_story_plan.json. When referencing methods, always base decisions on the actual visuals and dialogue evidence of the new material; never force the cut density or blank space proportion from the reference video as rigid quotas. No production script mechanically validates this file, and its presence or absence does not change any gate access.

The U/ directory's reference_breakdown.json and reference_frames/ extracted frame grids contain original video plot facts and visuals. They are for local deconstruction use only and must not be copied into new projects or submitted to public repositories.

6. Capability Boundaries

• Entirely local computation; does not call MiMo or any large model, does not modify upstream video understanding outputs, and does not score the quality of the finished video or calculate "differences between new and reference videos."
• Shot cut points are based on ffmpeg scdet scores plus manual review of frame grids: slow dissolve transitions cannot be automatically detected; two consecutive fast cuts less than 0.3 seconds apart, cut points exactly on the soft_score boundary (different ffmpeg versions may produce 4.00 vs 3.989), and hard cuts into dark scenes after intense motion (score may drop to 0.9) are not directly counted as automatic cut points—they are all collected into review windows for manual inspection. In a test on a 5-minute episode commentary video, the algorithm initially detected 82 cut points and captured all 5 missed true cut points within 11 review windows (the remaining 7 windows contained fast zooms, page turns, and motion shots); after the agent manually added these 5 cut points via frame review, the result was 100% consistent with the frame-by-frame ground truth (27 cut points total distributed across the opening, dark scenes, and middle section), with zero false positives.
• Narration proportion and speech rate are labeled-level precision: estimated from manually annotated audio intervals combined with coarse-grained ASR windows, not aligned with per-character timestamps.
• The anti-spoiler leak scan is based on literal string matching; the Agent must still actively self-check when writing to ensure methods describe general creative techniques, not the original video's plot summary.