zenstory-ai/video-recap

1. Positioning & Workflow

This skill is a self-contained video commentary and editing production skill that converges video understanding, story intent, scene editing, voiceover writing, dubbing synthesis, and audio-video mixing into a single coherent workflow. All stages share the same working directory work_dir for JSON and MP4 outputs:

understand.py (video understanding) ─▶ Agent creates story/audio plan and writes clip_plan.json or narration.json ─▶ [cut.py (editing and re-output summary)] ─▶ validate.py (local baseline validation) ─▶ voiceover.py (TTS dubbing) ─▶ assemble.py (mixing and subtitle rendering)

For daily use, directly call the main entry point scripts/recap.py to automatically chain the full process and pause at creation nodes. You can also call the single-stage entry points under scripts/ (understand.py, cut.py, validate.py, voiceover.py, assemble.py) as needed.

Deconstructing an existing finished video into a production reference (production_reference.json) is not on the default production chain. When a user requests deconstruction for learning: first run understand.py on the sample (recommend setting ASR_SEGMENT_SECONDS=5), then use a separate deconstruction skill to execute measurement (measure), annotation, validation (check), and export (export). Copy the exported production_reference.json into the work_dir of the next run, or register it as a production_reference template in the resource library and bind it via --project. The main flow will not automatically run deconstruction, nor will it mechanically compare new finished videos against reference ones.

The entire flow supports resume after interruption: after the Agent writes clip_plan.json or narration.json, simply re-run the same recap.py command to continue. On resume, the script will check the source video path, file size, modification time, and runtime parameters recorded in recap_run_manifest.json. If the work directory comes from a different video or parameter combination, it will directly refuse reuse; video understanding cache will only be hit when the source and configuration are exactly identical. The resume command printed at each pause retains the original call style, automatically converts video and path parameters to absolute paths, supplements --work-dir, and can be copied and pasted to any directory to run directly.

The visual flow --edit-mode full|cut and the audio strategy --audio-mode narration|source-mix are independent: narration follows the standard commentary dubbing chain, while source-mix does not generate dubbing but directly mixes original audio with optional background music. The system only supports the five legal routes listed in the table below; all other combinations will be rejected with an error at startup:

Input	--edit-mode	--audio-mode	Agent Pause Points	Flow	See
Single video	full	narration	1 time: write narration.json	Understand → Write → Validate → Dub → Mix	§4
Single video	cut	narration	2 times: first write clip_plan.json, after cutting write narration.json for the finished video	Understand → Cut → Re-output timing summary → Write → Validate → Dub → Mix	§4
Multiple videos	cut	narration	2 times: same as above, each segment must have source_id	Per-source understanding → Cut → Write → Validate → Dub → Mix	§4.5
Single video	full	source-mix	No pause	Directly mix the current full video (no understanding or TTS)	§4.7
Single/Multiple videos	cut	source-mix	1 time: write clip_plan.json	Understand → Cut → Mix (no voiceover writing)	§4.7

2. Creative Responsibilities

The creative decision-making power belongs entirely to the Agent. Before starting, first determine the creative control mode for this round (CREATE open creation / DIRECTED directed creation / REVISION revision for finalization, independent of --edit-mode). As one cohesive creative brain, complete five judgments before entering dubbing and rendering:

1. Director's Judgment: Clearly define what the audience clicks to watch, whose point of view (POV) to follow throughout, what the core suspense is, where the emotional arc starts and ends, and which card is held until the very end.
2. Story Editing: Each beat must convey "what substantial turn the situation has taken." Never write a running list of scene summaries.
3. Visual Editing: Select the most striking actions, reactions, and the best entry and exit points. Cut out any dragging transitions.
4. Sound/Voiceover: First decide for each beat whether the scene, original dialogue, action sounds, silence, or voiceover takes the lead. Then write the script following the principle of "pad half a sentence, yield half a sentence, follow up with half a sentence."
5. Audience Review: From the perspective of a first-time viewer, separately check the visuals without voiceover, listen to the audio with eyes closed, and verify the opening and closing promises.

The division of labor for the three control modes, the locking and non-rollback rules under REVISION, specific editing and writing techniques, and the format specifications for recap_story_plan.json, visual_audio_board.json, and style_card.json are detailed in references/creative-editing-playbook.md. These JSON files are used to solidify reviewable creative decisions, ensuring that shot selection, original sound pauses, and commentary lines all adhere to the same story intent from start to finish.

3. Environment & Remote Call Notes

# ffmpeg: brew install ffmpeg | apt install ffmpeg | choco install ffmpeg
export MIMO_API_KEY=***

The same MIMO_API_KEY can drive the entire default chain. To ensure transparent data flow, the remote network call stages involved in this skill are explained below:

• Video Understanding (understand.py): Sends segmented audio windows to MiMo ASR (mimo-v2.5-asr) and sends extracted JPEG frames along with the background_research.json summary to MiMo VLM (mimo-v2.5). If the work directory already contains complete analysis outputs, passing --brief-only or --edited-storyboard-only results in purely local execution with no network requests.
• Voiceover Dubbing (voiceover.py): Sends the commentary text to the corresponding TTS service according to --tts-provider (default auto prioritizes configured MiMo, then Fish Audio, never automatically selects index-tts):
  • mimo-tts: Sends to MiMo TTS (mimo-v2.5-tts); if --voice-ref <audio> is explicitly passed, the reference audio segment is also sent to mimo-v2.5-tts-voiceclone (ensure voice authorization is obtained before use).
  • fish-audio: Requires FISH_API_KEY. Sends each commentary text segment and FISH_TTS_REFERENCE_ID to FISH_TTS_API_URL (default https://api.fish.audio/v1/tts).
  • index-tts: Only enabled when explicitly specifying --tts-provider index-tts. Sends the text and INDEX_TTS_VOICE to the user's self-hosted INDEX_TTS_ENDPOINT (see references/index-tts.md for details).
• Commentary Validation (validate.py), Video Editing (cut.py, shot_review.py), and Audio-Video Assembly (assemble.py): Purely local deterministic logic and ffmpeg/ffprobe calls. No network requests are initiated.

Token Plan keys starting with tp- are sent to the China cluster by default. You can switch clusters via MIMO_TOKEN_PLAN_CLUSTER. For the full list of configuration items, see references/config-playbook.md; for the format of all intermediate outputs, see references/data-schema.md. scripts/... below are relative to this skill's directory.

4. The Seamless Creative and Editing Workflow

4.1 Background Research and Video Understanding

As long as the film name, series name, or science topic can be identified from the file name or user instructions, first research the character relationships and background information according to references/research-guide.md before starting. Write the findings into work_dir/background_research.json. This way, subsequent video analysis can directly use correct character names and understand subtext, avoiding generic descriptions like "man in black" or "young woman." For multi-video concatenation, this is also written under the project root work_dir, and the script will automatically sync to each source's subdirectory sources/<source_id>/.

Then start the understanding and orchestration entry point:

python3 scripts/recap.py <video> --work-dir <work_dir> --context "background"

After the script completes audio extraction, silence and end-of-sentence pause anchor detection (speech_boundary_anchors.json), scene segmentation, VLM frame analysis, and ASR transcription, it generates a storyboard montage storyboard/storyboard_*.jpg and a creative brief agent_narration_brief.md, then automatically pauses.

4.2 Director's Intent and Shot Selection Plan (cut Phase 1)

After thoroughly reading agent_narration_brief.md and directly viewing the extracted storyboard frames, first write recap_story_plan.json (listing at least 2 candidate plans, selecting the main POV, opening hook, emotional arc, and silent reserve card) and visual_audio_board.json (clarifying for each beat which moment to choose, why to enter/exit, and who leads the sound).

In the first phase of cut mode, then write the work_dir/clip_plan.json based on the original source timeline (SOURCE) (do not write narration.json yet):

{
  "target_duration": "3m",
  "clips": [
    {
      "start": 12.0,
      "end": 38.0,
      "reason": "b01 | hook | knowledge: unknown→threat | POV=Shen Man | Catches fiancé embracing heiress, discussing marriage | entry=push door open, see both figures | exit=turn around, grip household registry tightly"
    }
  ],
  "required_evidence": {
    "nodes": [
      {"id": "e1", "source": "/abs/source.mp4", "start": 14.2, "end": 18.6, "track": "video", "content": "Pushes door open, catches them embracing"}
    ],
    "before": []
  }
}

Shot selection and cut points must adhere to the following baseline rules:

• reason Seven-Part Structure: Write sequentially: beat_id | narrative function | state change (A→B) | POV=character | substantive highlight of this segment | entry=specific action/audio anchor | exit=specific reaction/landing point. This ensures every segment serves the story's progression.
• Mandatory Visual and Audio Evidence (required_evidence): Must retain high-impact actions, reaction moments, or key original sound Q&A. Write these into the top-level required_evidence (register nodes with id/source absolute path/start/end/track/content, and before for non-reversible order pairs); if boundary adhesion cuts away any mandatory interval or disrupts the sequence, the script will directly report an error and block.
• Late Entry and Shot Selection: By default, follow a chronological "late entry" (cut directly into conflict outbreak, absurd contrast, or key action; cut out slow-paced warm-ups like entering a room or exchanging pleasantries). Only allow a cold opening suspense shot of <= 2.5s if the chronological conflict occurs too late, and it must be understandable without setup, must not repeat the same camera angle from the main video, and must only leave suspense without spoiling the outcome. When the voiceover lowers the original sound, prioritize reaction shots of the listener, action details, or stable shots; avoid placing the face of a character who is talking incessantly but silenced under the voiceover. Do not repeat the same nodding or blank staring reaction shot more than twice.
• End-of-Sentence Safe Cut Points and Breathing Tail: Cut points must fall at the original video's start/end, end-of-sentence pause anchors, or within silence windows, or be seamless continuous joins of adjacent segments from the same source. When adhering to end-of-sentence points, automatically reserve a maximum of 0.30s for reaction/breathing within the silence window, never crossing hard cut points. The minimum segment duration is 0.8s. The final beat of the video must allow 1.5–2.5s of lingering expression after the final action or line, and end cleanly before transitioning to the next scene. Interrupting an original dialogue line will trigger a unsafe_clip_sentence_boundary block and provide nearest_safe; if falling back to a hard cut adhesion simultaneously exposes <= 0.20s flash frames and truncated lines, it will directly trigger a shot_speech_conflict block.
• High-Frequency Quick Cuts and Flash Frame Check: After cut.py renders edited_source.mp4, it automatically calls shot_review.py to generate shot_review.json and directly blocks extremely short flash frames (flash_slivers) with duration <= 0.20s (approximately 1–5 frames). For action scenes or dense shot-reverse-shot sequences, you can also manually run ffmpeg -i <video> -vf "select='gt(scene,0.35)',showinfo" -f null - or the read-only tool scripts/shot_review.py (usage in references/shot-review.md) to locate high-score cut points. However, you must distinguish between cut points inherent in the original video and cut points created by this assembly. If they are irrelevant short shots from the original video being entirely deleted, or if cut points are moved to the outside of a fast-cut cluster, that's fine. If they are cut points created by this assembly (e.g., remnants of a few frames from a neighboring shot at a hard cut edge, or fragmented shot-reverse-shots), then expand the entry/exit points or merge adjacent segments from the same source. It is strictly forbidden to use transitions to cover up such issues.

After writing clip_plan.json, re-run recap.py. The script will call cut.py to perform a three-step boundary adhesion (hard cut adhesion → end-of-sentence silence adhesion with tail → frame grid alignment), render a constant frame rate edited_source.mp4, complete a flash frame check, and then remap ASR, VLM, and end-of-sentence anchors to the output timeline of the finished video (speech_boundary_anchors_output.json, storyboard/edited_storyboard.*), followed by another pause to wait for commentary writing.

4.3 Commentary and Original Sound Silence Creation (full / cut Phase 2)

Entering the writing phase (full mode at the first pause, cut mode at the second pause after cutting edited_source.mp4), first review the post-cut storyboard, complete visual_audio_board.json with the finished video output times, and write style_card.json as needed. Then compose work_dir/narration.json:

[
  {
    "start": 0.2,
    "end": 5.0,
    "narration": "After five years of dating, carrying a household registry to propose, Shen Man pushes open the door and catches her fiancé embracing an heiress, discussing a marriage alliance.",
    "pause_after_ms": 250,
    "overlaps_speech": true
  }
]

Note: In cut mode, narration.json must directly use the OUTPUT timestamps of the edited video (0 .. total duration of edited video) (when calling the validator independently, pass validate.py --mode cut_output). Follow nine core writing rules when writing (detailed rules and examples of good/bad practice are in references/creative-editing-playbook.md):

1. Reveal the hook or audio highlight within the first 2.5 seconds, directly presenting a specific contrasting action or core suspense. Do not start with character resumes, date announcements, or pleasantries. A cold opening exceeding 2.5s with neither voiceover nor original dialogue will trigger a weak_cold_open_silence block.
2. Clarify background, character relationships, and causality. Control naming frequency. No "seeing is saying": When background_research.json provides character names, strictly forbid using generic appearance descriptors like "man in black" or "young woman" (violation triggers generic_character_label block). For videos 60 seconds or less, name at most 1–2 core characters (name at most 1 person in the first sentence, while clarifying their identity). Refer to all other minor characters by their relationship/status (e.g., "ex-girlfriend," "her boss," "imperial physician"). Drive every sentence around "situation → action → immediate consequence."
3. Audio-visual alignment "pad half, yield half, follow up half" and mandatory synchronization for strong visuals: Defer to high-impact original dialogue, slap sounds, or silent reactions before a change of expression. After the original sound settles, follow up with half a sentence to reveal the subtext. When the visual shows strong actions like kissing, slapping, kneeling, fainting, or revealing an object, the causality must be positively addressed either by retaining the original sound or through an adjacent voiceover. Prohibit a dead silence window exceeding 5.0s continuously (no voiceover, no original dialogue). Violation triggers an excessive_dead_air block.
4. Strictly adhere to dual red lines for original sound entry and release: When lowering original sound to introduce voiceover (overlaps_speech: true), start must be tightly adjacent to the end-of-sentence anchor or silence zone (violation triggers interrupts_source_sentence block). When releasing the original sound after the voiceover ends, similarly do not restore volume abruptly in the middle of a character's sentence (violation triggers unsafe_source_release / unsafe_source_handoff block during validation and assembly stages, respectively).
5. Strictly control word count within the time window, maintain natural speaking speed, and end with complete sentences: Each segment must end with a full-sentence punctuation mark (incomplete sentence triggers a hard incomplete_sentence block). Rely on concise wording to fit the time window, not on high-speed jam-packing. In full mode, exceeding the budget by 1.25x triggers a direct over_budget block. The validator will never arbitrarily delete characters for the Agent.
6. Eliminate editorializing, moralizing, and telegram style. Let the ending be conveyed through visuals: Write natural, fluid spoken sentences with varied lengths. Do not write wordy, staccato fragments. Strictly forbid commentary and cliché phrases like "This means...", "This is a contest of...", "Let power and pride step aside," "The wheel of fate," etc. (triggers editorial_commentary_tone warning). The story's ending must be played out on screen and through original sound. Do not add a superfluous summary voiceover after the ending has been delivered, nor drag in an original sound tail that repeats the same result after the closing narration.
7. Anchor on specific visual evidence and discipline on instrumental background music: Use visible looks, props, and action details. Do not fabricate internal character thoughts beyond what is shown. If using external background music (BGM_PATH), it must be purely instrumental (zero vocal lyrics). If the original video already has unavoidable high-volume vocal interludes, do not layer external background music to avoid multi-track vocal conflict.
8. Approve Original Sound Subtitle Text: When enabling the original subtitle overlay and retaining high-impact original sound passages, simultaneously write the proofread original dialogue lines in work_dir/original_subtitles.json according to the edited video timeline, correcting any ASR homophone errors.
9. Complete three rounds of self-check after writing: First, read through the full text to check if the story line flows and facts match the original video. Second, check against the time windows for any talking over or exceeding the word count. Finally, run validate.py (or simply re-run recap.py). If writing external promotional titles, cover frame, and on-screen text plans (packaging_plan.json), these must be executed strictly according to references/promotional-copy.md after the main story is locked.

4.4 Dubbing Synthesis and Audio-Video Mix Rendering

After writing narration.json, re-run the same recap.py command:

1. Local Deterministic Validation (validate.py): Generates narration_lint.json. Performs hard gate checks for time overlap, out-of-bounds, word count over budget, incomplete sentences, cold opening silence, dead silence exceeding 5 seconds, interrupting original speech, half-sentence release back to original sound, generic character labels, and placeholder leakage. Stops immediately if any error exists.
2. TTS Dubbing (voiceover.py): Synthesizes speech segment by segment and writes to tts_segments/ and tts_meta.json. Caching is precisely hit based on text, voice, model, and reference audio fingerprint. If the audio duration of a segment falls below the normal lower limit of speech rate (TTS_MIN_SPEED_RATE), suspected of model hallucination, it automatically retries and retains rejected files (*.rejected.wav) out of the database.
3. Audio-Video Assembly (assemble.py): Mixes and renders the final MP4, including the video, segment-by-segment voiceover, original sound (lowered and then raised at end-of-sentence points, see references/data-schema.md for source_handoff_status), optional BGM (BGM_PATH), optional decorative transition sound effects and sophisticated visual packaging (work_dir/packaging_plan.json: supports 9:16 vertical stage/series dual-gold-line frame, 16:9 cinema stamp and creamy white paper feel subtitle strip, @2x oversampled character nameplates/data cards/calligraphy on-screen text, and double-layer soft glow keyword highlighting subtitles; or packaging_layers.json static overlays), and a smooth fade-out at the end (default 350ms). It also outputs a backend-agnostic timeline.json (passing --export-jianying allows simultaneous export of a CapCut draft).

If any segment of voiceover cannot fit its available time window even within the allowed slight acceleration limit, the assembler directly reports a no_safe_fit block. The voiceover tail will never be truncated. For more details on audio modes, audio routing, and external visual packaging specifications, see references/audio-modes.md, references/audio-routing.md, and references/packaging.md, respectively.

4.5 Multi-Video Concatenation and Resource Library

Multi-video input only supports cut mode. The project-level brief lists the fixed source_id for each video. When writing clip_plan.json, each segment must specify its source_id:

python3 scripts/recap.py ep1.mp4 ep2.mp4 --edit-mode cut --target-duration 10m --work-dir work_dir_multi_ep

To store single-episode understanding results in a local directory for cross-episode reuse, enable the lightweight file material library (--material-library-dir .video-materials --save-materials and --use-materials), and directly use grep to search JSON/MD files. Under the same library root directory, you can also register reusable production resources (BGM, sound effects, voices, fonts, stickers), templates with adoption records (subtitle_style, packaging, production_reference), and sample videos. Query them via scripts/library.py check|list|show, and bind them to a single run with --project recap_project.json. After each assembly, the actual resources used and their authorization status are recorded in work_dir/resource_lock.json. See references/resource-library.md for details.

To measure the vertical pixel range of hardcoded subtitles in the original video for precise masking, run tools/measure_subtitle.py <video> in the full source repository, or directly view the extracted frames to measure the top and bottom boundaries, then pass --subtitle-y-top and --subtitle-y-bot.

4.6 Final Viewing and Delivery Review

No script metrics, cut-point scores, frame montages, or automated QC reports can replace watching the video and listening to the audio with your own eyes and ears. Before each delivery, you must directly open the final deliverable video file for this round and complete a five-step review:

1. Watch through at normal speed: From start to finish without speeding up or skipping. Note any jarring or stuttering points as you go.
2. Scrutinize each edit point: Examine 0.5–1 second before and after each cut point for flash frames, half-dissolves, action jump cuts, or truncated original sound half-sentences.
3. Close your eyes and listen purely: Without watching the visuals, listen to the audio. Check if the voiceover is fragmented or telegraphic, if the sound flows smoothly at transitions, and if key original dialogue lines are clear.
4. Isolate three key moments: Individually re-watch the first 5-second opening hook, the mid-point emotional climax, and the final scene. Confirm that the entry timing, emotional hold, and lingering feeling at the end are all solid.
5. Revision cross-check and specification baseline: In REVISION mode, first confirm point by point that "the explicitly requested changes are actually done, and the locked items that were not mentioned remain completely untouched." Finally, check final_qc.json for decode integrity, total duration, flash frame scan, and audio-video encoding specifications (--require-final-qc is enabled by default; blocking items directly determine the exit code; only pass --no-require-final-qc for debugging).

When delivering the finished video, be sure to check the metadata.warnings in final_qc.json. If it is not empty (e.g., after enabling partial fault tolerance, partial_tts missing segments appear, or if the local ffmpeg lacks libass triggering a subtitle_burn_degraded fallback to external .srt), you must truthfully inform the user. Do not simply report "rendering successful."

4.7 Finished Video with Original Sound, No Commentary Needed

# Directly mix the current full input video; does not implicitly run understanding, ASR, or TTS
python3 scripts/recap.py locked_picture.mp4 --work-dir source_work --audio-mode source-mix
# Generate an editing plan via the cut flow, then directly assemble the finished video with original sound, without pausing for narration.json
python3 scripts/recap.py ep1.mp4 ep2.mp4 --edit-mode cut --work-dir cut_work --audio-mode source-mix

source-mix will go through the complete mixing, smooth fade-out, and audio re-encoding. Always use a brand new work directory when switching audio modes. See references/audio-routing.md for details.

5. Self-Check and Read-Only Dashboard

python3 scripts/doctor.py
python3 scripts/dashboard_server.py --root <directory> [--port 0] [--open]

doctor.py checks local ffmpeg/ffprobe, subtitle filters, and connectivity of various remote service keys. dashboard_server.py starts a strictly read-only minimal review dashboard on 127.0.0.1 (only responds to GET and HEAD). It automatically scans the finished video working directories and library.json under --root, providing an intuitive display of video playback, voiceover and shot alignment, and material library preview.

6. Outputs and Parameters

Core deliverables:

• recap_<video>.mp4: The final commentary or original sound finished video.
• subtitles.srt / subtitles.ass: Generated subtitle files.
• work_dir/: Complete set of intermediate analysis, planning, and validation outputs. Field contracts are in references/data-schema.md.
• work_dir/recap_story_plan.json and visual_audio_board.json: The Agent's finalized director's intent and storyboard audio-visual decisions.

The full set of command-line parameters is based on the output of python3 scripts/recap.py --help and the --help of each stage script. --style is a free-form text requirement passed directly to the writing Agent without translation; it is not a fixed preset or enum toggle.

7. Capability Boundaries

• Local deterministic baseline validation (residual/flash frames, dialogue truncation, voiceover overlaid on dialogue, strict word count time window upper limit, tail truncation, mandatory evidence loss, placeholder leakage, decoding and packaging specifications) always runs as a hard gate. The scripts only handle mechanical execution and engineering floor support.
• Semantic and narrative quality aspects such as story arc, opening hook, audio-visual breathing, and fact consistency are not subject to model scoring gates. These are autonomously controlled by the Agent against visual evidence, original sound timeline, and references/creative-editing-playbook.md to complete pre-delivery audio-visual review.