High-Precision Video Script Reverse Engineering

Turn a dynamic video back into a static text script that can be reproduced for filming. Core criterion: a reader who does not watch the video can still "see" it — what is said, how it is said, how the face changes, who the eyes are looking at, and why it is said.

When to Call

The user provides a video file and asks to "break down the script / restore the script / frame-by-frame analysis / scene analysis / convert to a text script / how was this video shot"
Need to extract the sales talk structure (pain point → empathy → evidence → offer → call to action) from a reference video
Need to provide a performance prompt sheet (tone, expression changes, gaze targets) for an actor/director
Need to reverse-engineer storyboards and editing rhythm from a benchmark video

Not applicable: pure subtitle extraction (use ASR alone), video editing itself, music transcription.

Unbreakable Rules (First, Remember These Five)

Zero Dialogue Cleanup: Keep "um / uh / hmm / huh / ah / sigh / oh / tsk" — they are key evidence of naturalness and character assignment. Do not beautify or delete.
Eyes Must Be Named: Write "[A]'s gaze goes to [B's specific identity/body part] + duration of stay." Prohibit vague expressions like "looks at the other" or "deep eye contact."
Facial Expressions Must Show Trajectory: Progress in stages: [start] → [middle] → [end] (e.g., wide-eyed → pupil dilation → brow relaxes, lips pursed). Prohibit only a single static adjective.
Vocal Quality Must Describe Sound Texture: Pitch, volume, speed, breath (breathy voice/cracked voice/gasping), accent placement, pause length — write at least three of five items.
Speaker Must Pass Verification Chain: ASR diarization will inevitably be wrong → segmented re-transcription comparison (P6) → per-second subtitle baseline table (P3.5) → lip-movement frame adjudication. Only after all four layers of evidence have passed may you write [Speaker].

Execution Flow (P0–P7)

P0 Material Probe

ffprobe -v error -show_entries format=duration,size:stream=codec_type,codec_name,width,height,r_frame_rate \
  -of default=noprint_wrappers=1 "<video>"

Based on this, decide the sampling density (see table below) and confirm if an audio track exists (skip P1 if no audio track, infer from visuals instead).

Duration	Frame Sampling Strategy	Max Frames	Description
≤2 min	1 frame/2s + key segment 1 frame/s	60–80	Short video/ad, all details needed
2–10 min	1 frame/5s	60–100	Narrative short
10–30 min	1 frame/10s + hook segment increased density	80–120	Short drama/long talking head, first and last 90s need 1 frame/3s
>30 min	1 frame/20s + segmental sampling	100–150	Only chapter-level restoration, first outline then deep dive

Cost Budget: Keep total ViewImage calls per task to ≤60, with 8 images per batch in parallel; when frame count is insufficient, prioritize "expression turning points, product close-ups, prop close-ups, subtitle cards."

P1 Full Audio Transcription

ffmpeg -y -v error -i "<video>" -vn -ac 1 -ar 16000 -b:a 96k audio.mp3

Use AudioTranscribe (supports mp3/wav/m4a/mp4, etc., ≤1GB or 5 hours) to transcribe. Output text with timestamps [start, end, speakerID] — this is the skeleton for all subsequent timelines.

Note: The returned speaker IDs are just cluster numbers — not equal to character roles. This must be jointly determined by semantics + tone + visuals.

P2 Frame Extraction

# First, try scene change detection (use this for videos with many transitions)
ffmpeg -y -v error -i "<video>" -vf "select='gt(scene,0.15)',showinfo" -vsync vfr -q:v 3 kf_%03d.jpg 2> scene_log.txt
grep -o "pts_time:[0-9.]*" scene_log.txt | head -100

If <10 frames are hit → indicates single-camera continuous shooting (talking head/drama). Switch to uniform frame extraction: -vf "fps=1/10,scale=576:-1"
Threshold can be lowered to 0.08 and try again; if still no transitions, abandon scene detection
For vertical short videos, unify to scale=576:-1 — enough to see expressions without wasting tokens

P3 Batch Frame Viewing

Batch 8 images in parallel with ViewImage, unified query template (to ensure comparability):

Describe this vertical video frame: person appearance, expression, action, scene, clothing, subtitle text, composition

Key points for frame reading:

Direction of character relationships (who is dragging whom, who is chasing whom, who is being taken away, who gets into which car) must be confirmed via frames — ASR will never tell you this; when two same-gender adults are in the same scene, cluster IDs cannot be directly used as character roles
Subtitle text is a valuable high-information free source (drama title watermark, brand name, product subtitles, price subtitles, disclaimer, dialogue character names) — copy them frame by frame; names appearing in subtitles (e.g., "Gu Chuan, stop stalling") combined with "who is being dragged" in the visuals, is solid evidence for dialogue attribution
Note persistent elements (watermarks, disclaimer text, logos) — write these in the document header
Remember the corresponding second for each frame (kf_001 = 0s, kf_002 = 10s...), you will need it later for scene descriptions

P3.5 Full-Play Subtitle Grid: Establish a Per-Second Dialogue Baseline Table (Primary Evidence for Speaker Attribution)

For videos with burned-in subtitles (almost all short dramas and sales videos), first get the "official answer" for the dialogue:

# Crop subtitle strip at 1 frame per second (first extract 1 full frame to confirm subtitle vertical position, in this example y=710, height 140)
ffmpeg -y -v error -i "<video>" -vf "fps=1,crop=576:140:0:710,drawtext=fontfile='<font>':text='%{pts\:hms}':x=8:y=6:fontsize=24:fontcolor=white:box=1:[email protected],tile=3x10" -q:v 3 sub/grid%02d.jpg

30 frames per image → for a 14-minute video, about 29 images. Read each with ViewImage to get "timestamp → subtitle text"
Blue-bordered/white-bordered subtitles are usually narrator transition captions (e.g., "The two were forced to part ways"). Categorize them separately, do not mix with dialogue
Two-line subtitles may lose the top half when cropped — when in doubt, go back to the full frame for that second

This per-second baseline table can solve four types of issues in one go: missing sentences, extra sentences, text typos (he/she), filler word attribution — when ASR conflicts with subtitles, trust the subtitle text first, then use lip-movement frames to adjudicate the speaker (subtitles do not indicate who said it).

P4 Build the Skeleton: Slice the Dialogue Stream into Scene Segments

Use the ASR timestamps as the axis, cut segments by scene/space change (change of location, change of counterpart, change of emotional baseline)
Name each segment in the format: ⏱️ [mm:ss - mm:ss] Event Name (Functional Label), e.g., [04:15 - 05:10] Probing at the Nursing Table · Lip Balm Debut
Write at the beginning of each segment > Scene Description: (lighting, composition, positions of main objects, character placement)
Arrange lines of dialogue in sequence by time; when multiple people overlap, separate lines and annotate "(spoken simultaneously with X)"

P5 Five-Dimensional Tagging (Tags for Each Line of Dialogue)

Tag	Required Content
🎙️ Sound Texture	Pitch progression / Volume / Speed / Breath (breathy voice/cracked voice/gasping/sighing) / Accent on which word / Pause length in seconds
😐 Expression Change Trajectory	`[start]→[middle]→[end]` three stages, describe specific movements of eyebrows, eyes, mouth, nostrils, and jaw
👀 Eye Contact	[Subject]'s gaze goes to [Specifically Named Object], duration of stay; gaze movement must include the path
🙌 Action Details	Hand posture, body orientation, movement trajectory, prop interaction (micro-expressions like white knuckles/clenching/finger trembling are priority)
🫧 Presence & Aura	A one-sentence characterization of the character's psychological state at that moment (from a director's perspective, not restating the expression)

Filler words must be analyzed as standalone sentences: A single "hmm" must specify whether it is a nasal breath sound or a throat sound, whether it is a sneer or an admission of defeat, and the interval to the next sentence — this determines whose line it is.

P6 Two-Pass Cross-Validation (Core Value of This Skill, Cannot Be Skipped)

Problem: Transcribing the whole video in one pass can merge adjacent speakers into one segment, write he/she as the same character, and write questions as declarative sentences.

Method: Select 4–6 "high-risk segments" slice and re-transcribe, then compare line by line with the full script.

High-risk segment characteristics (if any hit, slice it):

A segment contains dialogue from ≥2 characters (especially whisper segments, quarrels, crowd scenes)
Two same-gender adults appear in the same scene (similar voices, clusters are easily merged into one ID)
"He/she" appears and the referent could be ambiguous
Overlap, interruption, simultaneous speech occur
Fillers form isolated sentences (hm/oh/eh/ah)
Departure, pulling, chasing, station scenes ("who is being taken away, who is chasing" is very easily written backwards)
Timestamp span >20 seconds and speaker ID unchanged (indicates merging)

# Cut a 70–90 second segment (avoid silent boundaries, leave 1s margin at start and end)
ffmpeg -y -v error -ss 15 -t 75 -i "<video>" -vn -ac 1 -ar 16000 -b:a 96k seg1.mp3

Run AudioTranscribe on each segment, then evaluate according to the table below:

Difference Type	Assessment Method	Handling
Speaker reversal	Check if vocal positions alternate within the segment (A/B/A/B); then combine with visual lip movements and plot causality	According to alternation pattern + plot logic, add `📌 Correction Note` in the body
He/she confusion	Homophones are bound to be wrong, rewrite according to the actual gender of the referent	Directly correct, do not keep the original character
Missing sentence (absent from the full script)	Sentences appearing in the segment transcript but not found in the body	Re-add, adjust the segment timeline accordingly
Need to split merged sentences	Subject-verb disagreement in one sentence ("You go quickly stop stalling I'll wait")	Split into 2–3 lines according to meaning, attribute to each speaker
Question/statement misjudgment	Whether the ending tone rises, whether there is a "?" semantic	Accusatory sentences use "!", questioning sentences use "?"
Voice-over vs. vocalized	Whether the segment is continuous with the same vocal position before and after, whether there is a track switch	Continuous same position = vocalized whisper, must note "the other did not hear clearly" rather than "internal monologue"
Action direction reversal	Check frames: whose hand is grabbing whom, who gets into which car, which name is called in subtitles	Reverse and rewrite the entire segment, add `📌 Correction Note` with frame evidence in the body

Three-Pass Visual Verification: Conclusions involving action direction, character relationships (who is taken away/who is chasing/who is dragging whom) cannot be settled by the second pass's pure text comparison alone — must return to dense frame extraction (1 frame/s) to verify with frames, and use later back-referencing lines (e.g., "You took me away 15 years ago") to cross-lock. See real cases in references/pitfalls.md #9.

Four-Pass Subtitle Baseline Verification: Before delivery, use the per-second subtitle baseline table from P3.5 to align with each line of the script — for any line where the script dialogue does not match the subtitle text, or there is a subtitle sentence that is not in the script, correct/append each line; for points with attribution doubts, extract full frames to check lip movements for finalization (extraction time = second the subtitle appears). Two special rules:

Flashback audio bleeds through: After a flashback scene ends, the original audio may continue into the real shot (e.g., in a real shot the mother has her lips closed, but the subtitle shows a childhood "shh") — attribution follows the audio track, not the visuals;
The subtitle's "she" may not be female: The original subtitle often mixes he/she. Determine the actual referent based on semantics and later back-referencing, but when quoting the original subtitle, retain the original text and annotate.

Verification records must be included in the deliverable (table: timestamp / original text / post-verification text / basis of judgment + overlap rate + remaining uncertainties) — this is the credential for the script's credibility.

P7 Delivery and Self-Check

Produce a single Markdown file; structure as per references/script-template.md; before delivery, go through references/qc-checklist.md item by item.

Finally, use present_files to display, and in the response include: video identification conclusion (title/category/duration), core plot in one sentence, sales structure, and how many corrections were made this time.

Script Tools

scripts/extract_assets.sh does the entire process in one step: probe → audio → scene detection → uniform frame extraction → high-risk segment slicing:

bash scripts/extract_assets.sh "<video>" "<outdir>" [frame interval in seconds]
# Example: bash scripts/extract_assets.sh ./v.mp4 ./frames 10

The script outputs probe.txt, audio.mp3, kf_*.jpg (with index↔second mapping), scene_log.txt, and prints a workload table of frame count/duration.

Reference Files

references/script-template.md — Complete deliverable skeleton (character profile / tone & aura summary table / segment five-dimension / sales methodology / performance prompt sheet / verification records)
references/qc-checklist.md — 16-item pre-delivery checklist for self-inspection
references/pitfalls.md — 10 categories of high-frequency errors and countermeasures (with real cases)

Output Conventions

Use uniform time format mm:ss (use hh:mm:ss for over 1 hour). Write ranges in section headers, can mark single-point seconds within dialogue blocks
Unify character names: first occurrence write full name, in the body use 【】 brackets, e.g. 【Mother Gu】
The same character with different identities (alias/childhood/adult) must be filed separately, and the correspondence must be clearly written in the file
Product names, brand names, and prices must be verified against subtitles frame by frame (ASR often mishears brand names, e.g., "Hu Luo Wen Su" → "Carrot Su")

video-script-reverse-engineering