BigBeeCN/vertical-talking-head-edit

Vertical Real-Shot Talking Head Short Video Edit (Windows + Codex + Jianying)

Applicable scenario: You have footage shot by others (talking head + B-roll) and need to produce a 30–50 second, 1080×1920 vertical short video, which a human editor finalizes in Jianying or exports directly.

Hard rules (not to be violated) are in references/AGENTS_rules.md; place it into the project as AGENTS.md before starting work. This file only describes the workflow.

Input

• Source directory (read-only, never modify originals; first copy a set to the working directory).
• The editor's personal style habits (if any). Habits stated by the editor = hard standard; reference videos are only for learning techniques, not parameters. Conflicts are resolved by deferring to the editor.
• Optional: reference videos, brand logo placement, end-screen CTA copy (do not write unverified phone numbers/addresses).

Output

• spec.json (storyboard, the sole editing reference), rough cut or final cut mp4, .srt, per-second QC mosaic, QC report.
• Route A additional deliverable: Jianying draft; Route B additional deliverable: transparent overlay ProRes 4444 .mov.

Step 0: Environment Self-Check (Do not start if any fails)

Run the self-check as per references/windows_env.md: ffmpeg full version (with zscale, tonemap, prores_ks), Python 3.10/3.11, faster-whisper + whisperX, Playwright + Chromium + target Chinese font, optional demucs / RVM; Jianying draft directory has been added to config and included in writable_roots. Codex sandbox is disconnected by default; models and dependencies must be installed in advance. If any item fails, report what is missing—do not force-run. Specific commands for each step are in references/windows_commands.md.

Step 1: Material Inventory

1. ffprobe each file: resolution, frame rate (variable?), rotation, duration, color transfer (HLG/PQ = HDR). Write as inventory.csv.
2. Do not export SDR intermediate files: convert on the fly in the filter chain when reading frames. Apply fps=30 to all materials; for HDR materials, the read frame filter chain starts with HDR-to-SDR conversion, otherwise the picture looks grey (also for thumbnail extraction—commands in windows_commands.md section 2).
3. For each clip extract thumbnails and tag them (content, people, usable segments, occlusion/black frames/initial darkness, other brands). Mark unusable clips and time ranges.
   • Check: every clip has tags; clips with transfer characteristics arib-std-b67 or smpte2084 have read frame filter chains starting with HDR conversion.

Step 2: Transcription and Proofreading

1. faster-whisper rough transcription: medium model, device='cpu', compute_type='int8', language='zh', vad_filter=True, word_timestamps=True, initial_prompt with domain-specific vocabulary. VAD can miss sentences; run a second pass with VAD turned off and condition_on_previous_text=False.
2. First, manually proofread the full transcript against the audio: technical terms, numbers, homophones; Whisper may also fabricate sentences that were never said—delete them. For words that cannot be clearly heard, cut them from the audio directly—do not guess.
3. Before transcription, always extract 16 kHz mono WAV with ffmpeg first, then feed to whisper. Then send the proofread text to whisperX for forced alignment (write results into the align field of spec), obtaining per-word start times (0–3 frame error). whisperX word end times are unreliable; use energy detection for cut points.
   • Check: the transcript contains no unproofread technical terms or numbers; each sentence has per-word start times.

Step 3: Narrative Restructuring (Define Narrative First, Then Effects)

1. The first second must be a hook: "contrast + result", using original speech. Do not place B-roll or logo at the start.
   • The hook can splice two non-adjacent original sentences, but each half must carry a src_t, and splicing must not change the original meaning.
   • Choose one of two sound options, write into spec: the first sentence plays normally with big text overlaid; or bring the result sentence (≤3 seconds) to the start as a cold open, then return to normal order.
2. Delete sentences that do not advance the story (slogans, padding, repeated opinions). Target 30–50 seconds.
3. Mark 3–4 narrative turning points (start hook / scene transition / climax result / closing quip). Explosive effects go only at these positions; the climax is the strongest, then the beginning. Two explosive points must be at least 6 seconds apart; a ~30-second video gets 3, a ~50-second video gets 4.
   • Check: every on-screen text can be found in the transcript with its original speech and timestamp, and verify context (the same word may have different meanings in different sentences).

Step 4: Write storyboard spec.json, ask user for confirmation

Field specifications are in references/spec_schema.md; keyword category examples are in references/kw_class.example.json (the words in the example are from one project; rewriting is needed for other projects). Each segment should clearly record: source clip and in/out points, layout, subtitles, kw_class, effect preset, B-roll, sound effects, music segment, src_t (original speech timestamp). Send the storyboard to the user for confirmation before rendering.

Step 5: Editing

1. Use 10 ms frame RMS energy detection for in/out points (not Whisper word ends). Clean cuts: no breath sounds, no fillers.
   • Proven method: first cut by energy with threshold = max(10th percentile + 14 dB, −48 dBFS). Utterance considered started if 3 consecutive 10 ms windows have voice; leave 0.10 s before and 0.14 s after, then manually tighten. Check: total continuous length on both sides of the cut below −42 dBFS is ≤90 ms.
   • Editor's habit target: only 1–2 frames (≤67 ms) between the start of a word's onset and the cut. Has not been verified with this standard (to be tested). Verification method: after separating vocals with demucs, measure from the cut point to the start of 3 consecutive 10 ms windows above −42 dBFS; each cut point ≤67 ms, initial consonant not cut off, re-run ASR on the cut segment without missing words—only then adopt as default.
   • Optionally, produce a comparison version with 2–3 frames between sentences for the editor to choose from.
2. Frame-accurate export of each segment: place -ss before -i, -vf includes fps=30, use -frames:v nf for fixed length (nf = round((out − in) × 30)), libx264 -preset veryfast -crf 12 with three BT.709 color markers; audio spliced per sample point, each segment with 8 ms fade-in / 15 ms fade-out to prevent pops.
3. Jump cuts from the same camera angle add 1.08x push/pull; transitions into/out of B-roll use whip pan + horizontal blur.
4. B-roll is inserted only where the original speech mentions the corresponding content—do not insert just to fill rhythm.

Step 6: Subtitle Effects (Two Levels)

Regular level: one every 1.5–2 seconds, max interval ≤3 seconds, restrained.

• Bottom subtitles: white text with black stroke, max 12 characters per line, entire sentence appears (whether to color keywords is a parameter decided by the editor).
• Emphasis placed on middle-layer large golden text (keywords 1.75×, numbers 2×), per-character entrance 115%→100%, 6–8 frames; optional hand-drawn underline/circle, paper texture, soft shadow.
• Optional moves: number roll, physical stickers with matching foley, cumulative list (font size ≥1.2× bottom subtitle, zoom in after gathering), vertical parallel golden sentences (hide the same bottom subtitle sentence when vertical appears), opening identity tag (used only once at the start, marked "not original speech").

Explosive level: 3–4 per video, counted in the overall density, each one fully stacked:

• Text width 75–85% of screen width (85% is the upper limit, safety zone takes priority); push in 1.15–1.25 within 4 frames; shake screen 2–3 frames with motion blur; dim background 6–30%, moderate desaturation (parameter, ensure portrait layers are clear on mobile), do not color grade the person layer; pre-mixed composite heavy hit sound effect (counted as 1).
• Intensity levels: the heaviest push-in 1.25, screen shake 3 frames; others push-in 1.15, screen shake 2 frames.
• Each technique is used only once throughout the entire video: text behind person (keying) / text mask transition / full-screen number roll + red stamp / vertical golden sentence write-on + sweep light.
• Text behind person: place text above shoulder line, avoid hands and props; head must cover text 25–30% (if <15%, it's invisible on mobile phones); before stroking, erase accidental keyed objects like chairs from the mask.
• Red stamp: the only red color in the whole video, width 30–40% of screen width; smash in from 150%→100% within 3 frames + 2 frames screen shake.
• Freeze frame: can only be placed in natural silence after a sentence ends (vocal < −45 dB); choose the frame where the mouth is closest to closed; ≤0.4 seconds; max one instance per video; heavy hit sound effect pressed at the start of freeze. This is the only exception to "no silence gap allowed."
• All entrance animations start 1 frame early.

Step 7: Audio

• Vocals: HPF 80 Hz → noise reduction → light EQ → compression.
• Music: change track per chapter (editor's habit), crossfade 300–600 ms; side-chain ducking when speech is present. Use only confirmed licensed or self-produced music.
• Sound effects: approximately 0.5 per second, evenly distributed. Per 4-second window: 1–3 effects, minimum gap between any two effects 0.25 seconds. Default level: effect RMS is 14 dB below vocal RMS (db=-14). Content-matched foley takes priority over generic pops.
• Loudness: first pass measure with loudnorm=I=-14:TP=-1.5:LRA=11; second pass use measured values with linear=true, then immediately append aresample=48000,alimiter=limit=0.7:attack=1:release=30:level=false; finally recheck with ebur128, apply gain g = min(−14 − I, −2.9 − Peak). Target −14 LUFS, true peak ≤ −1 dBTP.

Step 8: Rendering and Delivery (Choose one or combine)

• Route A: Jianying Draft (default when editor only uses Jianying). Codex produces rough cut, subtitles, sound effect and music timings, creates a new draft using pyJianYingDraft. Editor opens Jianying for finishing and export. Limitations and notes in references/jianying_route.md; the entire route is verified on a real Windows machine.
• Route B: Overlay (mandatory for explosive effects). Effect layer uses HTML/Canvas; Playwright drives the system-installed Chrome to capture transparent PNGs frame by frame: viewport 1080×1920, device_scale_factor=1, launch args --font-render-hinting=none, wait for window.READY and document.fonts.ready before capturing, omit_background=True; all animations written as pure functions of time t. PNG sequence read with -framerate 30, encoded as ProRes 4444 (-alpha_bits 16), then composited with ffmpeg overlay. Add the overlay layer as a video track in Jianying draft: not yet tested. ffmpeg drawtext cannot do per-character entrance or text behind person.
• Fallback delivery: rough cut video + SRT + storyboard; finishing done by editor in Jianying.
• Before overlaying, apply slight color grading to the base layer: eq=contrast=1.04:saturation=1.07.
• Final encoding: libx264 -profile:v high -preset slow -crf 17 -maxrate 16M -bufsize 24M -g 60 -r 30 -pix_fmt yuv420p -color_primaries bt709 -color_trc bt709 -colorspace bt709, AAC 192k, -movflags +faststart.

Step 9: QC (Deliver only after all pass)

Follow each item per references/qc_checklist.md. The report only writes about things you have personally examined by sampling frames and listening to audio—each item with numbers and units. For things not measured, write "not measured"—do not fabricate values.

Cover

• Work division: Planner writes copy and selects candidate frames (written into spec's cover field); editor produces layout and exports (1080×1920 and 3:4).
• Main title: condense original speech; word order can be adjusted (e.g., "tightening screws for a month and a half"); do not add facts not present in the original speech; ≤7 characters, max 2 lines. Sub-label: result number. Stickers: 2–3.
• Both title and face must be inside the middle 1080×1440 (y 240–1680) area; Douyin home grid crops to 3:4.
• Provide at least 3 candidate frames each time, with timestamps and reasons. Mark schematic frames as "not final".
• Scan the entire segment in 0.5-second steps for profile or expression frames; line of sight determines layout (person looking top-left → person placed bottom-right, title placed top-left). Mirror flipping is forbidden if the clothing or background contains text.

Success Criteria

• Duration 30–50 seconds; first second is an original-speech hook; 100% of on-screen text traceable to original speech.
• Regular effects: max interval ≤3 seconds; 3–4 explosive effects with graded intensity and no repeated techniques.
• Sound effects: ≈0.5 per second, no simultaneous overlapping; beat-matching error ≤2 frames; −14 LUFS ±1, true peak ≤ −1 dBTP.
• Safety zone violations: 0; no unexpected black/freeze frames; text does not cover face or brand logo.
• User has confirmed the storyboard; all deliverables are complete.

Common Pitfalls → Countermeasures

Pitfall	Countermeasure
HLG footage looks grey	zscale + tonemap=hable to SDR, gamma ≈ 1.12
Whisper word times too early, word tail extends into silence	Start time from whisperX alignment; cut point from energy detection
VAD misses sentences	Run a second pass with VAD off
On-screen text is not original speech / context misused	Every line has src_t; QC cross-checks with transcription
Effects all have same intensity, looks plain	Split into regular and explosive levels; explosive fully stacked
Text behind person not visible	Measure coverage ratio; head covers 25–30%
Chair keyed into person silhouette	Place text above shoulder line; erase before stroking
Beat-matching misaligned	Measure from "first visible frame"; entrance starts 1 frame early
Sibilance mistaken for sound effect	Separate vocals with demucs, detect only on accompaniment track
Sound effects clumped together	Per 4-second window 1–3 effects; climax composite counts as 1
Chapter or large text enters Douyin button area	Right boundary ≤ x 930
Freeze frame while talking, lip sync off	Freeze only in silence, ≤0.4 seconds
Animation slides across face or out of frame	Replace with in-place fade-out + target position fade-in zoom
Inspection tool caches old images by path	Use new filenames for each check image
Command fails due to Chinese paths or spaces	Pass argument lists with Python subprocess; file reads/writes explicit utf-8
Whisper hallucinates unsaid sentences	Proofread full transcript against audio before alignment
Want to switch to alternative rendering (e.g., Remotion)	Not verified; do not replace; effect layer uses HTML/Canvas + Playwright
New draft not shown in Jianying list, wrong font	Enter/exit an existing draft or restart; open draft twice