Talking-Head Motion Edit Pipeline (3:4 Motion-Enhanced Video)

The speaker says something, and the visual responds. Graphic segments are "a single protagonist acting out a chain," not page-turning cards.

This skill is built on the base skill cangshu-koubo-wudongxiao: it uses that skill's scripts and rules for trimming flubs, pauses, cut points, subtitle sources, and sensitive words. This skill only adds 3:4 cropping, graphic segments, B-roll, BGM, and sound effects. Both skills must be installed in the same skills directory.

File Map

File	When to Read
`references/style-bible.md`	Must read before writing the plan: rhythm, live-segment overlays, entering/exiting graphics, transitions, camera, art style, sound effects
`references/scene-guide.md`	Refer to when writing graphic segments (hand-written scenes): coordinates and camera, kit/lux/toon tools, art style themes, clipping checklist, sound effect hints, example breakdowns
`references/plan-schema.md`	Check fields when writing `plan.json`
`references/metaphor-bank.md`	Run through the divergent thinking process for each graphic segment's argument
`references/demo-patterns.md`	Pick demonstration patterns: Section 2 (pattern library), Section 4 (grabbing attention)
`references/layout-34.md`	Read Section 2 safety area before writing the plan; read Section 1 for shooting advice (when the user asks how to shoot)
`references/sfx-palette.json`	Sound effect inventory: clip → asset, trim, gain
`references/board-draft.md`	Only view when drafting or aligning the timeline
`assets/example-plan.json`	Complete example; graphic segments are the template's `custom/stock1.tsx`, `stock2.tsx` (white card style); animation style example is `custom/toon_panel_example.tsx`. The example's `tag`/`pill` capsules and "label flies into graphic segment" are old practices; do not copy them.

Environment

uv venv -p 3.12 <temp-dir>/venv && VIRTUAL_ENV=<temp-dir>/venv uv pip install mlx-whisper "opencv-python-headless<5" numpy scipy pillow

opencv must be below version 5 (version 5.x lacks a face detector). Throughout this document, $SK = this skill's directory, $B = the base skill's scripts/ directory (cangshu-koubo-wudongxiao/scripts at the same level as this skill), $PY = the python from the venv above.

Runs only on macOS (Apple Silicon): transcription uses mlx-whisper (model reads from local cache first, otherwise auto-downloads mlx-community/whisper-large-v3-turbo), keying uses the system's Vision (requires swiftc from Xcode Command Line Tools). Also requires ffmpeg and Node 18+.
Rendering uses Remotion 4.0.520: new_project.sh will run npm install (if KOUBO_REMOTION_CACHE is set, look for the same-version dependencies in that directory first for reuse). Remotion has its own license: free for individuals and small teams; companies above a certain scale must buy a Remotion commercial license. Remind the user once before starting.
The sound effects library is not distributed with the skill. On first use, get user consent (source: 30 from mixkit.co + 2 packs from kenney.nl, ~3.3MB), then download with python3 $SK/scripts/fetch_sfx_library.py <directory>. After that, use build_sfx.py with --library <directory>, or set the environment variable KOUBO_SFX_LIB. If the user does not want to download, set meta.autoSfx: false in the plan and omit cues for a no-audio version.
BGM, image libraries, product images, and character sprites are all provided by the user; do not search for them online.

Workflow

1. Accept Input, Create Project

Original video (required); final draft of the talking-head script or subtitle SRT from CapCut/other software (if available); brand text (meta.brand, the user's account name or brand name; if not available, do not enable the brand word wall); background information (for data cards); licensed BGM, images, and product images.
Ask if this video belongs to a series with established practices (art style, intro, BGM): if yes, follow them and do not ask again.
bash $SK/scripts/new_project.sh <project>. Ask the user where to place the project.

2. Trim Flubs, Align Subtitles (Using Base Skill), Output 3:4 Footage, Keying, Sound Effects

Trim flubs, repetitions, and pauses; correct cut points; handle subtitle sources and sensitive words; align subtitle timing. All using the base skill's scripts and rules (its references/cut-rules.md, captions.md), performed inside <project>/base/. This skill does not create a separate set of rules.

$PY $B/prepare_source.py <original-video> <project>/base/source.mov    # Step 0: prepare source.mov; if the user already applied beauty/smoothing filters in export, use the three-parameter version (see base skill Step 0)
$PY $B/transcribe.py <project>/base/source.mov <project>/base/raw_transcript.json
#    Write <project>/base/edl.json according to the base skill's cut-rules.md (source set to source.mov; if there is no fixed series intro, write "intro": false at the top level)
$PY $B/cut_takes.py <project>/base
$PY $B/find_ng.py <project>/base                        # Phrase-level transcription to find duplicates; must run
$PY $B/edl.py rebuild <project>/base --remove <start-end>...  # Delete flubs, trim pauses >0.4 seconds, tighten both sides of cut points, remove fragments
$PY $B/cut_takes.py <project>/base && $PY $B/find_ng.py <project>/base
$PY $B/transcribe.py <project>/base/cut.mp4 <project>/base/cut_transcript.json --prompt "The following is a Mandarin talking-head script. I am <speaker name>, talking about <topic>."
python3 $SK/scripts/srt_captions.py <subtitle.srt> <project>/base/subtitle_source.txt      # User provided SRT: use its text for the draft
python3 $SK/scripts/captions.py <project>/base/cut_transcript.json <project>/base/subtitle_source.txt   # No subtitle provided: create draft from transcription
#    Edit subtitle_source.txt according to the base skill's captions.md: correct typos, proper nouns, [keywords], sensitive words
$PY $B/captions.py <project>/base
$PY $SK/scripts/from_base.py <project>/base <project>/public  # Correct cut points + crop to 3:4 + render 8-frame optical flow fusion, write words.json, cuts.json, copy captions.json
bash $SK/scripts/matte.sh <project>/public/footage34.mp4 <project>/public/person.webm
python3 $SK/scripts/build_sfx.py <project>/public --library <sfx-library-directory>

From here on, all timing uses public/words.json's timeline (consistent with base/cut.mp4 when no speed change). Default: no speed change; only use from_base.py --speed 1.1 if the user requests a speed change (using the same approach as the base skill: series intro at original speed, then overall speed change with pitch-preserved audio; subtitles and word timings are converted accordingly). Follow the base skill's rules for intro stabilization and vocal silence removal: do not do them proactively; ask first.
If from_base.py prints that the "face center y" deviates from 0.3–0.55, check the crop and specify with --face-y; if the original video is already 3:4, scale to fit without cropping.
For keying, render one frame over a dark gray background and check the hair and shoulders; if the key is poor, note it and remove the meta.person field when writing the plan (do not write an empty string); the template will degrade gracefully.
build_sfx.py builds sound effects from the library according to references/sfx-palette.json; do not find additional material (selection principles are in style-bible Section 10).
Subtitles: add captions.json to the plan's captions, set meta.captionStyle: "pop" (keywords enlarged 1.32x and highlighted in yellow, appear with a bounce, disappear in one frame).
The default sharpness values for footage (crf 10 + light sharpening), keying (crf 18), and rendering (PNG frames + crf 12) are already hardcoded in the scripts and templates; do not lower them (they would look blurry).
If BGM is not yet decided, provide an audition first (user requests recommendations, or it is not established in the series): pick 6-9 stylistically different tracks from the user's licensed library (if attribution is required, remind the user at delivery). For each track, find a position already on beat, then create a playlist from the first 8 seconds of the trimmed footage. Let the user choose:
```
$PY $SK/scripts/bgm_audition.py <project>/public/footage34.mp4 <project>/bgm_audition --track "name=path@start-second" --track "name=path@start-second"
```
The audition audio is about 11dB quieter than the voice (20dB quieter is nearly inaudible). After selection, hard link or copy the chosen track into <project>/public/ (meta.bgm.src only recognizes filenames under public/); for the final video, the BGM is about 12.5dB quieter than the voice: run it again with --rel -12.5 for this single track, set meta.bgm.volume = 10^(printed gain/20), meta.bgm.from = start second. If the track is shorter than the video, it loops. Inform the user at delivery. If the user has no BGM, do not add any.

3. Segment Arrangement, Gather Additional Materials

Segment according to style-bible Sections 2 and 4: explain mechanisms, give examples, present data, make comparisons into graphics; express opinions, conclusions, or transitions back to live segments; graphic segments ≤16 seconds; write chapters (3 segments for short videos).
Information density relies on verified data: for key arguments in the script, search online for numbers with traceable original sources to use in data cards; if unsure, do not use them. Use general terms for on-screen sources, record actual links in the delivery notes; mark personal estimates in the script as "personal judgment." First show the user the supplementary data for confirmation.

3.5 Art Style, Intro (Decide Direction First)

Two paths for art style: lux.tsx for a refined white-card style (fixed color palette, no themes), or toon.tsx for writing scenes using meta.style to choose from 9 themes (style-bible Section 7). If the series has an established style, use it; otherwise, start with toon, and after writing the first graphic segment in Step 5, generate a style reel for the user to choose. When the user wants to change the style, only change the elements and animations; do not change the background color or word wall.
The wall theme uses the color of the real background wall: before using it, extract a frame from public/footage34.mp4, measure the background wall's color, and update the wall base color in the toon.tsx theme table. This ensures the split set blends smoothly into the graphic background.
For a brand word wall behind graphic segments (rows of brand text scrolling left and right alternately), write meta.wall: true (enabled by default for the dark theme; only works with toon scenes). The text is taken from meta.brand.
Character: default to the Kid drawn in code within toon.tsx. Only enable meta.qkid: true if the user has their own character sprite (transparent PNG): place the image in public/ipq/, register dimensions in src/scenes/custom/ipq.json (format described in scripts/split_sheet.py). The morph transition between live speaker and character (host: "morph") also requires a sprite; if none is available, do not use it; plan_check.py will block it.
For a serialized show ("Welcome to XX Episode N"), use the fixed intro custom/intro.tsx: set meta.series with the show name, episode number, and title. In the plan, place the corresponding line in a B segment approximately 3.2 seconds long. Registration method is described in custom/index.ts comments.

4. Think Through What Each Graphic Segment Will Show

For each argument, use metaphor-bank Section 1 to write one candidate for each of the four levels (Literal / Metaphor / Exaggeration / Absurdity), score them, and pick one. At least half of the video's segments should choose Exaggeration or Absurdity (provided it's understandable in 1 second). Save the table as <project>/creative_divergence.md.
Decide "protagonist + chain + state-linkage table" (style-bible Section 5): the same protagonist transforms continuously; at least 3 state-linkage chains; at least 1 callback across segments or worlds; data cards emerge from the protagonist's side as annotations, then retract after the point is made.

5. Write Hand-Written Scenes

Following the scene-guide, write src/scenes/custom/<name>.tsx and register it. Key points:

Camera movement only at sentence boundaries. Dwell time in a position: median ≥2 seconds (minimum 1.4 seconds) and completely still. A single movement lasts 0.6–0.9 seconds, mostly push-pull; use Camera3D + orbitAt for a 3D perspective over the entire canvas.
Art style not decided yet: first write the first graphic segment (toon), arrange it in the plan, then run $PY $SK/scripts/style_reel.py <project> <start-second> <end-second> --themes dark,white,chalk,news to generate a style playlist for the user to choose from. If they want "more options," use other themes (blue, neon, water, toon, wall). Only change the art style; do not change the actions or text. Once selected, record it in meta.style before writing the remaining segments.
Art style follows Section 3.5: use lux.tsx for refined white card, toon.tsx for all others (Stage wraps scenes, colors only from T.*, text and lines directly on the background use T.onBg / T.onBgSoft); only add Tilt to flat surfaces.
Draw objects with code (flat vectors in custom/objs.tsx; add missing ones there), do not use realistic icons.
Keyframe timing: verbs ≤0.1 second error, nouns 0.3–1 second in advance, numbers 0.4 seconds in advance.
First element appears after contentStart(seg) (nothing appears during cut transition animations); any element that has been still for 5–7 seconds must have a new element or change (style-bible Section 5, information density).
Only split a B-roll segment into multiple blocks if the information does not fit on one screen. Move the camera between blocks (shift half a screen, each screen filled; style-bible Section 6, "multi-canvas within a segment"); if it fits on one screen, do not split.
When the script mentions a concrete object (a painting, a PPT slide, a mini-app, a manuscript), you may insert a real image: only use user-provided licensed images, placed in public/real/; AI-generated images get an "AI-Generated" tag; only use images, not videos found online (style-bible Section 7). If the user has not provided an image, draw it with code.
During layout, follow scene-guide Section 5 to prevent clipping and intersection; cues only for key transition sound effects.

6. Write plan.json

Write segments, captions, chapters, and live-segment overlays according to plan-schema. Numbers in live segments go into the top-left data card (metric); conclusions use foreground large text; parallel keywords use side words (sideWords); the conclusion of a reasoning chain uses a sticker (sticker); occasionally, two- or three-character keywords use a large text behind the speaker at the top (topWord, look: "steel"); occasionally, list items use a top-left info box (infoBox); when both appear at the start, the large text first flies into the info box's corner badge. Follow style-bible Section 9 for entering/exiting graphics (enter / host / exit): split set with a corner half-body shot; fade out with a sphere; do not mix them; use split set only when the background has distinct objects on both left and right sides, otherwise use fade out; if the B-roll's start or end has a character and a sprite is available, a morph transition can be added (host: "morph" + hostPath); if exit is omitted, it defaults to a hard cut. Live segments should not be shorter than approximately 2.2 seconds. Do not use small text capsules (tag, pill) in the top-left or top-right—they feel too AI-generated. Also avoid annotation cards, sticky notes, news tickers, decorative text, and non-text motion prompts by default.

$PY $SK/scripts/fit_bigtext.py <project>
python3 $SK/scripts/plan_check.py <project>/src/plan.json

For split-set segments, generate split-set footage (separate the frame into three layers: empty wall, left object, right object from the cut frame): $PY $SK/scripts/split_plates.py <project>. If segment boundaries change, re-run. If the split boundaries are incorrect, adjust thresholds in the script based on your background; if you cannot get it right, switch to fade out. fit_bigtext.py calculates occlusion (≤35%) for large text behind the speaker, ensuring it does not overlap the top progress bar; if it does not fit, move it in front of the person. For plan_check.py, clear all "must fix" items; for "recommend fixing" items, address each one or explain why not.

7. On-Screen Text Self-Check

python3 $SK/scripts/screen_text.py <project>/src/plan.json --with-captions

Text from hand-written scenes is also extracted. Text within inserted real images cannot be extracted and must be checked manually. If a compliance/sensitive-word-checking skill is installed in the project, delegate to it for read-only detection (specify the target platform and whether it is a commercial promotion; report text cards and subtitles on separate lines). If not, manually check each line against the base skill's captions.md sensitive word list and the target platform's common compliance violations, then show the extracted text to the user for verification. If risks are found, fix them first, then re-extract and re-check. If only positions are shifted later, with no text changes, re-extract and compare with the previous run line by line. If identical, keep the previous conclusion.

8. Preview

cd <project> && npx tsc --noEmit -p . && npm run preview
bash $SK/scripts/stills.sh <project> <name> <second> <second>

stills.sh outputs half-resolution stills stitched horizontally; view them at each camera landing position and element transition point. Show the half-resolution version to the user, then modify scenes and plan based on feedback. If the user wants to compare multiple approaches: render the same segment with different morphs (Morph-<dissolve|iris|flip|poof>) or large text styles (Top-<steel|extrude|...>).

Information density check: measure visual changes per segment (subtracting subtitle strip and progress bar). If any gap exceeds 4 seconds without a new event, add something.
If after writing the plan a flub is found (the same sentence said twice): do not modify the plan's timing. Instead, use python3 $SK/scripts/cut_slips.py <project> <start-end>... to trim the footage and write meta.cuts and other fields according to the printed output (plan-schema Section 1). Then re-transcribe a few seconds around the cut points to confirm only one instance remains. If a dark theme was chosen, check the entire video segment by segment for "dark text on dark background" or "white text on white card."

9. Final Render + Quality Check

python3 $SK/scripts/render_final.py <project> --ver v1 --py $PY

A single command runs: render → measure audio delay introduced by rendering and compensate if clipping → measure lip-sync (qa/lipsync.txt, if median audio-video offset exceeds 20ms, it needs attention) → final audio quality check → render pure sound effect tracks and measure loudness individually → output storyboard collage (5 frames per second for graphic segments and chapter transitions, 2 frames per second for live segments), write results in out/<ver>/qa/quality_report.txt. Then:

Inspect each qa/sb_*.jpg for four types of errors: elements crossing the red line (progress bar area), being clipped by the frame, two elements intersecting, or two text layers overlapping. For suspicious areas, extract a close-up: $PY $SK/scripts/storyboard.py <final-video> --at 25.1,25.25,25.4 --w 405 --hud --out <image>.jpg.
Clear all "needs attention" items: adjust meta.sfxVolume if overall sound effects are too loud or too quiet; adjust individual cue vol if specific ones are off; delete one if they overlap. If remaining differences are within 0.3dB, they are measurement errors; note this and stop adjusting.
After modifications, increment the version number and re-run. If only audio was changed (sfxVolume, cue vol, BGM volume), do not re-render the video: use bash $SK/scripts/remix_audio.sh <project> v1 v2 --py $PY (re-render only the audio track, merge with the original video, and re-run quality checks). If any visual element changed, re-render the entire video.

10. Delivery

Provide the final video path and duration, the percentage of each world type, the number of sound effects and their individual loudness ranges, and the overall loudness (all in quality_report.txt), along with the on-screen text self-check conclusion and whether manual review is required. Give the path to the high-resolution version directly; do not compress a mobile version.

For a 3:4 cover: place the screenshot in public/, set meta.cover with two lines of text, then run npx remotion still src/index.ts Cover out/cover/cover_3by4.png --frame=0. Also run the cover text through Step 7's self-check.
If the user wants different sound effects, list the candidate assets and let the user listen to them. Then modify the source list in $SK/references/sfx-palette.json (all future projects will use it) and re-run build_sfx.py; if the change is only for this project, copy the source list into the project directory, modify it there, and rebuild with build_sfx.py <project>/public --palette <that-file>. The src/sfx-palette.json file in the project is an artifact; do not edit it manually.

Hard Rules (Each Has Caused Issues Before)

On-screen text must come only from the script, the talking-head script, or user-provided information; do not fabricate numbers, comments, or sources. Watermarks must use the user's brand.
Any graphic segment longer than 6 seconds must be written as a hand-written scene; the board canvas is only for drafting (auto-layout may create errors, like PPT slides).
A graphic segment must have one protagonist and one chain, with at least 3 state-linkage chains; do not switch between isolated props as if flipping PPT slides.
The camera must come to a complete stop: dwell time median ≥2 seconds, minimum 1.4 seconds, completely still; do not "slow-push while holding" (the audience cannot see clearly if transitions are too fast).
Use Tilt only to save a flat-looking scene; do not add it to every card unnecessarily.
Text in live segments uses frosted-glass data cards, top-left info boxes, word-by-word rising large text, side words, and stickers; do not use small text capsules in corners. Large text behind the speaker (topWord) is for occasional use only (perceived as lower quality, text easily blocked by the speaker, and capsules feel AI-generated).
No clipping or intersection: the entire path must stay within the safe area, not cross the top progress bar, not intersect other elements; pipes, water flows, and beams must stop at the contact surface; during morphing, two text layers must not overlap.
Sound effects: few and quiet. Use them only for key transitions: 0.3–0.7 per second in graphic segments, at least 0.5 seconds apart, each 1–6dB quieter than the median voice level. Use bright, realistic assets (too many or too loud becomes noise and feels AI-generated).
To sync template changes into a project, only use bash $SK/scripts/sync_template.sh <project>: it does not touch the project's own plan.json, sfx-palette.json, the hand-written scene registration table scenes/custom/index.ts, or the project's own scene files (a full rsync once overwrote the sound effects list with an empty one, resulting in two final render versions with no sound effects at all). However, remotion.config.ts and the tool files under custom/ (toon.tsx, lbkit.tsx, intro.tsx, qkid.tsx, ep.tsx, real.tsx) are overwritten every time. Project-specific props should be written in separate files. new_project.sh will reject an existing project. To copy improved template code back from a project into the skill directory, use sync_back.sh <project> <file-relative-to-src>....
After finding a problem, fix it immediately, re-render, and provide the result; do not just report your thinking.
Live-segment foreground text on a light wall must have a dark outline (the template's BigText and SideWords already have two-layer text shadows + soft shadow; do not remove them). For dark themes, text and lines written directly on the background use T.onBg; gray text on cards uses T.inkSoft (white text on white background is illegible).
All multi-segment audio/video concatenation (BGM audition, style reel, playlist) must be resampled to 48kHz and re-encoded using the concat filter. Do not use -c copy (mixing 44.1kHz/48kHz can pitch-shift the voice). Write the concatenation command in Python; in zsh, $i:a is interpreted as a path modifier, and set -- $x does not split by spaces correctly.
Remotion's static file server does not follow soft links: use hard links (ln) or copy assets from other projects into public/. When rendering the final video, remotion still can time out; always use stills.sh for self-check stills (deleting old images first; fail with an error if rendering fails). Otherwise, you might mistake an old image for a new one.
Motion serves the content: cut transition animations must not cover content (default to hard cut when returning to live segments). Elements must not be blocked by keying, spheres, or subtitles. Do not mix two types of cut transition animations (do not add a sphere after a split set).
Information density: any element that has been still for 5–7 seconds must have a new element or change, unless it is itself providing a large amount of information. The character is a supporting element; do not design motion around the character.
Do not animate the camera for its own sake: if information fits on one screen, do not split it into blocks. Only split a block within a single B-roll segment, moving half a screen and filling each screen (page-turning between different B-roll segments is meaningless).
Real-world assets inserted into graphic segments must only be user-provided images, not videos found online. The image must depict what the script is talking about. AI-generated images get an "AI-Generated" tag (real-world videos break the art style). User-provided authorized real-life photos are still a live segment (world S), not restricted by this rule.
Audio and video must not be separated: footage and clips all come from the same source.mov and the same editing table. The template must mix BGM and sound effects; audio can only be a separate layer, with the delay measured and compensated by render_final.py. Do not manually shift audio (rendering introduces a delay of tens of milliseconds, causing lip-sync errors).

Modifying the Template

The template is in assets/remotion-template/src/: Film.tsx (world switching, split-set/fade-out/morph transitions, speaker, audio; sfxOnly for sound-only output), timing.ts (cut transition timelines, contentStart, time conversion for trimmed flubs), morph.tsx + morphState.ts (live-to-character morph transitions, requires sprite), components/ (base primitives, overlays live-segment overlays, hud progress bar, extra top-left info box and top large text behind speaker), scenes/kit.tsx and lux.tsx (hand-written scene tools and white card style), scenes/custom/ (types.ts types, index.ts registration table, example stock1 and stock2; toon.tsx animation tools + 9 themes + brand word wall; qkid.tsx optional character sprites (images in public/ipq/, dimensions in ipq.json, default empty); ep.tsx product-UI-style props; objs.tsx flat vector objects; real.tsx real images and multi-canvas within segments; lbkit.tsx animation-style props; intro.tsx series intro; toon_panel_example.tsx examples), Root.tsx (Koubo34 outputs final video based on meta.style; Style-<theme> outputs style reel; Morph-<method> / Top-<style> outputs morph and large text comparisons; Cover outputs cover), Cover.tsx, scenes/scenes.tsx (single-card layouts), sfx.tsx (auto-attach sound effects), scenes/board.tsx (for drafts), scenes/props.tsx (for cartoon-style draft props, TablerIcon inside is also used in the final video). Fonts are in theme.ts under F (defaulting to macOS system fonts like PingFang, Avenir; replace with licensed fonts before commercial use). To add a field, first modify plan.ts, then add a line in plan-schema.md. For improvements made in a project that should be kept for the future, use bash $SK/scripts/sync_back.sh <project> <file-relative-to-src>... to copy them back to the template. To sync to an existing project, use sync_template.sh.

QQTangBa/cangshu-koubo-motion-edit

What is cangshu-koubo-motion-edit?

Ask in your favorite AI

Documentation