TaurenRen/srt-whiteboard-workflow

Hand-Drawn Whiteboard Animation Complete Workflow (SRT → Hand-Drawn Animation Video)

Based on practical testing and lessons learned from the open-source project srt-whiteboard-animation (github.com/geeklee/srt-whiteboard-animation). Output: A "hand drawing while explaining" whiteboard animation MP4 with voiceover + key screenshots + operation screen recording.

Hard Rules (Failure to follow means rework, all from real-world failures)

1. NEVER deliver with example images. The examples/ folder's scene-01-monkey-mountain-banana.png is only for pipeline verification. You must generate images for the specific topic requested. Using example images as the final deliverable = failure.
2. Source images MUST be cleaned. AI-generated cream paper backgrounds have heavy texture particles. The renderer's adaptiveThreshold will interpret paper texture as ink, resulting in lines appearing as dots and background noise. You MUST run scripts/clean_source.py to remove paper texture before rendering.
3. The last element MUST NOT use the full canvas as its region. allowed mask = region − subsequent regions. If the last element's region is the entire canvas, it will carve out the allowed areas for all previous elements, making their content invisible and losing all color (confirmed in practice). The final scene must pick a real, small area (e.g., a cloud band at the top).
4. Hand assets must be scaled to the canvas. The original drawing-hand.png is 1069×1472. Pasted directly onto a 1280×720 canvas, it would cover 40%+, severely obscuring the content. In practice, scale it to 55% (587×809), so the hand's width is about 21% of the canvas width. This looks normal.
5. Rendered output duration will be SHORTER than annotated. The renderer determines frames based on the actual drawable pixels in each region; sceneDurationMs is only an upper bound. In practice, an 8×8000ms=64s annotation only rendered 45.8s. You MUST pass --total-ms and calibrate against the actual duration measured by ffprobe.
6. Screen recording must NOT be ended with kill. Force-killing ffmpeg will drop the moov atom, causing the file to throw a moov atom not found error. Use -t <seconds> to let the recording end naturally.
7. Screen recording and screenshots are MANDATORY. They are a requirement, not optional. Screenshots must cover: source image, annotation verification image, render console, keyframes of the final video. Substituting "documentation notes" for screenshots is considered incomplete.
8. One canvas tells one story, and it MUST be a "progressive progression". Real-world failure: Placing the Red Flag, Bridge, Snowy Mountain, and Pagoda side-by-side on a single image and drawing them all at once, the user said "the drawing is a mess." The correct approach is to arrange elements along a single main line (geographic route / timeline), with the pen moving continuously along that line. The viewer should see "the pen walking," not "adding details all over the canvas."
9. Spatial layout MUST follow the real direction. To draw the Long March, you must follow real geography: Ruijin is in the southeast → move west → head north → northern Shaanxi. If the composition is wrong, even perfect pacing won't save it. Before drawing, confirm the relative positions of each node on a map.
10. Hand-drawn rhythm = narration rhythm, one element per sentence. Each element's durationMs MUST equal the actual duration of its corresponding voiceover sentence (determined by sync_timing.py via silence detection). Never average it out. The pen stops drawing for the Nth element exactly when the Nth sentence finishes.
11. Scenes and narration must have a one-to-one semantic correspondence. Real-world mistake: The Pagoda (Wuqi Town) was placed in the summarizing scene, resulting in the video drawing something else while the narration said "arrived at Wuqi Town." The final summarizing sentence ("The 25,000-li Long March...") must NOT be paired with a new element. Let it fall on the "frozen frame after drawing everything is complete." The renderer will automatically add empty frames to fill --total-ms.
12. Subtitles must be burned separately; do not rely on force_style. Pure SRT has no PlayRes, so libass uses its internal default resolution for FontSize/MarginV, making precise adjustments impossible in practice. Follow the make_srt.py → srt2ass.py (explicitly set PlayRes = video dimensions, and use {\an2\pos(x,y)} to anchor the bottom edge) for control.
13. Subtitles are limited to ONE line. Two lines will simultaneously occupy the top and bottom parts of the screen. For Chinese, reading two lines feels breathless. If one line is insufficient, split it into multiple consecutive subtitle entries (distribute the duration proportionally by character count). Do not compress lines or reduce font size. In practice: 46px font size + single line ≤14 characters, which takes up about 60% of the screen width on 720p, just right for mobile viewing.
14. The canvas must be moveable; don't cram the entire image into one fixed shot. User feedback: "It doesn't have to draw the whole canvas." The correct approach is to render a larger canvas (2x) and frame only the current element to full screen in each scene. After drawing, pan the camera to the next element, and finally zoom out to the full image for the ending. A fixed, full-canvas view from start to finish means each element only takes up 10% of the screen, making details unreadable.
15. The pen tip MUST follow the content. The upstream contour-wipe erases the entire area from bottom to top while the pen tip scans horizontally in a disconnect → viewers see "the hand moving but nothing being drawn." Change to a snake-like brush where the color appears wherever the pen sweeps.
16. Write text when explaining key nodes. Drawing only objects without writing makes it hard for viewers to remember "what year, what place." Bake place names/years into the base image using KaiTi font (add_handwriting.py), then have the renderer write them out column by column in textwrite mode—the pen advances along the strokes, characters appear one by one, making it the same action as "handwriting."
17. After enlarging the canvas, two things are mandatory to avoid slowness or blurriness: ① Pre-enlarge the source image to the canvas size using INTER_CUBIC (the renderer's internal resize is INTER_AREA, which is equivalent to nearest-neighbor when enlarging, turning lines into blocks); ② Change the inking process to operate locally on the bounding box (the upstream creates a new full-canvas mask for each segment, which is 4x slower on a 2560 canvas).
18. The video before audio mixing MUST have 3-7 seconds reserved for the final zoom out. The last scene should NOT be scheduled to the very last millisecond of the narration. Otherwise, there is no time for the "show the full image" closing action.
19. The framing box MUST use the "real ink bounding box," NOT the annotation region. The region is a hand-drawn box, generally larger than the graphic, and in dense images, it may span 2-3 neighbors. Using it directly for the zoom results in: "blank space in the center of the frame, the graphic squeezed into a corner, and neighbors cut in half." The renderer has built-in functionality: inside set_target(element, allowed), use ink_pixels & allowed to find the bbox and then expand it. Exception: text scenes do NOT use bbox—label text is often wider than the region, and applying bbox will clip the whole sentence outside the frame (a real-world example: "Conversational AI" went off-screen).
20. Scene transitions must first pan the camera, then start drawing. At the beginning of each scene, idle for about 5 frames (0.22s). Otherwise, the pen starts drawing before the camera has arrived, painting outside the visible area, leading to "the hand is moving but nothing appears."
21. --zoom-fill should be set based on the base image's density, use 0.86 for sparse images, 0.70 for dense timelines. A single parameter for all situations will cause failures on both ends.
22. Accept that vertically long elements won't fill the screen. In a 16:9 frame, a vertical element can only occupy 60–70% of the height and 25% of the width—this is a geometric constraint. For more screen coverage, draw the element wider or output in portrait mode.
23. Generate the entire voiceover at once. NEVER generate sentence by sentence and concatenate. Concatenating different segments results in inconsistent speed/pauses across segments and inevitable timeline misalignment. Users have described it as "nonsense." Jimeng can handle 1500+ characters in one go (≈2 minutes), feed the entire script at once.
24. Timelines come from the voiceover's own transcription timestamps, NOT from silence detection. Pauses ≠ sentence boundaries. If the number of sentences doesn't match the number of pauses, you can only average the duration by character count, leading to incorrect subtitles. Use asr_align.py (ffmpeg → faster-whisper word_timestamps → character-level timeline).
25. If the voiceover is changed, ALL subtitles and annotation durations MUST be re-generated. The subtitle timestamps are a property of that specific audio, not the video. Old subtitles with new audio is equivalent to not doing it at all.
26. Before delivery, verify the script vs. transcription using difflib. If the similarity is <0.8, or if entire sentences are missing or duplicated, the audio has issues and must be regenerated. Word homophone misrecognition (e.g., '弘/红', '尊/遵', '武/吳') is a normal ASR phenomenon, not a reading error.
27. After voiceover generation, a silence check is mandatory. Feeding a whole script to TTS doesn't mean the platform doesn't internally split it: Jimeng inserts -120dB absolute silence between blocks (measured 4.00s / 2.60s in practice). If the silence ratio is >20% or there are pauses >1.2s, first compress them with tighten_audio.py, then proceed with timeline alignment.
28. Subtitle start points MUST snap to the actual voice frame. When ASR encounters long silences, it often marks the start of the next sentence at the beginning of the silence. Viewers will "see the subtitle and wait for the sound." asr_align.py defaults to snap_to_voice to fix this; do NOT turn it off.

Directory Specification (Mandatory)

D:/04-Project-Tests/<YYMMDD>-<Project-Abbreviation>/
  00-Resources/          Project resources and source code notes
  01-Materials/          Original AI-generated images, hand assets
  02-Subtitles-Voiceover/SRT, voiceover mp3, duration table
  03-Annotations/        <ImageName>.png + <ImageName>.annotation.json (must be same name and directory, required by the preview station)
  04-Render-Output/      Per-scene silent.mp4, final composite mp4
  05-Recording-Screenshots/ Recording mp4 + key screenshots png
  06-Docs/               Scripts, operation log, acceptance checklist

Prerequisites

git clone https://github.com/geeklee/srt-whiteboard-animation.git
cd srt-whiteboard-animation && python scripts/prepare_env.py   # Creates .venv with opencv/numpy/av

All subsequent rendering commands must use srt-whiteboard-animation/.venv/Scripts/python.exe.

If you get the error _lay_ink() takes 6 positional arguments but 7 were given, it's a known bug: In render_stream_whiteboard.py, the line self._lay_ink(writer, ink_frames, [], set(), None, allowed) has an extra None which should be deleted. (scripts/check_bug_fix.py can auto-detect and fix this).

Workflow (7 Steps: Storyboard → Image Generation → Spatial Board → Annotation → Voiceover Alignment → Rendering → Subtitles)

Phase 1 · Script and Storyboard (Confirm before proceeding)

1. Write 6–10 sentences of narration, each 6–10 seconds long, total duration 40–90 seconds.
2. Parse the storyboard (optional, can be done manually):

   <ENV_PY> srt-whiteboard-animation/scripts/parse_srt.py subtitles.srt --target-sec 30 --min-sec 25 --max-sec 35
   

3. Output a storyboard table (Scene / Core Idea / Visual Subject / Subtitle Text), then wait for user confirmation.

Phase 2 · AI-Generated Thematic Hand-Drawn Images

For writing the image generation prompt, follow the project's "Unified Image Generation Visual Specification". Every prompt MUST include:

• Minimalist hand-drawn line art / pure sketch style / Notion-style restrained doodle, 16:9, plenty of white space, elements non-overlapping
• Background: warm cream #F5EBD7, lines: dark gray #333333, only small accents of red/orange/blue
• Absolutely NO text (Chinese characters/letters/numbers/labels/place names/titles/watermarks), NO realism/3D/gradients/high saturation

Use ImageGen or Jimeng image to generate. Each image corresponds to the core meaning of one scene. Elements must be separable by rectangular regions.

After generation, three things are mandatory:

1. Remove watermarks: AI images often have AI生成 WORKBUDDY in the bottom right. Remove it with cv2.inpaint.
2. Convert to 16:9: Generated images are often 3:2. Pad left and right with cream paper color, then scale to 1280×720 (do NOT center-crop, as it will remove the main subject).
3. Clean: Remove paper texture, keep lines and colored accents.

   <ENV_PY> scripts/clean_source.py original_image.png 03-Annotations/main_image.png
   

   Principle: paper = large kernel median filter, lines = gray < paper-18, color = saturation>70 and gray<190. Keep only lines|color, fill the rest with paper base. Force 8px border to paper base to remove edge residues.

Phase 2.5 · Spatial Narrative Board (Determines if the final video is "messy"; MUST be drawn first)

Before writing the image generation prompt, first use text to define the main story arc. Two categories:

Theme	Main Arc	Layout Principles
Historical route / Battle / Migration	Real geographic route	Start at the real location; if the route goes "West→North→East", lay out the nodes faithfully; node x/y must match the real direction
Technology history / Company history	A single horizontal timeline	Track spans the full width, nodes are equidistant, icons alternate top and bottom (top→bottom→top), avoid piling them all on one side

Hard Constraints:

• Only ONE main arc allowed. Do not insert "supplementary explanation" type elements scattered around the arc.
• Elements must not overlap, and each must be contained within a rectangle that does not cross the main arc (otherwise regions will carve each other out).
• The main arc must span the full width: white space should be evenly distributed on both sides of the arc. Avoid a situation where everything is squeezed into the bottom right with a huge empty area on the top left. In practice, the southeast starting point of the Long March was in the bottom right, so the first 20 seconds had everything stacked in the bottom right and the top left empty—this is a composition problem that must be fixed in the image generation prompt by moving the start point towards the center and letting the route fill the canvas.

In the image generation prompt, specify the position of each element on the canvas ("a small red flag in the bottom right" or "a chain bridge slightly to the center-left"). Don't just write "draw a bridge"—if the position isn't specified, the AI's composition will be unusable.

Phase 2.6 · Two Base Image Paradigms (Tested parameters; copy directly)

Whether the final video "fills the screen" and "the camera can move" is determined at the image generation stage. Two paradigms, each with its own set of parameters. Don't mix them:

A. Geographic Route Type (History/Battle/Migration) — Representative: Red Army Long March Route

Item	Value	Rationale
Composition	Start at the real location, route West→North→East spanning the full width	Wrong composition cannot be saved by good pacing
Number of nodes	6–8 stations	One station = 2 scenes (draw object + write text) → 12–16 elements
Camera --zoom-fill	0.86	More white space between elements; needs to push in close to avoid empty frames
Density	Elements can be large or small (a bridge is 3x wider than a flag, which is fine)	The route itself has elements of different scales

B. Horizontal Timeline Type (Technology History/Company History/Product Iterations) — Representative: History of AI Agents (The version the user said "the effect was very good." Copy these parameters directly.)

Item	Value	Rationale
Track	One S-shaped/wavy main track spanning the full width, 8 nodes equidistant	Equidistant → constant camera pan speed, looks steady
Icons	Alternate top and bottom (top→bottom→top…), don't put them all in one row	One row would only use half the screen height, wasting vertical space
Labels	Hang on the same side as the icon, but further out: top node's label is above, bottom node's label is below	So the label and icon are on the same screen; the camera doesn't need to pan away to write the text
Font size	Main name 36–40px KaiTi, year 0.6x below	When the camera pushes in close, it's ≈ 60–80px on screen, readable on mobile
Camera --zoom-fill	0.70	Nodes are close to each other; pushing in too close would cut neighboring nodes in half
One station, two scenes ratio	Drawing icon 0.60 / Writing "name + year" 0.40	Must allocate enough time to write the label; otherwise, the text isn't finished before the scene changes
Timeline itself	Draw it between two adjacent stations on the same scene as the icon of that station	Otherwise, the entire axis would appear at once, breaking the rhythm

Three hard notes for Type B:

1. Text scenes should NOT use the ink bounding box (see Exception to Hard Rule 19). Label text is often wider than the region; using bbox will clip the entire sentence outside the frame.
2. The label's region must be tall enough for "main name + year" as two lines, give 80–90px height. Only giving height for one line of the main name will cut off the year.
3. The ending MUST zoom out to the full axis. The narrative value of a timeline is its "overall span." If you don't zoom out in the last 3-5 seconds, it's as if you didn't draw it at all.

Baseline test parameters (1280×720 / 24fps): 8 nodes × 2 scenes = 16 elements, final video 43.5s, rendering 33s, --zoom-fill 0.70 --zoom-max 7 --hand-height 350.

Phase 3 · Pixel-Level Annotation (Most error-prone; must be verified)

Measure coordinates first; don't estimate by eye. Real-world lesson: Confusing the preview display size of 640×360 for the actual 1280×720 coordinates caused two rounds of fully incorrect coordinates. The correct approach is to use color/line detection to get the real bbox:

<ENV_PY> scripts/detect_marks.py 03-Annotations/main_image.png label   # Connected domain bbox for red/orange/blue accents
<ENV_PY> scripts/detect_lines.py 03-Annotations/main_image.png label   # Bbox for dark gray line clusters

For manual calibration, overlay a ruler: scripts/grid_ruler.py main_image.png with_grid.png (100px grid, marks real pixels).

Region Coverage Self-Check (Determines if the final video is complete):

• The x-intervals of all regions must form an unbroken band (overlap allowed, no gaps allowed)
• Pervasive subjects (route, timeline) MUST be fully covered by a few specific regions, otherwise the final video will have broken lines
• The last element picks a small area separately (see Hard Rule 3)

Write 03-Annotations/<name>.annotation.json:

{
  "sceneId": "scene-01",
  "canvas": { "width": 1280, "height": 720 },
  "storyBasis": "Summary of this scene's subtitle event",
  "sceneDurationMs": 56000,
  "elements": [
    {
      "id": "ruijin", "label": "Departing·Ruijin", "sequence": 1,
      "narrativeRole": "Start of the Long March", "subtitle": "Corresponding subtitle text",
      "type": "point",
      "region": { "x": 265, "y": 515, "width": 145, "height": 100 },
      "reveal": { "direction": "left_to_right", "startMs": 0, "durationMs": 8100,
                  "maskPaddingPx": 16, "protectedRegions": [] },
      "handPath": { "start": [275, 570], "end": [400, 575], "easing": "easeInOut" }
    }
  ]
}

• canvas MUST equal the real pixels of the source image; region must be integer pixels and within the canvas
• sequence is based on narrative order, not visual left-to-right
• In stream rendering mode, direction/handPath are only for preview station demonstration and do not affect the final video's pen strokes

Generate verification image and check frame by frame:

<ENV_PY> scripts/region_sheet.py main_image.png annotation.json 05-Recording-Screenshots/verification-region.png

Each box should exactly frame the target element; if it frames empty space or another element, the coordinates are wrong. Go back to the previous step.

Phase 4 · Voiceover and Timeline Alignment

① Voiceover: Generate the entire script at once. NEVER sentence by sentence (Rework Lesson from 2026-11-05)

"<TTS_PYTHON>  # Any Python 3.9+ with requests installed" \
  "jimeng-suite/scripts/jimeng_cli.py" audio \
  --text "The entire script, one paragraph, use periods to separate sentences" \
  --model "SeedAudio 1.0" --submode "All-in-one Voiceover" --voice "狒狒_讲解" \
  --out "xxx-voice.mp3" --outdir "<Project>/02-Subtitles-Voiceover" --timeout 900

• Seed TTS + Text Reading was taken offline on 2026-10-24, do NOT use it
• Feed the entire text via --text at once. It can handle 1500+ characters in one go (≈2 minutes of audio). Stop splitting it into many 3-second segments: In practice, splitting a 56-second script into 12 segments, generating them separately, and then concatenating them resulted in inconsistent speed/pauses/emphasis across segments, sounding like "nonsense." Also, the timeline and subtitles inevitably misaligned after concatenation—users directly judged it as unqualified.
• Only split by narrative paragraph (one paragraph = one group of sentences). A paragraph should be 15–40 seconds. The whole video should be 1–3 paragraphs. If you can do it in one, don't use two.
• Content review will block it: Military/violence/harm-related words (bloody battle, annihilate, defeat, enemy) will return a JimengContentRejectedError. Rewrite them with neutral descriptions ("paid a heavy price," "under fire blockade"). Writing years in Chinese numerals (e.g., 一九三四年) is more stable.
• A 99-second script was measured to be 62 seconds of output (1.19MB / 99.05s).

② Silence Check and Rhythm Compression (Mandatory, v5 Rework Lesson)

The phrase "feed the whole script to TTS at once" does NOT mean the platform synthesized it as one continuous segment: Jimeng will internally split a long text into chunks and then concatenate them, inserting -120dB absolute silence between blocks. In practice, a Long March v4 test with 317 characters / 99.05s had 31 pauses ≥0.4s totaling 27.2s (27.5%) . Two of these were pure digital zero-samples (4.00s @14.05, 2.60s @70.60). Natural pauses have background noise, about -50 to -60dB, never -120dB. The actual effective speech was only 71.85s (4.41 characters/sec, which is not slow). The feeling of "slow and weird" was entirely due to the blanks.

# 1) Check: Any silence >1.2s must be handled.
ffmpeg -hide_banner -i xxx-voice.mp3 -af "silencedetect=noise=-38dB:d=0.35" -f null - 2>&1 | grep silence_start

# 2) Compress: Preserve natural pauses at different levels; do not modify any speech content.
<ENV_PY> scripts/tighten_audio.py 02-Subtitles-Voiceover/xxx-voice.mp3 \
  02-Subtitles-Voiceover/xxx-voice-tight.mp3 --report 06-Docs/_tighten.json

tighten_audio.py: Uses a 50ms RMS window; values <-40dB are considered silence. Pauses ≥1.0s are kept at 0.55s, pauses 0.35–1.0s are kept at 0.32s, and initial silence is kept at 0.12s. The audio is then re-assembled from samples. The v4→v5 test showed a reduction from 99.05s to 83.15s, with silence ratio dropping from 27.8% to 14.0%, and no pause longer than 0.7s.

Compression changes the audio length. Therefore, ASR transcription, timing, SRT, ASS, annotation durations, and render total-ms MUST ALL BE REPROCESSED based on the compressed audio (Hard Rule 25). The --reuse-asr flag is only for adjusting subtitle line breaks. Once the audio changes, never reuse the old dump.

③ Timeline: Use the voiceover's own transcription timestamps; do NOT guess based on silence detection

<ENV_PY> scripts/asr_align.py 02-Subtitles-Voiceover/xxx-voice.mp3 06-Docs/script.txt \
  --model small \
  --out-timing 03-Annotations/xxx.annotation.timing.json \
  --out-srt 02-Subtitles-Voiceover/xxx.srt --max-chars 18 \
  --dump-asr 06-Docs/_asr.json

What asr_align.py does: ffmpeg converts to 16k mono → faster-whisper (word_timestamps=True) → produces a character-level timeline → locates each sentence of the script within the full transcription (exact match → first 8 char anchor → fallback) → outputs timing.json (compatible with segments[i/start/end/text]) and single-line SRT.

Why not use silencedetect: Silence only tells you "there is a pause here." Breath pauses, emphasis pauses all cause silence. If the number of sentences and number of pauses don't match, you can only average the duration by character count, guaranteeing misaligned subtitles.

asr_align.py also does sentence boundary snapping (snap_to_voice by default): When faster-whisper encounters long silences, it often marks the start of the next sentence at the beginning of the silence. This causes the subtitle to appear early, and viewers stare at the text waiting for the sound (in v4, this was 4 seconds too early). The script finds the true voice start point in a 20ms grid and corrects it. Use --no-snap to disable.

④ MUST verify audio content (the only hard standard for judging "nonsense"):

<ENV_PY> -c "..."   # difflib comparison: script vs full text from _asr.json

Similarity should be >0.8. For Chinese proper nouns, the small model will only make homophone misrecognitions (弘/红, 尊/遵, 武/吳, 锁/索). If the structure (sentence order, sentence length) is the same, the reading is correct. If entire sentences are missing, duplicated, or out of order, the audio itself is problematic and must be regenerated.

⑤ Three rules for subtitle line breaks (all within asr_align.py::split_subs):

• First, split by punctuation into short clauses. Only merge with the next short clause if the current block < limit×0.55—otherwise you get "踏上了举世闻名的长征之路那一年" (spanning across clauses).
• After merging, must not exceed limit (don't leave +3 tolerance; a 20-character split could result in a single-character orphan like "只" onto a new line).
• Only if a single block is still too long, split evenly. If the tail is ≤4 characters, merge it into the previous line.

⑥ Ending White Space (Implementation of Hard Rule 18): Add "tail": 0.30 to the last station. The last 30% of that segment will have no elements scheduled. The renderer will use this time for zooming out and freezing. In practice, a 99-second video had 4 seconds reserved, just enough to see the full image.

⑦ Scene Duration Allocation: One element per sentence (or per action within a sentence). durationMs is taken directly from that sentence's duration. Within a segment, use ratio to split "drawing the object / writing the text". See scripts/make_annotation_v3.py.

Phase 5 · Rendering, Composition, Verification

<ENV_PY> srt-whiteboard-animation/scripts/render_stream_whiteboard.py \
  03-Annotations/main_image.png 03-Annotations/main_image.annotation.json 04-Render-Output/xxx-silent.mp4 \
  01-Materials/hand-55pct.png \
  --cap-long-edge 1280 --fps 24 --total-ms <voiceover_duration_ms>

• If --cap-long-edge equals the source image's long edge, it will not be scaled. Adjusting it smaller speeds up previews.
• --total-ms is the voiceover duration in milliseconds, ensuring the video does not end before the voiceover.
• The last line of output is OUTPUT=<path>.

Compose the audio track:

ffmpeg -y -i xxx-silent.mp4 -i voiceover.mp3 -c:v copy -c:a aac -b:a 192k -shortest output_video.mp4

Three checks are mandatory (sample one frame each):

1. Opening: clean cream-colored background, no premature lines visible
2. Middle section: continuous lines, correct colors (red/orange/blue all present)
3. End: full image is complete, no broken lines, no large blank areas

If the duration deviation is >1s, adjust --total-ms or fine-tune individual durationMs values and re-render.

Phase 5.5 · Camera Movement and Text (v3 Core; Determines if it "looks like drawing")

The upstream renderer draws on one fixed canvas from start to finish. To make the video feel like camera work, add a render_camera_whiteboard.py subclass. No need to modify the repository code (the scripts/render_camera_whiteboard.py is ready to use):

<ENV_PY> scripts/render_camera_whiteboard.py 03-Annotations/main_image.png 03-Annotations/main_image.annotation.json \
  04-Render-Output/xxx-silent.mp4 01-Materials/hand.png \
  --canvas-long-edge 2560 --frame 1280x720 --fps 24 --grid-edge 20 \
  --zoom-fill 0.86 --zoom-max 7 --hand-height 350 --total-ms 56064

It does six things:

Mechanism	Implementation	Why
Camera framing	Renders at 2560×1440, crops per viewport per frame, scales to 1280×720	Elements can be drawn "full-screen," then the camera pans away
Ink bounding box framing	Target box = real ink bbox within the scene's allowed mask, NOT the region	Regions are often larger than the graphic or even span neighbors; using the region directly for framing → blank space in the middle of the picture, elements pushed to the edges (v3 first version failed here)
Camera follow	Exponential smoothing between adjacent scenes, tau 0.20/0.30 for text scenes pad 0.34, object scenes pad 0.05	Naturally creates panning effects; no need for explicit transition frames
Scene change buffer	Idle for 0.22s (about 5 frames) at the beginning of each scene before drawing	The camera takes 0.3–0.6s to pan from the last scene. If the pen draws during this time, it will draw "outside the frame," which is the main cause of "the hand is moving but nothing appears"
Ending full image	After the last scene, set the target to full canvas and add frames until --total-ms	"Finally, show the entire image"
Hand fixed to screen coordinates	The hand is not drawn onto the canvas; it's superimposed on the output frame (--hand-height 350)	Otherwise, when the camera pushes in, the hand fills the entire screen. In practice, 430 was still too large and covered the text being written

--zoom-fill must be set based on the base image's density (The same parameter cannot work for all scenarios):

Base Image Type	zoom-fill	Rationale
Sparse (route maps/flowcharts, large white spaces between elements)	0.86	Can safely push in close, making the frame look full
Dense (horizontal timelines, elements closely adjacent)	0.70	Pushing in too close will cut neighboring elements in half, looking cluttered

The hard geometric constraint of 16:9: Vertically long elements (like a chess piece, a phone) are limited by the frame's height. They can only occupy 60–70% of the height and about 25% of the width. This is geometry, not a bug. To make them fill the screen more, draw the element wider or switch to a portrait output.

Writing Text (to write place names/years):

# 1) First, bake the labels into the base image (coordinates are from 1280×720 annotation; the script will automatically lay out two lines based on font size)
<ENV_PY> scripts/add_handwriting.py base_image_2x.png 03-Annotations/main_image-v3.png 02-Subtitles-Voiceover/labels.json
# 2) In the annotation, set `"inkPath": "textwrite"` for this scene, where the region is the text box.

textwrite advances column by column from right. The pen tip is at the leading edge of the reveal, guaranteeing the text is fully written (skeleton tracking will miss thin strokes; don't use it). Wait for about 30% of the frames to let the camera position itself, then start writing—otherwise, text will be written outside the frame.

Label layout: Main name 34–40px KaiTi font (C:/Windows/Fonts/simkai.ttf) + year at 0.6x font size below, color (58,56,52) same color scheme as the line art but slightly lighter.

Scene Splitting: One station = two scenes. First, draw the object (0.6 of the time). Immediately followed by writing the text (0.4 of the time). The framing box for the writing scene is the text's own box, so the camera will push in slightly, looking like "the pen moves to the side to write."

Phase 6 · Subtitle Burning (SRT → ASS → Burn into Video)

The subtitle timeline should NOT be marked separately. Directly use the .timing.json from Phase 4—it is already aligned sentence-by-sentence with the voiceover. The moment the subtitle changes should be the moment the pen stops. This is "rhythm alignment."

# 1) timing.json -> SRT (**Single line**, automatically splits long text into multiple consecutive subtitles)
<ENV_PY> scripts/make_srt.py 03-Annotations/main_image.annotation.timing.json 02-Subtitles-Voiceover/x.srt \
  --max-chars 14

# 2) SRT -> ASS (Explicit PlayRes + \pos for bottom-edge anchoring; font size/position are real pixels)
<ENV_PY> scripts/srt2ass.py 02-Subtitles-Voiceover/x.srt 02-Subtitles-Voiceover/x.ass \
  --video-w 1280 --video-h 720 --font simkai --font-size 46 --margin-v 32

Parameters for srt2ass.py:

• --video-w / --video-h: Video actual resolution (PlayRes)
• --font: Font name (must be installed in the system)
• --font-size: Font size in pixels
• --margin-v: Pixel distance from the bottom edge
• --single-line: Force single line mode (default on)
• The output ASS format: Dialogue: 0,0:00:01.00,0:00:05.00,Default,,0,0,0,,{\an2\pos(640,688)}Single line of text…

Burn subtitles:

ffmpeg -y -i output_video.mp4 -vf "ass=02-Subtitles-Voiceover/x.ass" -c:a copy final_output.mp4

Three checks for the final video:

1. Subtitle lines do not exceed one line; long sentences must be properly split (no overflow)
2. Subtitle appearance/disappearance is synchronized with the voiceover (not too early, not too late)
3. The bottom of the screen is not blocked by subtitles; the minimum distance from the bottom edge is 32px.

Phase 7 · Screen Recording and Screenshots

# 1) Screen recording (record the entire process using a designated monitor or window)
ffmpeg -f gdigrab -framerate 15 -i desktop -t 120 -c:v libx264 -preset ultrafast -crf 28 \
  05-Recording-Screenshots/operation_recording.mp4
# 2) Key screenshots
# - Source image
# - Verification-annotation.png
# - Render console
# - The final video's frame 1 (full cream background), middle frame, and last frame (full image)

Note: The recording duration should be long enough to cover the entire operation, from Phase 3 annotation to Phase 6 subtitle burning.

Quality Acceptance Checklist (All items must be checked)

#	Check Item	Standard	Failure Action
1	Source image has no watermark	Bottom right area clean	cv2.inpaint to remove
2	Aspect ratio is 16:9	1280×720	Padding + scaling
3	Paper texture removed	clean_source.py run	Re-run cleaning
4	Annotation coordinates are accurate	Each box exactly frames the target element	Re-measure coordinates
5	Last element region is not the full canvas	Small area only	Re-select region
6	Voiceover generated at once, not concatenated	Single file, should be 1500+ chars or ≈2 minutes	Re-generate
7	Silence check passed, ratio <20% after compression	No pauses >1.2s	Re-compress or re-generate
8	Script matches transcription (similarity >0.8)	No missing/duplicated sentences	Regenerate audio
9	Timeline uses transcription timestamps, not silence detection	asr_align.py run	Re-run timeline
10	Subtitle start points snap to voice	snap_to_voice enabled	Re-run with default settings
11	Subtitle single line ≤14 chars, font 46px	No overflow, readable	Re-edit subtitle text
12	Camera framing uses ink bounding box	Framing not using region	Check render script parameters
13	Scene transition buffer 0.22s	Pen doesn't draw outside frame before camera moves	Check render script parameters
14	Ending has 3-7s for full image zoom out	tail parameter set	Re-annotate timeline
15	--zoom-fill matches base image type	Sparse 0.86, dense 0.70	Check render script parameters
16	Screen recording and screenshots present	Recording file + key screenshots	Re-record / re-capture
17	Final video total duration matches voiceover	Deviation <1s	Re-render with adjusted --total-ms
18	Subtitles burned correctly, not separate file	Subtitle is part of video	Re-burn with ASS