What does Comic Storytelling Video do?
Turn a chosen topic into a 110–135 second vertical short video: line art is drawn stroke by stroke, dialogue bubbles pop in one by one, with voiceover narration and large captions.
Output: <project>/out/final.mp4 (1080×1920).
The audio is the main thread. If you extract the audio track and listen to it alone, it must be a coherent story; the visuals are there to enhance, not as a prerequisite for understanding.
Five Steps
① Define the Story → <project>/story.json
Create the directory and copy the template:
mkdir -p <project>/raw && cp templates/comic/story.example.json <project>/story.json
First, fill in the three header fields. If you can't fill these in, don't write the panels:
"story_line": "Who / wants to do what / is stuck where / outcome, in one sentence",
"question": "The question thrown out at the beginning",
"answer": "The answer given in the last panel that resolves it"
The question should pick a point that "doesn't make sense," not a "what is this." The answer should not be a timeline.
Then write each panel's narration (voiceover) and bubbles[].text (dialogue). Narration should be 440–540 characters, including punctuation, calculated at 4.0 characters per second.
python scripts/pipeline.py <project> --stage script # But don't proceed further
① Verification: Three header fields complete → character count within range → each panel has narration → bubbles don't exceed panel boundaries → function exists in charts.py.
② Generate Images → <project>/raw/<id>.png
Write all prompts first, then generate images in one batch. Generating one image at a time and tweaking prompts mid-way will drag the process to one or two hours and cause art style drift.
python scripts/comic_gen.py <project> --list # Print prompts not yet generated; skip those already done
Generate each image according to the printed prompts, saving them into raw/. Must use image-to-image generation with a reference image from assets/reference/01-style-anchor.png, with similarity set to low.
python scripts/contact_sheet.py <project> # Stitch into an overview sheet, take a look
python scripts/pipeline.py <project> --stage image
② Verification: On the overview sheet, only check two things—are the required characters present and is there any large area painted black or blank? Ignore everything else (random text on the image, half-head height differences, messy lines, a few dozen pixels of tail offset) and proceed directly.
③ Produce → final.mp4
python scripts/comic_run.py <project>
Runs sequentially: voiceover → bubble layers / infographics → clean end credits → stroke-by-stroke rendering → vertical compositing → burn subtitles.
To re-run a single step, use the corresponding comic_*.py script.
④ Verification
python scripts/verify_media.py --file <project>/out/final.mp4 --expect <seconds>
ffmpeg -y -ss 3 -i <project>/out/final.mp4 -vframes 1 <project>/check/03.png
Extract two frames (at second 3, and at panel total length × 0.75) and compare: Is the image moving? Is the background color correct? Are subtitles out of frame? Then extract the audio track and listen with eyes closed: if you can say "who wanted what, what events happened, and how it ended," it passes.
⑤ Delivery
<project>/out/final.mp4. For a flat-screen version, use out/story_flat.mp4.
Ten Hard Rules
The first three govern the story; violating them means "listening without understanding anything." The last seven govern production; violating them means visual or audio errors.
1. Fill in story_line / question / answer first; if you can't, don't write panels.
Missing these three lines, writing eight panels only to find they're disjoint is the most expensive kind of rework.
2. The skeleton is chosen based on the relationship between protagonist and obstacle; don't use the same panel table for all topics.
| Protagonist vs Obstacle | Skeleton |
|---|---|
| One specific opponent, back-and-forth | Attack → Counter → Escalate → Counter → Deadliest move → Counter |
| The protagonist's own path is blocked | Old craft is dying → Try several things, none work → Something unexpected appears → Go all in → Succeed → Finally realize what you're really selling |
| An inexplicable phenomenon traced backwards | Anomalous event → Find that person → Their difficulty at the time → How they solved it |
| Reached the top, then brought down by something trivial | Peak → A crack → Nobody took it seriously → Bet wrong → Collapse |
The number of panels follows the story, anywhere from 5 to 9.
3. Don't number parallel events as rounds. If you replace "First round" with "First time" and the story loses nothing, then it wasn't a round—it was a label pasted on. Delete it. Similarly, don't use commentator jargon like "escalation," "peak," "showdown," "skeleton in the closet"—the audience hears code words.
4. The narration must be a complete, coherent story that can be understood; dialogue must be spoken aloud. "He replied," "Approved two characters," "A single sentence woke him up"—the listener only knows someone spoke, not what was said. It's not an error if the bubble repeats the narration; subtitles are inherently a repetition of the audio.
5. The generative model only draws line art; text is overlaid by code. Write "don't draw any text" in the prompt as a precaution, not as an image verification criterion.
6. Bubbles don't go into the engine. The engine would binarize colors into black blocks. Bubbles are overlaid at timing during compositing for better rhythm control.
7. Don't pass element_plan.
If passed, each object stops drawing after 1.6 seconds, filling the entire panel in two seconds. Without it, the render budget is distributed by √area, keeping total duration constant.
8. Must use image-to-image generation, with reference similarity set to low (0.3–0.5). High similarity causes the model to copy the reference image's pose, ignoring the action you wrote.
9. Characters go into characters, referenced in prompts with {name}; any two characters must differ by at least three attributes.
Specify headgear, beard, clothing, and body type. Give the protagonist distinctive headgear (soft cloth headwrap / no crown); leave square crowns, tiaras, and helmets for opponents and soldiers.
10. Use colorkey + overlay for background color; burn subtitles with an ASS file and declare PlayResX/Y.
Using blend=all_mode=multiply turns the background into fluorescent green; SRT subtitles passed to libass land in a 384×288 coordinate system, causing text to fly off screen.
Which Reference to Check
| Situation | Read |
|---|---|
| Don't know how to turn a topic into a story, what to put in each panel, or how to end | references/story.md |
| Writing prompts, defining characters, bubble layout, TTS, rendering and compositing parameters | references/production.md |
| Image / audio / duration errors | references/troubleshooting.md |
| Specific prompt templates and negative constraints | references/prompt-recipes.md |
| Voiceover syllables / voice tones | references/voice-design.md |
| edge-tts returns 403 | references/apiz-tts-fallback.md |
Directory Structure
<project>/
├── story.json Only configuration: title, three-line story, each panel's narration/bubble/prompt
├── charts.py Drawing functions for infographic panels (only needed if there are chart panels)
├── raw/ Original images from generative model ← You place them here
├── boards/ bubbles/ charts/ audio/ out/ ← Script outputs
└── subs.ass
First Time Use
bash scripts/setup.sh # macOS / Linux / Git Bash
powershell -ExecutionPolicy Bypass -File scripts/setup.ps1 # Windows
python scripts/self_test.py # 6 self-checks, run once