H3 Video: Full AI Short Drama Pipeline
This pipeline is distilled from rework experience on several real productions. Core idea: Plan first, confirm at each checkpoint, and if something goes wrong, only fix that layer.
Overall Flow (In Order, No Skipping)
| Step | What to Do | Checkpoint |
|---|---|---|
| 0 | Research: Study top-tier similar works and others' approaches, note down techniques | — |
| 1 | Write plan: Tone & style, dialogue table, segment breakdown, generation checklist (see templates/plan_template.md) | User confirms plan |
| 2 | Voiceover: Narration and off-screen voice use TTS separately, measure real duration | User listens and confirms voice timbre |
| 3 | Character Master: One full-body front view per character, shot in the story's world | User confirms master |
| 4 | First Frame: One per segment, only at the very start of the action | Do not generate video before user confirms first frame |
| 5 | Generate: Use fastest setting for draft first | — |
| 6 | Per-sentence speaker check, gaze check, seam check | If wrong, redo only that segment |
| 7 | Upscale, subtitles, loudness, cover | User accepts |
| 8 | Review: Score each item in the "methods to verify" list from the plan, document lessons in the knowledge base | — |
Seven Hard Rules
- Style must obey the story's tone. Before drawing anything, write the "Tone & Style" section: light source and darkness, color palette, clothing newness/dirt, expressions, background atmosphere. Character masters must be shot in that world, not on a gray studio background. For every image, first ask "Does it belong to this story?" before checking details. A dark story paired with clean, pretty idol faces is the most common failure.
- One generation has only one voice source. Never feed narration or off-screen voice into H3, otherwise characters will lip-sync to the narration—writing "mouth closed" in the prompt won't suppress it. Generate video without dialogue for narration segments; overlay the voice in post.
- Characters who are present but not speaking: Do not assign dialogue during generation; if a speaker needs to be hidden, shoot from behind or over-the-shoulder.
- The dialogue table is the single source of truth. Each line records type (dialogue/off-screen), speaker, who they are looking at, and original text. Budget approximately 3 characters per second; each line must finish within its segment. No subsequent step may alter the dialogue table.
- Gaze: Speakers look at listeners, listeners look at speakers. In prompts, clearly specify who looks at whom for every line.
- If the first frame has errors, redo the first frame—don't try to fix it with prompt tricks. When inspecting, zoom to full resolution to check hands, eyes, mouth, accessories, and stray objects.
- Do not deliver until per-sentence speaker check passes. For each dialogue line's time range, extract a dense frame grid (using
scripts/speaker_grid.py) and answer only a closed question: "Whose mouth is moving: A / B / Both / Neither," then compare with the dialogue table. Sparse sampling of 4 frames may miss issues.
Principles for Generation & Rework
- Draft first, then upscale: Generate at the fastest viewable resolution, then only upscale approved versions.
- Attribute failure first: Determine if the problem is in the prompt, first frame, script, or post-processing, then decide which layer to fix. If multiple segments share the same error, fix the shared reference (master); if only one segment is wrong, fix only that segment; change only one thing at a time.
- Use positive descriptions in prompts, don't pile up "don't..." statements. To avoid something, go back to the input (first frame, master) and fix it.
- Record the first frame, master, seed, and prompt for each segment so you can redo just that one segment.
- First 2-second test: View only the first 2 seconds; the audience should know what they are watching.
Reference Materials
references/pipeline_details_and_lessons.md: Step-by-step practices and pitfalls encountered.references/comfyui_nodes_and_params.md: Qwen-Image master generation, Qwen-Image-Edit image modification, VoxCPM2 voice cloning, H3 Ref2VA node setup and parameters, plus known failure workarounds.references/open_source_pipeline_survey.md: Comparison of 9 open-source AI video projects—what's worth adopting and what's not.templates/plan_template.md: Plan document template including dialogue table and verification table.
Scripts
All scripts connect to ComfyUI via the environment variable COMFY_URL (default http://127.0.0.1:8188). See scripts/README.md for usage.
| Script | Purpose |
|---|---|
t2i_masters.py | Batch generate character masters with Qwen-Image (style prefix + description per character) |
edit_image.py | Qwen-Image-Edit local image modification, keeping everything else unchanged |
voice_design.py | Design character voices per text description with VoxCPM2, output A/B two samples per character |
tts_clone.py | Clone voice with VoxCPM2 using one reference audio, batch output narration |
h3_ref2va.py | Assemble and submit H3 Ref2VA workflow (reference image, reference audio, first frame anchor) |
wait_download.py | Wait for task to complete and download all outputs |
ws_progress.py | View task progress (must use the client_id from submission) |
asr.py | Transcribe with faster-whisper to verify narration is word-perfect |
speaker_grid.py | Output dense frame grid per sentence from dialogue table for speaker check |
upscale_video.py | Upscale video frame-by-frame with RealESRGAN |
post_subtitle_loud.py | Burn subtitles and apply dual-pass loudnorm to -14 LUFS |
make_cover.py | Generate cover in 3:4 and 9:16 formats |