What does Video Generation Desk do?
Does one thing: Makes already finalized visuals move, and leaves a traceable record.
The video model here is responsible for three things: motion, camera movement, and speaking the narration. Character appearance, clothing, scene, color palette, lighting, composition, and on-screen text are locked in at S1/S2. This layer is not allowed to recreate them. Once it starts "improvising," the entire film's consistency breaks.
Narration no longer uses separate voiceover. Dialogue is written into the prompt, spoken directly by the model—this is the biggest difference from traditional workflows and the area that needs the most oversight.
I. Prerequisites for Entry
Four items are required; missing any one means sending back to the previous phase:
| Requirement | Location | What happens if missing |
|---|---|---|
03-generation-checklist.md | Produced by short-video-prompt, and G2 approved | Unclear how many segments to generate or their lengths |
| Original dialogue for each segment | Inside the per-segment prompt in the checklist | The model won't know what to say and will make something up |
| Asset image / first frame | 04-assets/01-asset-images/, 04-assets/02-first-frames/ | Character appearance varies per segment |
KIE_API_KEY | Environment variable | Cannot submit tasks |
No first frame doesn't mean it can't be done: Pure atmosphere, empty shots, and transitions can use text-to-video. But segments with characters must have an image—this is a hard requirement.
Segments without dialogue can skip narration, like empty shots or pure music sections. But if a segment needs speech, the dialogue must be in the prompt—it's the only source of sound now.
II. Three Modes, Mutually Exclusive, Cannot Be Mixed
The platform accepts only one mode per request. Crossing field boundaries switches modes and causes task failure.
| Mode | Required Fields | When to Use | Explicitly Prohibited |
|---|---|---|---|
text Text-to-video | prompt | Empty shots, atmosphere, pure background effects, transitions | Including first_frame_url / last_frame_url |
first First-frame image-to-video | prompt + first_frame_url | Default mode: needs to lock starting composition | Including last_frame_url |
first-last First-and-last-frame | prompt + first frame + last frame | Needs to lock both start and end to ensure smooth continuity between adjacent segments | — |
How to Map "Reference Image" Mode from the Checklist
The checklist's mode column from short-video-prompt might say "First-Last / Reference Image / Text." Current model has no reference image mode—fields like reference_image_urls are not open on this model. So a mapping is needed:
| Checklist Says | Actual Submission Mode | Instructions |
|---|---|---|
| First-last | first-last | Direct pass-through |
| Reference image | first | Use the reference image as the first frame, lock starting composition |
| Text | text | Direct pass-through, but only valid for shots without characters |
This is not a compromise; it's a capability boundary. If a segment truly needs "reference image to lock character + free composition," the only options are to lock the composition along with the first frame or go back to S2 to modify the segment split.
III. Parameter Specifications
| Parameter | Values | Default | Description |
|---|---|---|---|
duration | 4–15 integer seconds | 5 | Over 15: split into multiple shots; under 4: pass to post-production for effects or cropping |
aspect_ratio | 9:16 / 16:9 / 1:1 / 3:4 / 4:3 / 21:9 / adaptive | 9:16 | adaptive only for text-to-video; image-to-video requires a fixed ratio |
resolution | 480p / 720p | 480p | Default controls generation cost; explicitly pass 720p when delivery clarity is needed |
generate_audio | true / false | true | Cannot turn off. The narration is in this audio track; turning it off means no sound |
web_search | true / false | false | Only usable for text-to-video; generally not used |
Three common pitfalls:
- Aspect ratio must match actual output. If the model returns a result with a clearly mismatched ratio or black bars, mark it as quality failure; do not letterbox and deliver as final.
- Audio must be enabled. Narration and ambient sounds are all in this one track. Only use
--no-audiofor pure empty shots that truly don't need sound. durationmust be an integer. Passing 5.5 will be rejected.
IV. Seven-Step Pipeline
① Read checklist → ② Determine mode → ③ Write video prompt → ④ Submit → ⑤ Poll and download → ⑥ Media inspection → ⑦ Frame sampling QA
① Read Checklist, First Determine Feasibility
Check three things for each segment; if mismatched, stop and ask, don't guess:
- Is each segment's
durationbetween 4 and 15? (Split if over) - Does the declared first-frame file actually exist?
- Are all referenced asset IDs present in the asset list?
② Determine Mode
Determine mode per segment based on Section II's table. Try to keep the whole project consistent. Mixing isn't an error, but it makes adjacent segments look worse.
③ Write Video Prompt
Follow the seven-section structure in references/video-prompt.md. Two core principles:
- Only describe "change," don't redescribe "what it is." The first frame already shows the model what the product looks like; describing it again in the prompt makes the model redraw it from the text, instantly losing consistency.
- Copy dialogue verbatim from the script, don't change a single word. The narration is now spoken directly by the model; the original dialogue must be in the prompt. Changing even one word means the subtitles won't match the audio.
④ Submit
python scripts/kie_seedance.py create \
--prompt "…" --mode first \
--first 04-assets/02-first-frames/S01_f01_v1.png \
--duration 6 --out 04-assets/03-clips/S01_v1.mp4 --wait
Immediately write the taskId and fingerprint into the ledger after successful submission, then move on. Write after doing; if interrupted midway, you can't find it back.
④-2 Batch Run for Entire Film (Default 5 Concurrent Workers)
Running one segment at a time for 6 segments takes over 30 minutes, most of it waiting. Batch concurrent running takes about the time of the slowest segment.
# Preview first: inspect the assembled request body
python scripts/kie_seedance.py batch --tasks tasks.json --out-dir 04-assets/03-clips --dry-run
# Confirm then run; default 5 concurrent
python scripts/kie_seedance.py batch --tasks tasks.json --out-dir 04-assets/03-clips
# Reduce if hitting rate limits
python scripts/kie_seedance.py batch --tasks tasks.json --out-dir 04-assets/03-clips --concurrency 2
The task table format follows assets/batch-tasks-example.json: one entry per segment, with mode / first / last / duration / prompt having the same meaning as the single command.
Concurrency isn't always faster; there's a real pitfall: the ledger can get corrupted.
If 5 segments finish concurrently, and each tries to update 00-generation-ledger.md right after finishing, the five writes will overwrite each other, leaving only one segment's record. The correct approach is:
① Finish the whole batch, collect the items from the summary (already sorted by checklist order)
② Write back to the ledger in one go: write all five segments together, not one at a time
③ Immediately verify: the ledger should have as many rows as the checklist has segments
If you prefer to write as you go, reduce concurrency to 1—serial execution with write-as-you-go is the only way to avoid corruption. Make this trade-off yourself, but you cannot concurrently run and write piecemeal.
Rate limiting appears as code != 200 or HTTP 429, not "worse video quality." When you see these errors, reduce concurrency first; don't change the prompt.
Failure doesn't interrupt: If one segment fails, the rest continue. Check failed in the summary. To stop on first error, add --stop-on-error; it stops submitting new tasks but doesn't affect ones already running.
⑤ Poll and Download
--wait keeps running until download completes. You can also split: create to get a taskId → wait --task-id.
Polling interval starts at 6 seconds and gradually increases to 30 seconds, with a 15-minute cap. Timeout doesn't mean failure; check status first before deciding.
⑥ Media Inspection (Script-Checked)
# Segments with dialogue: audio track must exist
python scripts/inspect_clip.py --file 04-assets/03-clips/S01_v1.mp4 \
--ratio 9:16 --duration 6 --expect-audio
# Segments without dialogue (empty shots): should have no audio track
python scripts/inspect_clip.py --file S02_v1.mp4 --no-expect-audio
Checks five objective facts: file exists, size is normal, duration is correct, aspect ratio is correct, audio track exists. Passing only means "a playable file was downloaded," not that the visuals are usable or the dialogue is correct.
The audio check is now more important than before: the narration is in this audio track. No audio track = no sound for this segment = waste. An existing audio track but wrong words can't be caught by the script—only by listening.
⑦ Frame Sampling QA + Audio Listening (Human-Checked)
Visuals: Sample at least three points (start, middle, end). Check against the seven criteria in references/clip-review.md.
Audio: Since the model now speaks the narration, every segment with dialogue must be listened to against the original script, checking three things:
| What to Listen For | Criteria |
|---|---|
| Missing sentences | As many sentences as in the original script, you should hear that many |
| Mispronounced words | Are keywords, brand names, numbers correct? |
| Unwanted sounds | Music, other voices, noise that shouldn't be there |
When what's heard doesn't match the script, don't settle. Two options: regenerate this segment, or modify subtitles to match the actual spoken content. The one thing you cannot do is keep the original subtitles verbatim—if the audience hears one thing and reads another, it's more damaging than a mispronunciation itself.
This step must be listened to. For ambiguous keywords, repeat and transcribe at least three times for comparison, then do manual final review per sentence and write the conclusion into 04-assets/00-dialogue-final.json. See references/clip-review.md.
V. Three Layers of Status, Don't Mix Them Up
| Status | Who Judges | Meaning |
|---|---|---|
| Media passed | Script | File complete, parameters correct, audio track exists |
| Visuals passed | Human watching | Character unchanged, no distortion, can connect to next segment |
| Dialogue passed | Human listening | Dialogue fully spoken, keywords correctly pronounced |
Only when all three pass can the status be written as "Quality Passed."
Treating "download complete" as "usable" is the most common mistake on this chain—you only discover in the editing desk that segment 4's character changed, wasting previous assembly work. Now there's an extra layer: even if visuals are all correct but one word is mispronounced, the whole segment is still unusable because subtitles won't match.
The dialogue layer is the most likely to be skipped because it requires quiet headphone listening, not a quick glance.
VI. Ledger: Written in 04-assets/00-generation-ledger.md
One row per segment, recording six items: request fingerprint, mode, taskId, status, output file, failure reason.
The ledger's role is not just archiving—it's preventing duplicate submissions. Coming back a day later to continue the same film, reading the ledger tells you which segments are done, which were submitted but not downloaded, and which failed.
See rules in references/task-ledger.md. Four key points:
- Status is "Quality Passed" and file exists → Skip
- Has taskId and status is "Waiting / Generating" → Only query, don't create new
- Status is "Generated successfully, pending download" → Only download; if URL expired, query same taskId for new URL
- Status is "Failed" → Don't automatically retry; wait for user to explicitly say redo
VII. How to Handle Failure
First locate which layer is broken, then decide what to change. Don't start changing prompts right away.
| Phenomenon | Root Cause | Where to Change |
|---|---|---|
| Character distorted, product looks different | No first frame, or first frame itself is wrong | Add/replace first frame |
| Shake, flickering | Too many actions described | Reduce to one action |
| Text messed up, extra fingers | Prompt requires model to write text / do complex hand movements | Leave text to post-production, simplify hand movements |
| Wrong aspect ratio, black bars | Parameters don't match output | Mark as failure, regenerate; don't crop black bars |
| Segments don't connect | Last frame doesn't match next segment's first frame | Use first-last frame mode, or add a transition frame |
| Missing sentences, only half spoken | Segment dialogue word count exceeds shot length | Cut dialogue, or split segment into two |
| Mispronounced words, different words spoken | Uncommon characters, polyphonic characters, brand names buried in long sentences | Use common phrasing; make brand names a separate sentence |
| Dialogue drowned by music | Prompt didn't say "no music" or "don't cover voice" | Add an audio constraint line, regenerate |
| No sound at all | Accidentally used --no-audio, or audio disabled | Check submission parameters, regenerate |
| Submission error / 429 | Too high concurrency hitting rate limit | --concurrency 2, rerun; don't change prompt |
| Same segment keeps failing | Multiple variables changed at once | Change only one variable at a time |
Dialogue issues account for more than half of all failures. The commonality: the visuals look fine; only listening reveals the problem. So each segment must be heard before marking it as passed.
When reworking, change only one variable at a time. Changing prompt, duration, and first frame all at once means even if it succeeds, you won't know which change caused it, and the same issue will recur.
See detailed diagnosis in references/clip-review.md.
VIII. Boundaries
- Do not redesign visuals: Character, scene, color palette, lighting are locked at S2; only make them move here
- Do not change dialogue text: Script is read as written. To change words, go back to S1/S2; don't adjust here
- Do not create subtitles: Text is always added in post-production. Model can "speak" Chinese but cannot "write" Chinese
- Do not fix audio: Mispronounced segments can only be regenerated, or subtitles modified to match. No mixing, no re-recording here
- Do not edit: No splicing, no cutting to beat, no color grading—that's S4's job
- Do not judge "aesthetic quality": Only judge "usable or not." Look and feel belong to the review stage
- Do not guarantee first-time success: Generation is probabilistic; 2-4 candidates are normal, not failure
IX. Reference Materials
| File | When to Read |
|---|---|
references/api-and-modes.md | Checking API, parameters, three mode fields, error responses |
references/video-prompt.md | Writing video prompts, modifying prompts |
references/task-ledger.md | Building ledger, continuing work, deciding whether to resubmit |
references/clip-review.md | Frame sampling QA, judging failure, deciding rework direction |
| references/assembly-ops.md | S4 splicing, subtitle and mixing FFmpeg implementation details |
assets/gen-ledger-template.md | When building the ledger |
| assets/dialogue-final-template.json | When recording final dialogue, transcription assistance, and manual final review |
assets/batch-tasks-example.json | When running batch tasks |
Scripts:
| Script | Purpose |
|---|---|
scripts/kie_seedance.py | Submit / batch / query / wait for download / upload reference frame |
scripts/inspect_clip.py | Media inspection (requires ffprobe) |