Video Director

What does Video Director do?

Use this skill before calling Operate(resource="video", operation="generate", input={ ... }) for video work.

For AI chat, this skill is the single entrypoint for video generation. If the user says "make a talking-head video", "generate a video", "product promotional video", "ad", "digital human video", or any similar video request, invoke video-director first. Do not start from cosyvoice-ssml; that skill is an internal digital-human TTS substep and must not drive the video workflow.

If this skill was loaded because the user attached an image and asked for a promotional film / 宣传片 / 视频, treat the attached image as a product or visual reference for the video workflow. Do not generate a replacement product image as an "analysis" step.

Default Workflow

Treat the workflow as a gated state machine. Do not skip or reorder states:

1. plan_draft: Clarify the intended video mode from the user's goal and assets, then draft a concise but detailed video script and shot table.
2. storyboard_preview: Immediately after the shot table exists, generate storyboard contact-sheet preview image(s) with Operate(resource="image", operation="generate", input={ ... }). The storyboard image is a preview of the planned shots, not a standalone product poster, replacement product render, cover image, or final video keyframe.
3. user_review: Show the script, storyboard preview image(s), and explicit video specs together. Ask the user to confirm or revise the whole plan.
4. production_prep: After confirmation, decide whether this should be a <=15s single upstream clip, a long video_sequence request, or a manual editor/project assembly. Only use a video project pack if the user explicitly asks for a project/package/editor workflow, or the task is already bound to an existing pack.
5. tts_if_needed: If the confirmed task is a character talking-head / 口播 video, complete the TTS-first workflow below.
6. video_generate: Only after confirmation and any required TTS audio is complete, call Operate(resource="video", operation="generate", input={ ... }).

If the user has not yet confirmed the script, do not generate the video.

If the user has not yet seen and approved the storyboard preview image(s), do not generate the video.

If the shot table exists but no storyboard contact-sheet image has been generated yet, the next tool call must be image.generate for the storyboard preview, not video.generate, voice.speech, tts-director, or cosyvoice-ssml.

The storyboard contact-sheet preview is mandatory after the first shot script is written. It is not the same as final video keyframes; it is a quick visual proof of the planned shot sequence so the user can approve direction, composition, product placement, and continuity before video generation.

For product promotional videos from an attached/reference image:

• The first image.generate call, if any, must be for a storyboard contact sheet after the script and shot table exist.
• The storyboard prompt must explicitly say it is a multi-panel storyboard preview for the video, and it must preserve the attached/reference product's visible shape, material, color, and distinctive details.
• Do not call image.generate just to "analyze", "extract", "enhance", or recreate the product reference.
• Do not call video.generate until the user confirms the storyboard direction.

Asset-Library Character Talking-Head Rule

When the user wants a talking-head / 口播 / presenter / livestream welcome video using a character from the asset library, the character image is only the visual identity reference. It is not the speech track.

Keep these three inputs separate:

• voiceId: the reusable cloned voice identity stored on the character asset. This is the primary voice field for future TTS calls.
• voice sample / reference audio: fallback source material for cloning when no ready voiceId exists. Do not keep analyzing or presenting it once a ready voiceId has been found.
• generated TTS audio asset: the completed spoken script. This is the only audio that should drive a character talking-head video.

Use this exact order:

1. Script and storyboard first
   • Confirm the final spoken script.
   • Confirm the storyboard / shot table.
   • The Sound column must contain the actual spoken lines, not vague labels like "角色讲解".
2. Resolve the character voice
   • Read the selected character asset before generating media.
   • If the asset has a ready voiceId / voice_id, treat that as the resolved voice and stop inspecting voice samples. Report the resolved voiceId, not the reference-audio filename.
   • If the asset has a voice sample but no ready voiceId, call Operate(resource="voice", operation="clone", input={ ... }) first, wait for completion, and use the returned voiceId.
   • If there is no ready voiceId and no usable voice sample, stop and ask the user to provide or choose a voice. Do not fall back to a generic voice silently.
3. Generate the complete voice track with TTS
   • Call Operate(resource="voice", operation="speech", input={ ... }) with the full approved spoken script and the resolved voiceId.
   • Prefer one complete audio asset for the whole talking-head segment unless the user explicitly wants separate clips.
   • For expressive narration, long scripts, poetry, ads, or any multi-emotion performance inside this confirmed digital-human / VideoRetalk flow, activate tts-director first with Operate(resource="skills", operation="invoke", input={ "name": "tts-director" }), then use its guidance to split the approved script into semantic beats and assign model-appropriate delivery controls.
   • If the selected TTS model is CosyVoice, tts-director may activate cosyvoice-ssml only here, after the video-director workflow has confirmed this is a digital-human / VideoRetalk / asset-library talking-head video and the script plus role voice are resolved.
   • Add restrained delivery controls when they improve the approved performance: use speed for pace, pitch for tone, emotion for mood, expressive punctuation such as ～, ？, ！, ……, and MiniMax text markers such as <#0.6#>, (laughs), (sighs), or (breath) inside input only where the spoken rhythm needs them.
   • Do not write control instructions into the spoken text. For example, pass "emotion":"happy" and speed:1.08 instead of making the character say "用开心快速的语气".
   • If the approved voice track needs multiple emotional beats, submit one voice.speech call with ordered segments; the media runtime will generate each segment and merge the final audio. Do not make repeated voice.speech calls and then manually merge.
   • Wait until the TTS result contains a usable audio asset path, URL, or asset id.
   • Do not start video generation while the TTS job is only queued or running.
4. Generate video with visual reference plus generated audio
   • Use the selected character image as the visual reference.
   • Use the newly generated complete TTS audio asset as drivingAudio.
   • Identify it in the prompt preface as generated speech audio, for example Audio 1: generated Jamba TTS speech for lip-sync / speaking rhythm.
   • Then call Operate(resource="video", operation="generate", input={ ... }), normally in reference-guided mode.

This is mandatory because the video model needs a finished audio waveform aligned with the approved script. A character voiceId is not audio. A voice sample is not the approved spoken line. The spoken text inside the video prompt is not a substitute for TTS.

Forbidden shortcuts:

• Do not call video generation for character talking-head videos with only the character image and a script.
• Do not pass voiceId, voice sample audio, or "voice material" as drivingAudio.
• Do not rely on generateAudio to create the character voice for talking-head work.
• Do not describe the intended speech only in the video prompt and expect the video model to synthesize it.
• Do not split the TTS per storyboard row unless the user asks for separately editable audio clips or the video will be assembled from multiple independent speaking clips.

Recommended TTS payload shape:

{
  "voiceId": "voice_xxx",
  "segments": [
    {
      "input": "Full, confirmed opening speech for the talking-head video.<#0.5#>",
      "speed": 0.98,
      "pitch": 0,
      "emotion": "calm"
    },
    {
      "input": "Naturally move into a more impactful key point (laughs).",
      "speed": 1.05,
      "pitch": 0,
      "emotion": "happy"
    }
  ],
  "title": "Jamba welcome voiceover",
  "waitForCompletion": true
}

Recommended video payload shape after TTS completes:

{
  "generationMode": "reference-guided",
  "referenceImages": ["/absolute/path/to/jamba.png"],
  "drivingAudio": "/absolute/path/to/generated-jamba-tts.mp3",
  "drivingAudioLabel": "Generated Jamba TTS speech from the approved script",
  "storyboardShots": [
    {
      "time": "0-3s",
      "picture": "Jamba faces camera in the livestream room and smiles naturally.",
      "sound": "Welcome to the livestream, today I have a super practical item for everyone.",
      "shot": "Medium close-up, stable presenter framing."
    }
  ]
}

Video Project Pack Rule

A video project pack is not the default path.

Create one only when:

• the user explicitly asks for a 视频项目 / 视频工程 / 项目包 (video project package)
• the user explicitly asks to continue later inside the video editor / project workbench
• the task is already bound to an existing video project pack

Do not create one only because the request has multiple shots, long context, continuity risk, storyboard needs, or possible revisions.

When it is explicitly needed, create it with:

• Operate(resource="video", operation="create", input={ "explicitProjectWorkflow": true, "title": "...", "duration": "...", "aspectRatio": "...", "mode": "..." })

The project folder lives in:

• manuscripts/video/<project-name>/

It should be used to keep these files together:

• manifest.json
• script.md
• assets.json
• editor.project.json
• imported reference images / keyframes / generated clips / final output

After the pack is created:

• write the user brief and approved script back into the video project folder
• keep later keyframes, clips, and outputs in the same pack whenever possible

Otherwise keep the planning in chat, call video generate directly, and let the output live in the generated media library.

Hard Rules

• Video generation is locked to the official video route configured by the app.
• Do not choose arbitrary video endpoints or third-party video models.
• Use only these official model mappings:
  • text-to-video -> wan2.7-t2v-video
  • reference-guided -> wan2.7-r2v-video
  • first-last-frame -> wan2.7-i2v-video
• Treat first/last-frame transitions as a subtype of image-to-video work.
• A single upstream video generation segment must not exceed 15 seconds.
• For a final video longer than 15 seconds, submit one video.generate request with either the final durationSeconds or explicit videoSegments; the media queue will generate <=15s segments and concatenate one final video asset.
• Do not call video.generate repeatedly just to bypass the upstream 15 second segment limit.
• Do not skip the script review step just because the request sounds obvious.
• Unless the user explicitly asks for a longer continuous shot, a single shot should usually be 1-3 seconds.
• Without explicit user approval, any single shot must not exceed 5 seconds.

Mode Selection

• Use text-to-video when the user only provides text and wants a fresh video shot.
• Use reference-guided when the user provides one or more reference images and wants the video to absorb subject elements, style cues, props, scene motifs, or composition hints from those images.
• Use first-last-frame only when two images have explicit start/end semantics, such as "from A to B", "first frame/last frame", "start/end", or "initial state/final state".
• If the user gives two images but they are only style references, do not use first-last-frame; stay with reference-guided semantics instead.

Production Strategy

• Single Video Mode:

  • Use one generated video clip.
  • Default when the request is simple, the action is short, and the full idea fits inside one coherent clip.
  • A single generated clip must not exceed 15 seconds.

• Long Video Queue Mode:

  • Use this for one final AI-generated video longer than 15 seconds.
  • Submit one video.generate request with durationSeconds > 15; the runtime will create a video_sequence queue job, split the generation into <=15s segments, wait for each segment, concatenate the segments, and return one final video asset.
  • Use explicit videoSegments when the storyboard needs scene-by-scene, beat-by-beat, reference-image, first/last-frame, or driving-audio control.
  • Each videoSegments[] item must be <=15 seconds and should correspond to a coherent group of approved storyboard rows.
  • Keep waitForCompletion: true unless the user explicitly asks to run the long generation in the background.

• Manual Multi-video Project Mode:

  • Use this only when the user explicitly asks for separate clips, a project/editor workflow, imported clip assembly, manual post-production, or separately editable outputs.
  • In normal chat/agent video generation, do not manually generate one clip at a time and concatenate with ffmpeg; use Long Video Queue Mode instead.
  • When manual assembly is explicitly needed, generate each clip deliberately, then concatenate them in storyboard order after all clips succeed.

• If the request has multiple shots, clear continuity requirements, or a risk of visual drift, ask one more question after showing the contact-sheet preview:

  • whether separate storyboard keyframes should also be generated before video production.

• If separate storyboard keyframes are generated, later video generation should preferentially use image-based modes, and for transition-heavy segments should prefer first-last-frame.

Storyboard-First Rule

Every video task needs a storyboard contact-sheet preview after the script table and before user confirmation.