digital-human-video

Digital Human Video Generation

What This Skill Does

Input three things—a script, a portrait photo, and a voice sample—and produce a vertical talking-head short video featuring the person in the photo: Clone the person's voice to read the script, use Seedance 2.0-mini to make the photo's mouth move in sync with the audio, add a Suno-generated BGM, burn in rich-styled subtitles with keyword highlighting and karaoke progression, and finally compose with ffmpeg. The entire pipeline relies solely on documented public APIs; any agent that can run Python and ffmpeg can execute this skill, with no dependency on a specific client.

Every parameter and pitfall in the pipeline has been validated in real projects (see references/), not just on paper.

Input Required from the User Before Starting

1. Script: Save as script.md in the project root, one sentence per line, formatted as "1. Line" "2. Line". Sentences are the unit for segment generation; if a sentence is too long (over 15 seconds), split it into two.
2. Portrait Photo: A photo under assets/, best in vertical 9:16 ratio (e.g., 1152×2048). Photo quality directly determines the final video quality: front-facing, even lighting, no sunglasses or mask. If the user has no suitable photo, generate one using GPT Image 2 (see references/api-reference.md) and ask the user to confirm the likeness before proceeding to the video stage.
3. Voice Sample: A 15–30 second clean vocal recording under assets/ (no background music, no reverb) for Fish Audio cloning.
4. Two API Keys: One each for Fish Audio and APIMART. Pass via environment variables. Never write them to any file; never show them in command output. Scripts read from environment variables and exit with a prompt if a key is missing; keys are never written to disk.

Costs & Accounts

Component	Mode	Cost (CNY)	Model Source	Notes	Referral Link
Person	GPT Image 2	0.1	APIMART API	Only needed if no photo exists	https://apimart.ai/register?aff=QVui
Video	Seedance 2.0-mini	10	APIMART API	720x1280, billed per second	https://apimart.ai/register?aff=QVui
BGM	Suno	0.35	APIMART API	Instrumental mode, two tracks at once
Voice Clone	s2.1-pro-free	0	Fish Audio API	Free tier, voice_id reusable forever	https://fish.audio/?aff=X7QT3YAX3N4Q4
Subtitles	whisperx small	0	Local	One-time environment setup

Measured real-world: Seedance 2.0-mini 720p ~ $0.039/sec, a 30-second video ~12 CNY; Fish Audio clone requires account balance (zero balance returns 403), TTS uses free tier (no cost); Suno $0.05 per track. A finished video realistically costs 10–13 CNY.

Five-Stage Pipeline

Each stage has a gate: if not satisfied, do not proceed to the next stage. State is stored in output/state.json, enabling checkpoint resume.

Stage A – Voice (Fish Audio)

1. python scripts/avatar_voice.py clone: Upload voice sample to create a private clone model, retrieve voice_id.
2. python scripts/avatar_voice.py tts: Synthesize WAV for each sentence from script.md.
3. python scripts/avatar_voice.py fixwav: Repair streaming WAV headers (Fish returns headers claiming a length longer than actual; some decoders will truncate audio without this fix).
4. python scripts/avatar_voice.py qc: Check each sentence for duration, average volume, and clipping.

Gate: All sentences have volume ≥ -30dBFS and no clipping. Estimated duration is a reference only; do not treat it as a hard target—using atempo to stretch audio to meet a target duration is strictly forbidden (see hard rules). The user must listen and confirm the voice sounds like the real person before proceeding.

Stage B – Video (Seedance 2.0-mini)

1. python scripts/avatar_video.py upload-image: Upload the portrait photo to APIMART to get a public URL.
2. python scripts/avatar_video.py host-audio: Host each WAV file via tmpfiles.org, parse out the /dl/ real direct link (APIMART has no audio upload endpoint; direct links expire after 60 minutes, so submit immediately after upload).
3. python scripts/avatar_video.py submit: Submit each sentence with core parameter generate_audio=False (video is silent; lip sync is driven by external audio—this is the digital human switch). Duration is rounded up to the nearest integer based on audio length. A cost estimate exceeding $1 per segment triggers automatic circuit break.
4. python scripts/avatar_video.py poll + download: Poll and download.

Gate: Spot-check frames for no face distortion/swapping; lips must open and close. If audio changes, this stage must be re-run—old video lip sync was driven by the old audio; simply swapping the audio track does not guarantee sync.

Stage C – BGM (Suno)

1. Read the script and write a music brief (style/emotion/BPM/energy curve/avoidances). See references/prompts.md for the template.
2. python scripts/avatar_bgm.py generate: Generate two BGM candidates in instrumental mode.
3. python scripts/avatar_bgm.py mix: Trim, fade in/out, compress to ~-16dB below vocals, apply sidechain compression with ducking, and mix into the final video.
4. User listens and selects one track. Music taste is subjective; automation stops at "generating qualified candidates," leaving selection to the human.

Stage D – Subtitles (ASS Rich Styles, Local, Zero Cost)

No online services required. Uses ASS/libass to achieve subtitle effects comparable to professional editing software. Six capabilities of avatar_caption.py (each verified pixel-by-pixel, not theoretical):

Capability	Configuration	Description
Cue Splitting	Write `
Keyword Highlighting	spec.keywords	e.g., AI → cyan, 5000 people → gold; matches entire terms regardless of language
Karaoke Word-by-Word	spec.karaoke.enabled	Highlights the current word being spoken; previous words stay lit
Dual Text	spec.dual_text	Say "ten yuan" but display "10 yuan"; separate spoken from written form
Time Window Style Change	spec.emphasis	Specify 3–6 seconds with larger font size or different color for emphasis
Bilingual + Layout Animation	spec.points	Chinese main track + English key points track, with \fad fade-in and \t scaling pop-up

python scripts/avatar_caption.py spec    # Write output/caption-spec.json template
python scripts/avatar_caption.py all     # Generate ASS + burn into digital-human-captioned.mp4

Subtitle text source priority: --srt specified > whisperx's output/digital-human.srt > voice/qc.json duration + original script.md text. When the original script text is available, prefer it (add --prefer-script), as it is more accurate than ASR transcription and doesn't require whisperx.

Two critical libass conventions (both have caused real bugs; see references/pitfalls.md): the unread color for karaoke is SecondaryColour, the read color is PrimaryColour; \k units are centiseconds and their sum must exactly equal the cue duration—using percentages causes the karaoke to finish within one second.

If the user will add subtitles with tools like CapCut, this stage can be skipped, but the delivery note must mention it.

Stage E – Composition & QC

Trim the four video segments to match audio duration, scale to 1080×1920, 30fps. Audio track uses Stage A's TTS (discard the video's own audio track). Compose with ffmpeg. The composition command template is in references/prompts.md. QC checks: total duration matches sum of segments, waveform has zero offset from vocals, no clipping, full decode.

Hard Rules (Each Earned from Real Mistakes)

1. Do not use atempo or other post-processing to stretch audio to fit a target duration. Stretching drastically slows down speech and sounds dragging. To adjust speed, re-generate with Fish TTS's native prosody.speed parameter (e.g., s3=0.85, s4=0.9), or accept the natural duration.
2. API Keys exist only in process memory. Pass via environment variables; the script clears them on exit. Any string in output that matches a key pattern must be replaced with a placeholder. No keys in project files, reports, or logs.
3. If the same operation label fails twice consecutively, stop immediately. Write a report explaining the reason. Do not retry indefinitely. HTTP 401 stops immediately (key issue). Circuit break if cost estimate for a single segment exceeds $1.
4. Estimated duration is a reference, not a target. QC criteria must not be "filler-type."
5. User listening experience takes priority over automated statistics. Duration, volume, and clipping may all pass, but the listening experience may still be poor. If the user says something is wrong starting at the 10th second, treat it as a problem and trace back from that point.
6. Do not label anything as "final" until it passes manual review. When delivering, explicitly list which items have been verified and which have not.

Environment Pitfall Quick Reference

Running this pipeline on Windows has four confirmed pitfalls; read references/pitfalls.md before starting: Python requests must set trust_env=False otherwise they connect to the wrong proxy; the sandbox and the real filesystem are two different views; tmpfiles direct links require parsing the view page; Fish streaming WAV headers need repair. Each pitfall includes root cause and fix.

Script Usage

export APIMART_API_KEY=... FISH_API_KEY=...   # or let script prompt on input
python scripts/avatar_voice.py all     # Stage A full pipeline
python scripts/avatar_video.py all     # Stage B full pipeline
python scripts/avatar_bgm.py all       # Stage C
python scripts/avatar_caption.py all   # Stage D (no key required)

Script	Stage	Keys Required	Purpose
avatar_voice.py	A	Fish Audio	Voice clone, per-sentence TTS, WAV header repair, QC
avatar_video.py	B	APIMART	Upload image, host audio, submit lip-sync video, poll and download
avatar_bgm.py	C	APIMART (generate yes, mix no)	Suno BGM candidates, sidechain ducking & mix
avatar_caption.py	D	No	Rich-style subtitle generation and burning
avatar_compose.py	E	No	Compose, scale, transcode, final check

Scripts read script.md and assets/ from the current directory, output to output/, state in output/state.json. Rerunning the same command after interruption resumes from the checkpoint. Every script supports --help to list subcommands.

Security Boundaries (Commitment When Distributing to Others)

• Scripts connect only to four documented endpoints: api.fish.audio, api.apimart.ai, tmpfiles.org, and local ffmpeg/whisperx. No telemetry, auto-updates, or anonymous statistics.
• Do not read or write any paths outside the project directory.
• API Keys exist only in process memory, cleared on exit; all persisted text is filtered for key patterns.
• The entire package is fully auditable: SKILL.md + references/ + scripts/, no binaries, no executable installation steps.