TikTok / Douyin Short Video Generator

What does Douyin Short Video Generator do?

You (the agent) do the creative work — script, scene breakdown, search keywords, style choices, and quality review. The scripts in scripts/ do the mechanical work — TTS with word timestamps, asset search & download, caption rendering, ffmpeg composition, loudness normalization.

Everything runs with core ffmpeg only (no libass/drawtext needed) plus a Python venv. Works with zero API keys; PEXELS_API_KEY / PIXABAY_API_KEY env vars unlock real stock video clips (better) when present.

Workflow

0. One-time setup (skip if .venv/ and assets/fonts/ exist)

bash scripts/setup.sh

1. VERIFY FACTS FIRST, then write the script + storyboard

Your internal knowledge may be outdated or wrong — never put an unverified claim in a video. Before writing any scene text, use your web search tool to check every concrete claim: numbers, statistics, records, prices, dates, rankings, and every "first / biggest / only / fastest" statement — and anything time-sensitive (news, products, versions, etc.) regardless of how confident you feel. Rules:

• A claim goes into the script only as a source states it — adjust your wording to the source, not the other way around.
• Can't verify it? Replace it with a claim you can, or cut it. A wrong number in a published video is worse than a weaker hook.
• Record the URLs you relied on in the storyboard's "sources": [...] — they are echoed in review/report.txt so the user can audit the video.
• If you genuinely have no web access, stick to long-established textbook facts, avoid specific figures, and tell the user the video is unverified.

Read references/writing-guide.md for the script formulas (hook types, pacing, scene structure). Then create projects/<slug>/storyboard.json:

{
  "title": "The 5 Most Bizarre Creatures in the Deep Sea",
  "lang": "en",
  "aspect": "9:16",
  "voice": "en-US-ChristopherNeural",
  "rate": "+10%",
  "caption_style": "karaoke",
  "bgm": {"mood": "mystery", "gain_db": -16},
  "hook": {"text": "Deep Sea Secrets", "seconds": 2.3},
  "scenes": [
    {
      "text": "Did you know? In the deep sea where sunlight never reaches, creatures stranger than sci-fi films lurk.",
      "keywords": ["deep sea NOAA ocean exploration", "submarine dark ocean"],
      "providers": ["openverse", "wikimedia"],
      "badge": null,
      "effect": "kb_in"
    },
    {
      "text": "Number one, the anglerfish…",
      "keywords": ["humpback anglerfish", "anglerfish museum"],
      "badge": "TOP 1"
    }
  ]
}

Field reference:

field	values	notes
lang	en | zh	sets caption grouping + default voice
aspect	9:16 (default) | 16:9 | 1:1
voice	any edge-tts voice	zh: zh-CN-YunjianNeural (male, magnetic) zh-CN-XiaoxiaoNeural (female) zh-CN-YunxiNeural (sunny male); en: en-US-ChristopherNeural en-US-AriaNeural en-US-GuyNeural
rate	e.g. +10%	Marketing-style pacing: zh +8%+15%, en +5%+10%
caption_style	karaoke | pop | none	karaoke = word-by-word highlight (recommended)
bgm	{"mood": ...} | {"query": "style words"} | {"file": "path.mp3"} | {"mood":"none"}	Default to a mood: the curated table (upbeat, funny, inspiring, chill, tech, mystery, epic, sad, horror) is the exact MacLeod staples countless marketing channels run on — proven, safe, varied per project. Reach for query only when no mood fits the content's style (or the user names a genre): it searches real CC beat music (ccMixter style tags + Openverse/Jamendo; ND/NC excluded) and every candidate is screened by the beat tracker — no confident 75–175 BPM groove, no pick. Style words that map to tags: trap/phonk, hip hop, lofi/chill, edm/dance, epic/cinematic, quirky. You may set both — query is tried first, mood is the fallback. file = the user's own track
bgm.vibe	"spedup" | "slowed" | unset	TikTok signature sound: spedup ≈ 1.25× with raised pitch (common for quick-cut/list videos), slowed = slowed + reverb (emotional/atmospheric). Applied to any bgm source incl. file; beat grid is re-analyzed after processing so beat-sync stays accurate
beat_sync	true (default) | false	the BGM is beat-analyzed and every scene/shot cut snaps onto the beat grid when the grid is confident; VO is never truncated, only the breathing pads flex
hook	{"text","seconds"}	big top title card shown at the start, ≤ 8 chars/words
sticky_title	{"text": ...}	optional persistent topic bar at the top; off by default — only add if the user asks for one
sfx	true (default) | false	whoosh sound on scene transitions
scene keywords	list of English, concrete-noun queries	each entry = one shot (visual); the video cuts to a new visual every ~3s, so give 2–3 queries for scenes longer than ~4s (extra shots are auto-added cycling your queries if you give fewer)
scene badge	e.g. "TOP 1" / "TOP 1"	big stamped label shown at the scene start — use for listicles
scene providers	list	keyless images: openverse wikimedia nasa; keyless real video: wikimedia_video (Commons clips, server transcodes) nasa_video (PD space/science) archive_video (Prelinger public-domain historical film); with keys: pexels_video pexels_photo pixabay_video
scene effect	auto kb_in kb_out pan_left pan_right static	first shot's Ken Burns motion; later shots auto-cycle
scene emphasis	list of substrings	permanently highlighted words (pop style)
scene media	file path	bypass search, use your own file (e.g. one you downloaded yourself)

Rules of thumb (publish bar): total 45–75s, 6–9 scenes, one idea per scene, 2–3 shots per scene. Scene text 20–45 Chinese chars / 14–30 English words. Anything under ~40s or with single-shot scenes throughout will feel thin — don't ship it.

2. Run the pipeline

.venv/bin/python scripts/pipeline.py projects/<slug>

Or stage by stage: tts.py → assets.py → bgm.py → compose.py → check.py.

3. REVIEW — this is what makes the difference (mandatory)

Free-image search is imperfect. Always view these two files with your image tool and judge them like a human editor:

1. projects/<slug>/media/assets_sheet.jpg — after the assets stage. Shots are labeled 01a 01b 02a …. For every shot that does not clearly match the narration, refetch it:

   .venv/bin/python scripts/assets.py projects/<slug> --scene 3 --shot 2 --keywords "better english nouns"
   

   (Refetching automatically blacklists the rejected asset. Repeat until all scenes match. You may also download an image yourself with your own tools and point scene media at it.)

   The pipeline also text-audits every shot's source title against its query and prints [!] OFF-TOPIC flags. If you visually confirmed a flagged shot is actually right (e.g. an untitled but perfect photo), set "approved": true on that shot in media/manifest.json to clear it — your eyes outrank the text audit. Never blind-approve.

2. projects/<slug>/review/contact_sheet.jpg — after compose+check. Verify: captions readable & synced, no wrong/ugly frames, hook visible. Fix issues (edit storyboard → rerun compose, it's fast) until it looks like a video a professional editor would actually publish.

review/report.txt contains duration/loudness checks and a ready-to-paste attribution block (CC-BY sources + music credit) — always give it to the user together with final.mp4.

Asset search tips (biggest quality lever)

• Keywords must be English, 1–4 words, concrete visible nouns ("humpback anglerfish", "scuba diver silhouette"), never abstract concepts ("mystery", "success").
• wikimedia is best for animals/science/history/places; openverse (Flickr etc.) for lifestyle/scenery/mood; nasa for space; keys unlock pexels_video/pixabay_video real footage — prefer those when available.
• Add a context word to disambiguate: "NOAA", "museum", "aquarium", "macro".
• If a scene keeps failing, change the visual concept, not just the words (e.g. for "5% explored" show a diver silhouette, not "statistics").
• wikimedia_video searches support negative terms — Commons weather/space queries drown in satellite loops, so write e.g. "lightning storm -CIRA -satellite -JPSS". Non-English words unlock extra footage ("Blitz Gewitter", "tormenta rayos"). Prefer named subjects ("Fagradalsfjall lava drone") over generic nouns, and always eyeball video thumbs in the sheet: watch out for burned-in captions, watermarks, tutorials and CGI promos.

Troubleshooting

• edge-tts network errors: it retries 4×; rerun tts.py if it still fails.
• BGM download fails: rerun bgm.py (retries + cache), or --mood none, or drop an mp3 into assets/bgm/ and set bgm.file.
• BGM style doesn't fit: refine the query (bgm.py <dir> --query "..." then recompose with --skip-tts --skip-assets). For a truly trending sound, deliver with mood: none (or low gain) and tell the user to add the trending track from TikTok/抖音's in-app licensed music library at upload time — that is both legal and algorithm-favored. Cut timing will still be clean because scene pacing is unchanged.
• check.py prints a "cuts land on the beat" line — if beat sync was skipped (low grid confidence, e.g. beatless ambient music), pick a steadier track (--query "trap beat" etc.) and recompose.
• A provider erroring/empty is fine — others cover it; check per-scene logs.
• Recompose after any storyboard edit is cheap (--skip-tts --skip-assets via pipeline.py, or run compose.py directly). Changing scene text requires rerunning tts.py.
• Emoji in captions render on macOS (Apple Color Emoji); on Linux they are dropped unless NotoColorEmoji is installed. Never rely on emoji for meaning.

Deliver

Give the user: final.mp4, the attribution block from review/report.txt, and (if asked) review/cover.jpg as the cover image.