Knowledge Animation Video Generation

The user provides a voiceover (script or audio) + a style reference image. You are responsible for choosing a template, designing the storyboard, writing code, and rendering the final video.

Tech Stack: Remotion (React for video writing) + Python scripts (TTS/alignment/audio mixing) + ffmpeg.

Core Principle: Style Determines the Tool. First decide the type, then follow that type's template to select tools and write scenes.

Author: Xiao Bai (小白姐姐) · GitHub @newbaisays-aigc · MIT License Project URL: https://github.com/newbaisays-aigc/knowledge-animation-video

This Skill is independent of any specific AI tool. As long as your host environment can read SKILL.md, run shell commands, and write files, you can use it. For TTS, prefer components/common/scripts/tts.py (Edge TTS, requires internet); If the environment does not allow installing Python dependencies, fall back to the host's built-in speech synthesis capability, outputting the same vo.mp3 + sentence-level timing table. No model restrictions: any model capable of writing code can be used. When the codebase is large, prioritize models with strong reasoning capabilities.

Step 1: Opening Selection (Must be done first, do not skip)

Ask the user all 4 questions at once. If the platform supports tab-based questioning, use tabs; otherwise, list the numbered options in plain text and ask the user to reply.

Question 1: Video Type (One-sentence memory aid: choose 3D for impact, Hand-drawn for warmth, Formula for deduction, Infographic for structure, Archive for history, Gallery for artwork)

Number	Type	Suitable Content	Suitable Audience
A	3D Data	Hard data: data journalism, product launches, industry reports, financial analysis	Finance/tech bloggers, new media editors, marketing and product folks
B	Hand-drawn Doodle	Opinions and stories: history & humanities, social phenomena, book notes, psychology	Humanities teachers, knowledge-payment instructors, book bloggers
C	Math Formula	Abstract math/physics formulas, algorithms, AI principles	STEM teachers, students creating courseware, tech bloggers
D	Infographic	Processes, methods, system architectures, product features, business workflows	Workplace presentations, corporate training, B2B and SaaS operators
E	Vintage Archive	Historical figures, historical events, history of inventions, brand stories, vintage objects	History bloggers, humanities & science popularizers, brand content, cultural tourism
F	Art Gallery	Famous painting analysis, art history, cultural relics, design, architecture, photography	Art bloggers, art education teachers, museums & cultural tourism, designers
G	Help Me Choose	User unsure – after reading the script, you make a recommendation	—

Question 2: Aspect Ratio

16:9 landscape (1920×1080): Bilibili, Xigua Video, YouTube, courseware
9:16 portrait (1080×1920): Douyin (TikTok), WeChat Video Channels, Xiaohongshu
1:1 square (1080×1080): WeChat Moments, feed ads

Question 3: Voiceover Method

Script: user provides text, you generate TTS
Audio: user provides pre-recorded audio, you transcribe and align
No voiceover: only visuals + background music

Question 4: Subtitles

With subtitles (default recommended): sentence-by-sentence displayed at the bottom of the screen, ideal for silent mobile viewing
Without subtitles: cleaner visuals, suitable for adding subtitles later in editing software like Jianying (CapCut)

Note: When voiceover method is "No voiceover," subtitles are the only way for the audience to understand the content – remind the user that keeping subtitles is recommended.

After asking, send a single message requesting the two materials, then wait for the user's reply. Do not write scripts, create storyboards, or generate anything until both materials are received.

Use the following message (adjust the example type based on the user's selection):

Great, now please provide two materials:

Material 1: Voiceover Send a script or a pre-recorded audio clip. For a first run, a short one of 15–30 seconds (about 60–120 words) is recommended for testing and lower cost. If you don't have a script yet, tell me the topic and duration (e.g., "Introduce Napoleon, 20 seconds"), and I'll write a draft for you to confirm.

Material 2: Style Reference (Very Important) Style is hard to describe with words — showing me is best. Choose any of the following:

Upload one or several images

Send a link: Pinterest, Xiaohongshu, Bilibili, YouTube image or video links all work

If you really have none, reply "Use default style"

Tip for finding references: Search on Pinterest or Xiaohongshu for "[type keyword]" and pick your favorite.

Search keywords by type: Art Gallery → "museum art documentary motion"; Vintage Archive → "vintage documentary motion graphics"; 3D Data → "data visualization 3D"; Hand-drawn Doodle → "doodle explainer"; Math Formula → "3Blue1Brown style"; Infographic → "infographic motion".

Rules for receiving materials:

User only provides a topic but no script: Write the script first, send it to the user for confirmation, then proceed.
User only provides a voiceover but no reference: Remind the user again about the reference. Only use the template's default style after the user explicitly replies "Use default style."
User provides a link: Open the link to access the actual image or video (for Pinterest links, use https://widgets.pinterest.com/v3/pidgets/pins/info/?pin_ids=<pin ID> to get the image/video URL). For videos, uniformly extract 8–10 frames and compose them into a single image for review. If the API cannot be accessed, do not get stuck or guess the style from the link text — directly ask the user to upload a screenshot or image; only use the template's default style if the user explicitly says "Use default style."

If the user selected G (Help Me Choose), read the script and recommend based on the rules below, including a brief reason, then wait for user confirmation:

Script contains many numbers, percentages, rankings, growth → A 3D Data
Discusses a work of art (painting, artifact, architecture) worth examining in detail → F Art Gallery
Discusses real historical figures, events, inventions, brands and their backstories → E Vintage Archive
Mostly opinions, stories, psychological concepts, few numbers → B Hand-drawn Doodle
Contains formulas, theorems, derivations, algorithm steps → C Math Formula
Discusses steps, processes, frameworks, module relationships → D Infographic

Step 2: Read the Template

After determining the type, read only the corresponding template file, do not read all of them:

A → templates/3d-data.md
B → templates/hand-drawn.md
C → templates/math-formula.md
D → templates/infographic.md
E → templates/vintage-archive.md (also read components/vintage-archive/README.md, directly reuse the component library)
F → templates/gallery.md (also read components/gallery/README.md, directly reuse the component library)

Each template is structured into 7 sections: Specifications determine size, Style determines appearance, Rhythm determines timing, Scenes determine motion, Music determines mood, Tools determine implementation, Delivery determines quality checks.

Adjustment rules:

Specifications, Delivery: Largely unchanged; only replace the aspect ratio with the user's selection.
Style, Rhythm, Scenes, Music: Adjust based on the user's content and reference image.
Tools: Follow the template; do not mix tools from a different template.

Component Library: components/ contains well-tuned reusable components. When the corresponding template has a component library, you must read its README first, prioritize reusing components and scripts, and only write new code for scenes unique to this video. This ensures stable results and fewer errors. Currently available:

components/common/: Shared scripts for all templates, including TTS (tts.py), self-recorded audio alignment (align.py), three-version audio mixing (mix_audio.py), and export (export_versions.sh). Use these scripts directly in Steps 4 and 7.
components/3d-data/: 3D data reference implementation (SpaceX earnings flash news source code), including 3D Chinese characters, glowing track lines, grid ground, holographic objects, and mask transitions
components/gallery/: Art Gallery (WebGL liquid glass lens canvas, lens view mathematics, artwork carousel, magnifying glass, artwork archive card and detail typography, complete Mona Lisa example)
components/vintage-archive/: Vintage Archive (paper labels, photo cards, ribbons, stamps, red circles, brass nameplates, book flipping, brass magnifying glass, full-screen artwork, wooden table, film overlay layers, plus image fetching, aging, TTS, and audio mixing scripts, complete Napoleon example)

After creating a new satisfying effect, extract the common parts and add them back to the component library.

Step 3: Analyze the Reference Image

Carefully examine the reference image, extract and note the following items (all subsequent scenes must adhere to these):

Main colors: 3–5 (write the color values, e.g., #0B1E3F)
Background: solid / gradient / texture / dark or light
Texture: flat / 3D shading / glass / paper / line art
Font character: bold black / rounded / handwriting / serif
Composition features: whitespace amount, element density, centered or offset
If the reference is a video, also note: camera movement style, transition style, text entrance style, rhythm speed

If the reference style conflicts with the selected template (e.g., user chose Hand-drawn but provided a 3D glass-texture image), first inform the user of the conflict, present two or three blending options for them to choose from, then proceed. Do not make a decision on your own.

Step 4: Process the Voiceover, Determine the Timeline

Script Mode:

Lightly clean up the script: remove typos and symbols unsuitable for reading (like ~ * #); do not change the user's intended meaning.
Use components/common/scripts/tts.py to generate the voiceover and sentence-level timing (write numbers as Arabic numerals; if the environment does not support Python dependencies, fall back to the host's built-in speech synthesis, outputting the same vo.mp3 and sentence-level timing table).

Audio Mode:

Use components/common/scripts/align.py <audio> <transcript> <output_dir> for alignment. If a transcript is provided, use it for subtitles; if not, use the recognized text.
If there is no transcript, or the similarity is below 60%, present the result to the user for verification.

No Voiceover Mode:

Estimate each sentence's duration at roughly 4–5 Chinese characters per second.

Finally, obtain a sentence timing table, which will serve as the basis for the storyboard. The total video duration is determined by the voiceover duration; visuals follow the voiceover.

Step 5: Create the Storyboard and Wait for User Confirmation (Do not render directly)

List the storyboard in a table, one row per scene:

Scene	Time	Corresponding Voiceover	Visual Content	Animation	Sound Effect

Requirements:

One voiceover sentence usually corresponds to 1 scene; long sentences can be split into 2.
Each scene must have a "protagonist element" and clearly describe how it moves (from where to where, over how long).
Numbers and keywords from the voiceover must appear on screen.
Clearly describe the transition between scenes.

After sending the storyboard, ask the user: "Does this plan work? Which scene would you like to change?" Only proceed to Step 6 after receiving user confirmation. This step avoids wasting tokens on the wrong direction.

Step 6: Generate the Video

Follow the template's "Tools determine implementation" section to set up the project, write code, and render:

Initialize a Remotion project (reuse if one already exists in the environment), set the width and height according to the user's chosen aspect ratio, at 30 fps.
Write each scene as an independent component (Scene01, Scene02...), and assemble them in order in the main Composition. This way, modifying a scene later only requires changing that one file.
Extract the color scheme from the reference image as a shared theme file; all scenes should reference colors from this file.
Subtitles (only if the user selected "with subtitles" in Question 4): display sentence by sentence, aligned with the voiceover timing. If the user chose not to have subtitles, do not render the subtitle layer, but retain the sentence timing table for scene pacing. For landscape videos, place subtitles at the bottom, approximately 8% from the bottom edge; for portrait videos, place them in the lower-middle area, approximately 25% from the bottom, to avoid the platform's buttons and text areas.
Background music and sound effects: Generate or select them according to the template's "Music determines mood" section. Set background music volume to 15%–25% of the voiceover volume, and automatically lower it when there is speech.
First, render 3–5 keyframe screenshots for self-check (composition, text overflow, color consistency with the reference image). Only proceed to render the full MP4 when no issues are found.

When encountering errors, read the error message to locate the specific scene, then modify only that scene — do not rewrite everything.

Visual design is a hard requirement. Do not just place text and images on a solid background. Each scene must extract "decorative elements" from the reference and turn them into reusable components, such as: tabletop or scene textures, photo paper or card borders, torn-edge paper labels, ribbon banners, stamps, hand-drawn red circles, brass nameplates, book page flips, film grain, vignettes, and flickering. Numbers and keywords should appear on these decorative elements, not as bare text.

Material Sourcing and Licensing (Hard Requirement). When real images are needed (historical figures, famous paintings, artifacts), use components/vintage-archive/scripts/fetch_commons.py to download from Wikimedia Commons. This script also generates ATTRIBUTION.md, recording the author, license, and source page for each material:

Public domain: No attribution required, but it is recommended to keep the source in the end credits.
CC BY / CC BY-SA: Attribution is mandatory — include "Work name · Author · Source page · License name" in the end credits or description.
CC BY-SA: In addition to attribution, derivative works must be distributed under the same license.

After downloading, view the image before writing the scene (see Step 7 pitfalls). Before delivery, check against ATTRIBUTION.md line by line to confirm all attributions are in place. Fonts are downloaded by setup_assets.sh and are all OFL 1.1 / Apache 2.0, usable and embeddable for commercial use, but the license notice must be included with the final product.

Text readability rule: All text must be placed inside decorative elements — no text may float outside the paper, card, or banner. Stamps and emphasis labels should use high-contrast color schemes (e.g., yellow background with red text, dark background with light text); do not directly overlap semi-transparent red text on paper.

Music and Sound Effects: If the user has a local asset library (e.g., a sound effects folder on the D drive), first ask for the path, then pick background music from the library based on mood, and pick sound effects based on actions (page turn, paper, Polaroid, pencil, whoosh transition, drum hit, bass thump, ocean wave, etc.). Choose a background music segment that can naturally conclude at the end of the video. Only fall back to programmatic sound synthesis when there is no asset library.

Step 7: Delivery and Modifications

Export Versions: Before rendering, ask the user which audio versions to export (multiple selections allowed, sound effects included in all versions):

Full version: voiceover + music + sound effects
No voiceover version: music + sound effects (suitable for users who want to re-record their own voiceover)
No music version: voiceover + sound effects (suitable for users who want to add their own royalty-free music from a platform)

Method: Render the video visuals only once (muted), then synthesize different audio tracks separately — do not render multiple times. Scripts in components/common/scripts/ are ready: mix_audio.py outputs three audio tracks at once, and export_versions.sh handles the final composite export. Use the same approach for other templates.

When delivering, inform the user:

Final video file location, duration, aspect ratio, whether subtitles are included, which audio versions were exported
Which template was used, what the main colors are
A one-sentence prompt: "If any scene is unsatisfactory, just tell me, for example, 'I want a different style for the ending globe' — I'll only modify that section."

When the user requests modifications:

First confirm which scene number it is (if unsure, look up the description in the storyboard table).
Only modify that scene's component file; leave all other scenes untouched.
After modifying, first send a screenshot to the user for confirmation, then re-render the full video.

When Updating This Skill Itself

If the user requests changes to the Skill's content (adding templates, adding components, modifying documents) and pushing to GitHub:

Every push must have a new tag, with version format vMAJOR.MINOR.PATCH, consistent with the metadata.version in the frontmatter.
Do not manually run git tag / git push / gh release; instead, use tools/release.py:

python tools/release.py --bump minor --note "Description of this update"

This script will sequentially: update the version number → update CHANGELOG.md → repackage .skill → commit → create tag → push → create a GitHub Release (with the distribution package).

After making changes, do not ask the user "Should I publish?" — simply execute the command above.

Pitfalls Encountered in Testing (Learned from the first 27-second Vintage Archive video; following these will save many iterations)

Fetching Materials

Filenames on Wikimedia Commons can be deceptive. A file titled Winged Victory of Samothrace might actually be a close-up of the inscription on the wall. After fetching an image, always view it first before writing the scene — do not assign visuals based solely on the filename.
Vertical composition photos cannot be displayed full-screen (FullPainting / objectFit:cover will crop the subject). For vertical images, always use a photo card (PhotoCard) — the Vintage Archive template's default is "placing photos on a table," not filling the entire screen.

Writing Scenes

When placing multiple lines of text inside a container with curved or irregular edges, do not rely on guesswork based on flex centering. The lower edge of a ribbon (e.g., Ribbon) is a curve that arches upward in the middle — the subtitle once fell entirely outside the paper boundary. Always check against the height at the "narrowest point of the curve," or render a single frame and crop and magnify 2x to visually confirm the whitespace.
Leave enough stabilization time for element entrances: when a scene is less than 60 frames, elements entering via Drop (spring) may not have settled before the scene cuts (e.g., a paper piece flies off the right edge of the screen). After the incoming paper's at frame, leave at least 30 frames of static time.

Running Scripts

The entry parameter for export_versions.sh must be the file containing registerRoot() (typically src/index.ts). Passing Root.tsx will cause the error "does not contain registerRoot."
Before rendering, extract 3–5 keyframe screenshots for self-check — do not render blindly to black. The self-check method is to crop and magnify local areas to look for text overflow, rather than looking at thumbnails of the full frame — a 10px overflow invisible in the full frame becomes obvious when magnified.

newbaisays-aigc/knowledge-animation-video

knowledge-animation-video 是什么？

在你喜欢的 AI 中提问

文档

Knowledge Animation Video Generation

Step 1: Opening Selection (Must be done first, do not skip)

Step 2: Read the Template

Step 3: Analyze the Reference Image

Step 4: Process the Voiceover, Determine the Timeline

Step 5: Create the Storyboard and Wait for User Confirmation (Do not render directly)

Step 6: Generate the Video

Step 7: Delivery and Modifications

When Updating This Skill Itself

Pitfalls Encountered in Testing (Learned from the first 27-second Vintage Archive video; following these will save many iterations)

Pre-Delivery Checklist (Review this list before every full video render)

相关技能

steipete/video-frames

steipete/summarize

openclaw/openclaw

obra/superpowers

affaan-m/ECC

affaan-m/everything-claude-code