threerocks/sweety-statechart-video

Sweety · Statechart Video

Step by step, make things clear.

The standard is the reference works.

• Default look: Follows Ann Nguyen's the making of nhang (dynamic statechart on TypingMind; her video is a screen recording of that page).
• Rule checking and multiple drawing styles: Learns from Kevin Ngo's statechart-art (MIT).

The drawing engines and players are our own. When in doubt, refer back to the reference works and follow them.

Deliverables:

• outputs/film.mp4: Default 3:4 ratio (1080×1440), 30 fps, 2–3 minutes, with narration, ready for sharing; also outputs/film-master.mp4, an upload master at ~11 Mbps giving headroom for platform re-encoding;
• outputs/posters/01.png…: One full poster per chapter, plus 01-layout.png showing layout with blue outlines;
• outputs/cover.png, script.txt, editing/captions.srt;
• On demand: chapter clips, long images, interactive web pages, portfolio pages, and publishing packages (see Deliverables).

What the Film Looks Like

• Cover intro. Default: show outputs/cover.png for 0.5 seconds, then fade out over 0.5 seconds, followed by the main film. The intro is silent; the film's audio and subtitles are shifted back by 1 second.
• Scrolling paper. Each chapter is one poster; they connect end to end: the exit arrow of the previous chapter extends from the bottom edge into the next chapter's top.
• Poster:
  • Top-left: header with series, main title, subtitle, and a line of small red text;
  • Center: statechart, defaulting to cream-colored note cards with pushpins and small red labels;
  • Surroundings: large colored crayon illustrations.
• Each step:
  • A small dot runs along the new arrow, characters type out one by one;
  • The card squishes and bounces up, a pushpin falls and pins it, the card wiggles twice;
  • Current state turns pink, a pink circle draws around it like crayon;
  • Camera follows the action, can zoom in.
• Always moving:
  • Lines redraw 5 times per second;
  • Illustrations move with the state;
  • Characters mouth along with narration and point to the current card.
• Subtitle bar I.3 — Payment done, store beep received`: read words turn darker. For chapters with "same time" branches, an extra line shows current position on each branch.
• Narration: The subtitle of each step is the spoken line, read by our own synthetic voice, see Voices.
• Rule chapters:
  • Write one or more rules; the checker finds the shortest path that breaks each rule, draws red lines, then stamps it;
  • Poster has a legend: ✓ or ✗ for each rule, and how many cases were checked;
  • "Fix after breaking": add a condition, red line fades, green stamp appears.
• Beginning and ending: Can start with a question (hook); the ending defaults to scrolling through all posters and stopping on an end card.

How to Run (All Platforms)

• All commands use absolute paths: python3 <Skill directory>/scripts/scv.py <command> <job directory>. The job directory is placed in the user's working directory, not inside the Skill directory.
• First time on a new machine: run bash <Skill directory>/scripts/setup.sh: builds rendering environment, installs Chromium (uses existing Chrome if found), finally runs smoke tests.
• Voice generation runs only on Apple Silicon (MLX). If the environment is incomplete, voice will report what's missing; when the user doesn't need narration, write "voice": "none".
• voice and render each take tens of seconds to minutes. If platform limits command duration, run them separately or in the background.
• Tested on Codex, Claude Code, and Grok Build; platform-specific notes are in Platforms.

Workflow

1. Break down the story. Read Story Design, split into 2–5 chapters, each with one shape: flat, parallel branches, a parent state, or a rule chapter.
2. Write film.json, format per film.json Spec (has JSON Schema).
   • scv.py new JOB --title … creates a skeleton;
   • When an existing state machine (XState, Mermaid, SCXML, PlantUML, or /statechart skill's chart directory) exists, use scv.py import file JOB to start from a skeleton, then edit text and subtitles, see Import & Export;
   • Complete examples in examples/.
3. scv.py check JOB: Checks structure, steps, subtitle lengths, reachability. Rule chapters find red lines; fix then validates. Stop on any error.
4. scv.py layout JOB: Fully automatic layout.
   • Chapter posters rendered in work/layout/chN.png with a 100-unit grid and empty zones outlined with a green dashed line;
   • Coordinates in chN.json.
5. Write scenes.js: Each chapter has one bg (static) and one fg (drawn per frame, animating based on state).
   • scv.py draft JOB first creates a draft based on empty zones and state names (scene elements plus props), then refine;
   • Props, characters, scene elements, and brushes are in Scenes, type definitions in scenes.d.ts;
   • scv.py gallery outputs a catalog of all props and scene elements.
6. scv.py lint JOB: Reports illustration issues one by one, with fixes. Fix until no issues:
   • Cards, text, or arrows overlapping;
   • Too much empty paper (crayon theme >38%, pencil/linen etc. spacious themes >55%);
   • Illustration unchanged across a step;
   • Character proportions and ground line mismatched.
7. scv.py voice JOB: Generate voice per sentence, passing noise gate and ASR validation. Results are cached; only redo changed sentences.
8. scv.py plan JOB → stills JOB: Open work/stills/contact.jpg for step-by-step inspection. Also scv.py preview JOB for interactive preview in browser, auto-refresh on file changes.
9. scv.py all JOB: Voice, timing, rendering, posters, final film, deliverables, and publishing package in one run.
   • Tested: 93-second film renders in ~40 seconds, 36-sentence narration ~3 minutes;
   • Examples: examples/intro/ (2–3 min), examples/takeout/ (1.5 min), examples/message/ (1 min).
10. Batch: One directory per topic, run steps 4, 6, 8 for each, then scv.py batch JOB1 JOB2 ….

Hard Rules

• Cards and arrows:
  • Cards show visible description, 2–6 characters (e.g., "Delivering", "Ruined"), tag for one-line small text (preferably English);
  • Arrows describe what happens, within 8 characters; conditions in if, displayed in red [has ingredients];
  • Conditions the checker should also understand: add when (with vars, do).
• Size: Max ~8 cards and ~12 arrows per chapter; split into more chapters if needed. Posters automatically increase height when vertical space is insufficient; check will suggest splitting.
• Subtitles: One line per step, max 24 Chinese characters (or ~42 English characters), tell cause and effect, do not read cards verbatim. They double as narration: numbers written as words, or use say to give pronunciation.
• Step writing:
  • First step is "go": "start"; for chapters following a previous chapter's exit, first step is "go": "entry";
  • Cards only seen at the end must have a step that highlights them, otherwise they won't appear;
  • Rule chapters: step through to the violation state, then use {"trace": true} for one step drawing red lines;
  • Fix with rule.fix, then add a step {"fix": true};
  • List rules with rules.
• Illustrations:
  • Use every empty zone; one poster, one big scene (scene element), placed at bottom or sides;
  • Each state has a corresponding illustration that moves with it (S.on, S.since, S.step), appears with $.enter;
  • Characters 0.7–0.9× (protagonist can be 1.5×), standing on the same ground line; do not overlap cards, arrows, or text;
  • All drawn in code, no external images;
  • lint must pass before delivery.

Defaults and Switches

• Aspect ratio 3:4 (1080×1440). Any "aspect": "W:H" works, short side 1080:
  • Portrait and square: posters and frames match shape; card and arrow text auto-enlarge for readability on mobile;
  • Landscape: if no "same time" frame, layout left to right, posters and frames match shape, auto-widen if too long; if there is a frame, use square posters with camera scrolling;
  • Re-run layout after changing aspect ratio, illustrations reposition to new empty zones.
• Duration default 2–3 minutes, suitable for Xiaohongshu, Douyin, etc. Users specify "duration": 150, [60, 90], "1 minute" all accepted.
  • check estimates duration based on narration speed and tells you to add or remove steps; plan adds up to 0.7 seconds per step when close to target;
  • To achieve 2–3 min: typically 4–5 chapters, 9–12 steps each, 12–20 characters per sentence;
  • Cover intro added by finalize, adding 1 extra second; turn off with "cover_intro": false. work/timing.json is main film timing; delivered subtitles and outputs/editing/delivery-timing.json include intro.
• Speed speed:
  • Default 2 with narration (animation ~1.8s per step, then wait for narration to finish);
  • Default 3 without narration, matching reference video, animation 1.2s, each step pauses long enough to read.
• Looks, see Looks:
  • "theme": crayon (default), pencil (pen & colored pencil), night, kraft, riso, linen, season:autumn etc.; "brand": [brand colors] generates a theme from brand colors;
  • "look": card drawing style, ticket / note / sign / soft / plate / tag / slide / patch; parent state can be drawn as a storefront;
  • "arrows": thread / cord / pencil;
  • Camera and transitions: "camera": "push", "hook": "question", "outro", "episode" and "next", "player": {"spotlight": true};
  • State: mood (rain, night, alarm, fixed) and inset (microscope window).
• Voice "voice":
  • Default crayon-sister, alternatives blueprint-guy, sunny-girl, old-master, "none" for silent. All four are fully synthetic, no copyright issues;
  • Custom voice: scv.py add-voice my-voice --audio recording.m4a --start 12 --end 25 --text "transcript" --consent self --preview, then "voice": "my-voice", see Voices;
  • Steps can have tone (e.g., "surprised", "very slow") and speaker (that character mouths along);
  • States can have sfx (sound on highlight) and ambience (ambient sound while active);
  • "music": {"mood": "curious", "sync": true} adds instrumental layer, automatically ducked under narration, synced to strong beats at chapter starts and stamp actions;
  • Other languages: translations in i18n.<language>, scv.py all JOB --lang en produces English version.
• Watermark default Made by: @leo_xiaolei/Mr. Liu, pinned top-right; "watermark": null turns off.
• AI label: With narration, shows "Narration by AI" in the first 4 seconds and on end card ("ai_label"); film, audio, and poster metadata include GB 45438 AIGC field ("aigc" can be changed).
• Fonts (bundled, all redistributable):
  • Special Elite (Apache-2.0) with Noto Serif SC Bold subset (OFL), used in titles and cards, with typewriter ink effect;
  • Courier Prime (OFL) with LXGW WenKai (OFL), used in arrow text and subtitles;
  • Theme changes: use ZHS Song, LXGW Man-Hei (OFL) or Kevin Serif/Sans (MIT).

Running and Boundaries

• Rendering uses Chrome + Playwright (SCV_PYTHON, default to venv created by setup.sh at ~/.local/share/sweety-statechart-video/venv).

• Narration uses local VoxCPM2 MLX and Whisper (STATECHART_MLX_PYTHON), entirely offline; scripts/audio_doctor.py checks environment.

• Layout is automatic. Individual cards can be pinned with "at": [col, row]. Only one level of parent state supported; deeper nesting is split into chapters during import.

• scv.py export JOB --to mermaid|xstate|scxml exports the diagram back to documentation and code.

• More deliverables:

  • scv.py deliver JOB --what clips,longimage,social,gif,html,site: chapter clips, long images, platform-sized images, animated GIFs, interactive web pages (step-by-step mode, spotlight, microscope), portfolio pages;
  • When film.json has "deliver": [...], all includes them;
  • scv.py publish JOB creates publishing package.

  See Deliverables.

• Rendering issues: add --encoder legacy, for speed add --encoder webcodecs (hardware encoding).

• After engine changes: run scv.py golden to compare against reference frames, python3 scripts/test_engine.py to test various aspect ratios, themes, and styles.

• The checker is this project's own explicit state checker (inspired by TLC), does not run TLA+; results checked by TLC from /statechart skill can be imported directly.

• Synthetic narration is AI-generated content; label per platform requirements when publishing.

• Acknowledgments and licenses are in THIRD_PARTY_NOTICES.md in the repo.