Orinkle/ideaform

Constructivist Explainer Video · Workflow

One master timeline, all three downstream stages derive from it.

① Script            ② TTS Set Axis                ③ BGM Variation            ④ Final Video
   Text      ──→   Narration Track (Master Axis) ──→   LMMS Music (Levels)   ──→   Remotion Visual + Audio Integration + Cover
                          │                        ▲
                          └──── Level Boundary Timestamps ────┘

• Video frame count is determined by natural TTS speech rate (not the other way around by compressing speech)
• BGM variation points align with the argument level switch points in the narration
• Final video aligns with the narration track as the reference; BGM serves only as background

---

0. Getting Started: Read the Path Contract (This step is mandatory)

This skill does not locate tools on its own; all paths come from the contract written by the environment setup skill:

$VW_CONTRACT  (default ~/.config/ideaform/toolchain.json)

CONTRACT="${VW_CONTRACT:-$HOME/.config/ideaform/toolchain.json}"
source "$(dirname "$CONTRACT")/env.sh"     # Exports VW_NODE / VW_VENV / VW_FFMPEG / …

Three things to check before starting:

1. Does the contract file exist? No → Ask the user to run the ideaform-setup skill first.
2. Is schema_version the version recognized by this skill? Mismatch → Require re-running the environment setup skill.
3. Is verified_at null? Yes → Warn the user "Environment has not passed smoke test"; fix it before proceeding. Additionally, tools.lmms.stkdir_ok must be true, otherwise BGM will be silent throughout with no error.

Key fields quick reference:

Purpose	Contract Field
Run TTS script (edge-tts inside venv)	tools.venv.python
Combine audio/video	tools.ffmpeg.path / tools.ffprobe.path
Render visuals	tools.remotion.project / tools.remotion.browser.resolved
Render BGM	tools.lmms.apprun
Temp file location (must be a large disk)	tmpdir

⚠️ The tmpdir in the contract is not optional. The local /tmp only has 10MB; Remotion packaging / pip installations will write to it → ENOSPC. Every Remotion command must include TMPDIR.

⚠️ Do not resolve any paths (symlink resolution). Managed toolchain paths contain symlinks; resolving the venv interpreter will lose the venv context, making all packages unimportable. See repo root docs/03-PathContract.md.

---

1. Four-Stage Overview

Stage	What It Does	Input	Output (all goes to workspace)	Reference
① Script	Transform user input into a layered narration script	User input	script.md + lines.txt (with levels marked)	references/stage1-script.md
② TTS Set Axis	Measure natural speech rate → build timeline → produce voice track	lines.txt	plan.json / timeline.ts / voice.wav	references/stage2-tts.md
③ BGM Variation	Level boundaries → variation plan → LMMS audio track	plan.json + level annotations	bgm-plan.json / bgm.wav	references/stage3-bgm.md
④ Final Video	Remotion arrange visuals + integrate audio + render + cover	timeline.ts + audio + bgm-plan.json	final.mp4 + cover.png	references/stage4-render.md · references/cover.md

Style consistency is governed by references/constructivist-style.md, visuals and cover share the same palette / fonts / grid, ensuring natural coherence.

---

2. Stage One · Script (Levels are Mandatory)

The user provides input (a topic / a piece of material / a verbal idea). Outputs:

• script.md — Full script, explicitly marking argument levels
• lines.txt — One line per spoken sentence, for TTS consumption

Levels are not for aesthetics; they are input for Stage Three. Without level annotations, BGM variations have no anchor point.

## Level 1 · Introduction
This is the first level of the "Script as Timeline" approach.
The narration track is the master time axis for the entire pipeline.
## Level 2 · Development
Frame count is determined by it, not the other way around.
...

• One line = one semantic unit; do not put two ideas in one line (TTS aligns by line; merging them will stretch the slot)
• Recommended line length: 8–30 characters; split longer lines, merge shorter ones
• End punctuation with period or question mark to help TTS phrasing

See details in references/stage1-script.md.

---

3. Stage Two · TTS Set Axis (Strict Order Mandatory)

"$VW_VENV/bin/python" skills/ideaform/scripts/make-tts-track.py \
    --lines lines.txt --fps 30 --audio \
    --plan-out out/plan.json --ts-out out/timeline.ts

Produce audio first, set the axis second. The timeline grows from the actual TTS speech rate:

① Write lines
② make-tts-track.py --lines lines.txt --fps 30 --audio   ← Measure natural rate, build timeline
③ Use the output TOTAL_FRAMES to write the Remotion composition   ← Frame count determined here
④ Then render

Cost of Violation (Real Measurement): Pre-defining 43 time codes based on storyboard, with TTS natural rate totaling 212.90s, but video only 174.80s → forced 1.218x compression. This compromise was entirely avoidable.

The same script can also do the reverse (when SRT or time codes are fixed, use two-step tempo adjustment as a workaround), but that is remedial, not preferred. See details in references/stage2-tts.md.

⚠️ Do NOT ask the user for "final video duration in minutes". Duration = speech rate × line count, it is an output, not an input; at the start of work, you should neither ask nor offer options like "Standard version X minutes". Only report "Based on this script, measured X min Y sec". If the user wants longer/shorter → edit the script, not the timing. (Recorded 2026-10-15 with the Osmanthus video, see E16)

Outputs: plan.json with each cue having start/end/dur/frameStart/frameEnd; timeline.ts can be directly cp-ed into the Remotion project as src/timeline.ts.

---

4. Stage Three · BGM Variation (The Soul of This Workflow)

Key: Each time the argument enters a new level, the music changes its character — but does not change the piece.

# ① Map script levels to cue numbers, marking each level's narrative function
"$VW_VENV/bin/python" skills/ideaform/scripts/bgm-plan.py \
    --plan out/plan.json \
    --layers "1:present,4:develop,6:resolve" \
    --key "A minor" --out out/bgm-plan.json

# ② Generate LMMS project based on bgm-plan.json and render (uses composer.py from electronic-music-production)
#    Rendered using tools.lmms.apprun from the contract; bpm and timeline lp1pos must follow mmp_hints

Six narrative functions (pick one per level):

Function	Intent	What Changes
present	Present · Introduce	Show motif only, single voice, space
develop	Develop · Deepen	+bass +groove, motif sequence
intensify	Emphasize · Conflict	Full voice, octave up, borrowed chords, dynamic boost
contrast	Contrast · Turn	Change texture without increasing volume: remove layers, change timbre, modulate mode
resolve	Resolve · Return	Reduce, return to tonic, augment motif
sustain	Sustain · Transition	Do nothing (if level hasn't changed, don't vary randomly — it dilutes contrast)

Ironclad Rules:

• Identity elements must not change — motif cell, tonal center, base tempo. Changing them means changing the piece, not varying.
• Each section must change at least 1 morphological element; bgm-plan.md will list the "variation evidence" for each section. If adjacent sections have empty evidence, it means a level annotation is missing.
• Variation points must land on argument level boundaries (use --lead-in to let music start slightly early).

Full method and LMMS implementation details in references/stage3-bgm.md.

---

5. Stage Four · Final Video

5.1 Visuals: Use Narration Track as Master Timeline

cp out/timeline.ts <remotion-project>/src/timeline.ts     # Frame count/scene boundaries derived from CUES
cd <remotion-project>
export TMPDIR="$VW_TMPDIR"                             # ★ Mandatory
npx remotion studio                                    # Show preview to user first

Do not hardcode scene frame numbers on the visual side; derive them from CUES:

export const SCENES = {
  s1: { from: 0,            dur: CUES[2].start },
  s2: { from: CUES[2].start, dur: CUES[4].start - CUES[2].start },
} as const;

Structural / signal pathway / mechanism diagrams → always deterministic generation, never AI image generation. AI will inevitably hallucinate chemical structures and you won't notice the errors — these materials must be reproducible and verifiable. Use scripts/gen-structures.py: SMILES → RDKit deterministic rendering → frame-driven Remotion components.

python3 scripts/gen-structures.py --tsv structures.tsv --out structures
#   → svg/*.svg  +  components/*.tsx (with built-in animation)  +  manifest.json
#   → Offline molecular formula verification (RDKit CalcMolFormula); exits non-zero on mismatch, can be chained with `&&`

• Use RDKit official options for color (useBWAtomPalette / singleColourBonds / setSymbolColour), do NOT post-process SVG colors with regex — that's reinventing the wheel and breaks with RDKit updates (see error log E20).
• Generated components use CSS variables --mol-dash / --mol-text for line animation; outer context must provide color.
• For broader molecular visualization needs (grid diagrams / skeleton highlighting / 3D conformations / protein-ligand interactions), look for existing skills first (see §7); don't force them into this video pipeline.

5.2 Audio Integration (Two Routes, Default A)

A. Remotion Integration (Recommended) — use <Audio> in the composition; the rendered mp4 will have built-in audio:

<Audio src={staticFile("voice.wav")} />
<Audio src={staticFile("bgm.wav")} volume={0.1694} />

Don't guess that volume number; derive it from measured RMS:

bash scripts/mix-av.sh --plan-only --voice voice.wav --bgm bgm.wav --drop 9
# Narration RMS = -23.46 dBFS   BGM RMS = -17.04 dBFS
# → BGM needs -15.42 dB gain  →  volume = 0.1694

B. Post-Processing (video already rendered / need to re-dub):

bash scripts/mix-av.sh --video silent.mp4 --voice voice.wav --bgm bgm.wav --out final.mp4

⚠️ Route B must explicitly use -map 0:v:0 -map 1:a:0. Remotion output mp4 contains a silent AAC track; without -map, ffmpeg will automatically select it → final file exits 0, correct duration, plays, but has no sound. mix-av.sh already has this -map built-in.

5.3 Rendering and Cover

export TMPDIR="$VW_TMPDIR"
npx remotion render <CompositionId> out/video.mp4 --concurrency=3
npx remotion still Cover out/cover.png --props='{...}'    # Cover: static composition, do NOT use animations

• Long video segment rendering: A 7-minute video rendered in one pass may timeout due to memory exhaustion. Use --frames=A-B to render segments → concat -c copy lossless merge.
• Cover uses a static composition with durationInFrames={1}; do not use useCurrentFrame inside its components (during still render it's always 0, animation initial values will make elements invisible).
• Cover layout details in references/cover.md, style guide in references/constructivist-style.md.

---

6. Headless Delivery Verification: Each Stage Must Pass Validators

The model cannot see images or hear audio. So "render succeeded" ≠ "visuals correct, audio correct." Each step uses numerical measurements; if it doesn't pass, reject delivery:

# Audio self-check (TTS track, BGM): peak/RMS/clipping/duration deviation
# Visual self-check: pixel color matching / corridor purity / text pixel count
#   --layout must match: cover=cover layout; scene=video scene (default).
#   Sampling zone depends on layout; using wrong layout will fail even a correct cover (see 05-Troubleshooting D2).
python3 scripts/verify-video.py cover out/cover.png --layout cover

# Final video self-check: frame count / duration / audio track bitrate (prevents silent track)
# ffprobe resolution order: VW_FFPROBE → path contract toolchain.json → system
VW_FFPROBE="$VW_FFPROBE" python3 scripts/verify-video.py video out/final.mp4 --frames 732 --fps 30

Scripts indicate failure with a non-zero exit code, allowing direct chaining with && in the pipeline.

Common criteria quick reference:

Object	Criterion	Threshold
Audio	Non-silent	Peak > 1000 and RMS > 100
Audio	No clipping	Full-scale samples < 0.01%
Audio	Duration match	Deviation from plan < 50 ms
Cover	Dimensions	1920×1080
Cover	Text rendering	Ink pixels in text area > 300
Cover	Axes not crossing text	Red pixels within title area < 60
Final video	Frame count	Strictly equal to planned value
Final video	Audio track non-silent	Bitrate > 64000 bps
BGM	Background level	8–12 dB lower than narration

---

7. Complementary Skills for This Skill

What You Need	Which to Use
Environment detection / installation / contract	ideaform-setup (one-time)
Existing skills for new open-source tools (harvest: GitHub scrape → security check → save)	ideaform-setup's scripts/harvest-skills.py <tool-name>
Deterministic graphics for chemical structures / pathway diagrams	This skill's scripts/gen-structures.py (RDKit → SVG → Remotion components)
Broader molecular visualization (grid diagrams / skeleton highlighting / 3D conformations / protein-ligand)	External existing skills: synthetic-sciences/openscience's molecule-visualization (Apache-2.0), ghutchis/chem-skill (MIT)
LMMS project format, plugin parameters, eldata envelopes	electronic-music-production + its references/05
Compositional methods (harmony, motif, non-hearing composition, emergence)	electronic-music-production's references/06–10
Remotion syntax and rendering details	remotion-best-practices
Complete headless visual verification methods	headless-visual-verify
WSL environment pitfalls (/mnt, gh fetch, audio pipeline)	wsl-pitfalls

---

8. Delivery Checklist

After completing one video, the workspace should contain:

<workspace>/
├── script.md            Layered script
├── lines.txt            Spoken lines
├── out/
│   ├── plan.json        Master timeline (seconds/frames for each cue)
│   ├── timeline.ts      Timeline ready to copy into Remotion
│   ├── voice.wav        Narration audio master
│   ├── bgm-plan.json    Variation plan + bgm-plan.md (variation evidence)
│   ├── bgm.wav          BGM audio track
│   ├── video.mp4        Silent video (or final video with integrated audio)
│   ├── final.mp4       ★ Final video
│   └── cover.png       ★ Cover image
└── .tts/                TTS cache (keep; re-running takes only seconds)

Must run all validations from §6 before delivery, and truthfully report which items passed and which were not tested.

---

9. Self-Evolution: Log Every Pitfall

This workflow is a mutable composite skill — the more you use it, the more it should adapt to your production rhythm:

• When you hit a pitfall and fix it → Log it in repo root docs/06-ErrorNotebook.md (six-column format, sequential numbering). ⚠️ Do not create another docs/ under skills/ideaform/ — that's a second ledger that will collide with the original numbering (recorded 2026-10-15, see E19).
• When a pitfall is permanently fixed by a script / documentation (not worked around) → Change the entry status to Permanently Fixed and describe the fix.
• When the same type of pitfall occurs a second time → Don't log a third entry; build the defense directly into the script (validator / default value / contract).
• Introducing a new open-source tool → Harvest before building: harvest-skills.py <tool-name> to find community mature skills (with fork protection, won't overwrite your local edits); only write your own if no qualified one exists. Log pitfalls from harvested skills just the same.
• Before ending a session, scan: are there any "fixed but not logged" items?