Remotion Video — Encode the Actual Frames
You are the encoder and renderer. You take assets — a recording, a voiceover, b-roll, a transcript — plus React code, and you emit a real file: out/video.mp4. Your siblings write words and plans; you are the only one that produces pixels. The rigor here is reproducible frames: same input, same deterministic output, verified by a render that ffprobe can read.
You own the Remotion project scaffold, the <Composition> / <Sequence> / <TransitionSeries> graph, the captions pipeline (Whisper.cpp → toCaptions → createTikTokStyleCaptions), the silence-removal pass, and the npx remotion render invocation with its codec and concurrency flags.
The one decision: frames or words?
If the ask is to produce a file, you are in the right place. If it is to produce text or a plan, route out before writing a single .tsx.
| The ask | Goes to | Why |
|---|---|---|
| Produce an MP4/MOV, transitions, burned captions, render | here | These are pixels and frames |
| Write the script, hook, beats, on-screen caption text, edit decision sheet | ../video-shorts/SKILL.md | Those are words; the cut decisions, not the cut execution |
Master audio to a LUFS target, produce chapters + RSS <item> | ../podcast/SKILL.md | Audio mastering and feed, not video encode |
| Structure the lesson/video narrative arc and flow | ../course-storytelling/SKILL.md | Narrative architecture, not rendering |
| Design the thumbnail image | ../youtube-thumbnails/SKILL.md | A still image, not a video |
Boundary in one line: video-shorts decides the cuts and writes the caption text; you execute the cuts in code and burn the captions into frames.
Scaffold the project
npx create-video@latest --yes --blank my-video
cd my-video
npm i
npm run dev # opens Remotion Studio in the browser
Remotion's current stable line is 4.0.471; it needs Node 16+ (or Bun 1.0.3+), and local rendering targets macOS 15 (Sequoia)+. Since January 2026 Remotion ships Agent Skills — npx skills add remotion-dev/skills wires Remotion-aware guidance into Claude Code. Run it inside a Remotion project when you want the framework's own skill loaded alongside this one.
Pin fps and dimensions on <Composition> first, and never change them mid-project. Why: every duration downstream is measured in frames, and frames = seconds * fps. Change fps after you have written durations and every timing silently shifts. Vertical shorts are 1080×1920 @ 30; landscape is 1920×1080 @ 30.
The composition graph
// src/Root.tsx
import { Composition } from "remotion";
import { MyVideo } from "./MyVideo";
export const RemotionRoot: React.FC = () => {
return (
<Composition
id="MyVideo" // the id you pass to `remotion render`
component={MyVideo}
durationInFrames={300} // 10s at 30fps
fps={30}
width={1080}
height={1920}
/>
);
};
// src/MyVideo.tsx
import { AbsoluteFill, Sequence, useCurrentFrame, interpolate, spring, useVideoConfig } from "remotion";
export const MyVideo: React.FC = () => {
const frame = useCurrentFrame();
const { fps } = useVideoConfig();
const opacity = interpolate(frame, [0, 30], [0, 1], { extrapolateRight: "clamp" });
const scale = spring({ frame, fps, config: { damping: 200 } });
return (
<AbsoluteFill style={{ backgroundColor: "black" }}>
<Sequence from={0} durationInFrames={90}>
<AbsoluteFill style={{ opacity, transform: `scale(${scale})` }}>{/* scene 1 */}</AbsoluteFill>
</Sequence>
<Sequence from={90} durationInFrames={210}>{/* scene 2 */}</Sequence>
</AbsoluteFill>
);
};
Animate off useCurrentFrame() with interpolate() and spring() only. Never read wall-clock time (Date.now()) or call Math.random() unseeded inside a composition. Why: rendering is parallel and frame-addressable — each frame is computed independently, so any non-frame input produces a different pixel on re-render and breaks the "same input, same output" guarantee. If you need randomness, use Remotion's random(seed).
Transitions
Use @remotion/transitions (available since v4.0.53). <TransitionSeries> interleaves .Sequence (a clip, with durationInFrames) and .Transition (a presentation + a timing). The transition duration is subtracted from the total, so adjacent sequences overlap during the wipe.
import { TransitionSeries, linearTiming, springTiming } from "@remotion/transitions";
import { slide } from "@remotion/transitions/slide";
import { fade } from "@remotion/transitions/fade";
import { Easing } from "remotion";
<TransitionSeries>
<TransitionSeries.Sequence durationInFrames={90}>{/* scene A */}</TransitionSeries.Sequence>
<TransitionSeries.Transition
presentation={slide({ direction: "from-left" })}
timing={springTiming({ config: { damping: 200 }, durationInFrames: 30, durationRestThreshold: 0.001 })}
/>
<TransitionSeries.Sequence durationInFrames={120}>{/* scene B */}</TransitionSeries.Sequence>
<TransitionSeries.Transition
presentation={fade()}
timing={linearTiming({ durationInFrames: 15, easing: Easing.inOut(Easing.ease) })}
/>
<TransitionSeries.Sequence durationInFrames={90}>{/* scene C */}</TransitionSeries.Sequence>
</TransitionSeries>
Each presentation is a sub-import (@remotion/transitions/slide, /fade, /wipe, /flip, /clockWipe, /none).
| Presentation | Feel / when |
|---|---|
slide | Scene pushes the next in; directional momentum between beats |
fade | Soft crossfade; calm, neutral scene change |
wipe | A hard edge sweeps across; energetic, "next topic" |
flip | 3D card flip; playful, for reveals |
clockWipe | Radial sweep; countdowns, "time passing" |
none | A hard cut with no motion, but still as a TransitionSeries node |
linearTiming for predictable, frame-exact cuts; springTiming for organic motion. Why: linear is deterministic in duration so you can budget frames exactly; spring overshoots and settles, which reads as natural but needs durationRestThreshold so the render knows when it has finished.
Animated burned-in captions
The native @remotion/captions package shipped in v4.0.216 (the same release that deprecated the old convertToCaptions() helper). The pipeline runs once on a Node server, then the composition reads the captions:
- Transcribe.
@remotion/install-whisper-cppdownloads Whisper.cpp and a model (medium.enis ~1.5 GB) and transcribes the audio on a Node server to Whisper JSON. - Convert.
toCaptions()from@remotion/install-whisper-cppturns that JSON into aCaption[]with per-token timestamps. (convertToCaptions()is the legacy alias — deprecated as of v4.0.216; usetoCaptions().) - Segment into pages.
@remotion/captionscreateTikTokStyleCaptions({ captions, combineTokensWithinMilliseconds })groups tokens into "pages" that appear together.
The combineTokensWithinMilliseconds value is the page-size dial. Why: a low value (~200ms) keeps each word on its own page → word-by-word pop animation; a high value (~1200ms) packs a phrase per page. Low ms = TikTok word-by-word energy; high ms = readable phrases. Pick by the format, not by default.
Full Whisper.cpp install, the transcribe server, and the token-highlight caption renderer component (with safe-zone styling) live in references/captions-pipeline.md — read it before building the captions layer.
Automatic silence removal
The silence pass runs on the source audio/video BEFORE it enters Remotion, not inside a composition. Why: Remotion renders frames you give it; trimming dead air is an upstream edit on the asset, and doing it first means every downstream frame number already reflects the tightened timeline.
Use auto-editor (a Python + ffmpeg engine) for a first pass that cuts dead space by audio loudness:
auto-editor input.mp4 --margin 0.2s --edit audio:threshold=4% -o tightened.mp4
--marginpads each kept region so cuts do not clip speech.--edit audio:threshold=4%sets the loudness floor below which a region is "silence".--export premiereemits an EDL/XML instead of a file, to re-import into an NLE.
pip distribution is stale/discontinued — install via the official binary or pipx, not pip install. Why: the PyPI package lags behind and may not match the documented flags. When auto-editor is unavailable, the low-level fallback is ffmpeg's silencedetect / silenceremove filters. Both, with the full flag matrix, are in references/render-and-pipeline.md.
B-roll overlays
Stack the overlay above the main video by layering <OffthreadVideo> (or <Img>) inside an <AbsoluteFill>, gated by a <Sequence from>:
import { AbsoluteFill, Sequence, OffthreadVideo, staticFile } from "remotion";
const fps = 30;
const broll = { start: 4.0, duration: 3.0 }; // seconds
<AbsoluteFill>
<OffthreadVideo src={staticFile("main.mp4")} /> {/* base layer */}
<Sequence from={Math.round(broll.start * fps)} durationInFrames={Math.round(broll.duration * fps)}>
<AbsoluteFill style={{ /* e.g. inset for picture-in-picture */ }}>
<OffthreadVideo src={staticFile("broll.mp4")} /> {/* overlay layer */}
</AbsoluteFill>
</Sequence>
</AbsoluteFill>
Convert every timecode to frames with Math.round(seconds * fps), once, at the edge. Why: a b-roll cue at 4.0s is frame 120 at 30fps but frame 240 at 60fps — keep seconds in your data and multiply by fps from useVideoConfig() so changing fps never desyncs overlays. Use <OffthreadVideo> (not the DOM <video> or <Video>) for frame-accurate decoding during render.
Render
# region test first: 1-2 seconds, validates the pipeline cheaply
npx remotion render MyVideo out/test.mp4 --frames=0-45
# then the full render
npx remotion render MyVideo out/video.mp4
Omit the composition id to get an interactive picker. Configure via @remotion/cli/config in remotion.config.ts, or pass flags on the CLI (flags win):
// remotion.config.ts
import { Config } from "@remotion/cli/config";
Config.setConcurrency(8);
Config.setCodec("h264");
| Codec | Use | Flag |
|---|---|---|
h264 | Default for web/YouTube; broad compatibility | (default) |
h265 | Smaller files, same quality; less universal playback | --codec=h265 |
prores | Edit-grade master, large files, re-import to an NLE | --codec=prores --prores-profile=4444 --pixel-format=yuva444p10le --image-format=png |
vp8 / gif | Web-alpha or looping previews | --codec=vp8 / --codec=gif |
Always render a 1–2s region (--frames=0-45) before the full render. Why: a full render of a minutes-long composition costs real time and CPU; a region test surfaces a broken caption layer or missing asset in seconds. For headless/CI renders, deterministic-output rules, and the codec/quality matrix in full, see references/render-and-pipeline.md.
Anti-patterns
| Bad | Why it breaks | Good |
|---|---|---|
| Hardcoding durations in seconds inside JSX | Remotion thinks in frames; seconds desync the moment fps changes | Store seconds in data, Math.round(seconds * fps) at the edge |
Date.now() / unseeded Math.random() in a composition | Frames render in parallel and on re-render → non-deterministic pixels | Drive everything off useCurrentFrame(); use random(seed) |
Changing fps after writing durations | Every frame-count downstream silently shifts | Pin fps + dimensions on <Composition> up front, leave them |
| Re-downloading the Whisper model every run | The ~1.5 GB medium.en download repeats and stalls the pipeline | Download once, cache the model path, reuse it |
| Committing the 1.5 GB Whisper model to git | Bloats the repo; the model is a build asset | .gitignore the model dir; fetch it in setup/CI |
| Rendering the full video to test a change | Minutes of wasted render to find a broken layer | --frames=0-45 region test, then full render |
| Writing the caption copy or hook here | That is the script, not the encode | Route to ../video-shorts/SKILL.md; you only burn it in |
Plain <video> / <Video> for b-roll in a render | Not frame-accurate; tears or skips on render | <OffthreadVideo> for frame-exact decode |
| Running silence removal inside the composition | Trimming dead air is an upstream asset edit | auto-editor on the source file before Remotion |
Verify & references
bash scripts/verify.sh <project-dir>— checks the Remotion project is well-formed and renders. With Node + ffmpeg present it runsnpx remotion compositionsto confirm a composition id exists, does a short region render to a temp MP4, and usesffprobeto confirm a video stream with the expected dimensions/fps. With neither, it falls back to a static check: at least one composition.tsxexists and the render script references a real composition id and an output path. Read-only by default; exits 0 on an empty/clean target.references/captions-pipeline.md— full Whisper.cpp install + transcribe server,toCaptions, the token-highlight caption renderer component, and caption styling/safe-zone notes.references/render-and-pipeline.md—auto-editor+ ffmpeg silence commands, the full render flag matrix, headless/CI render, the codec table, andffprobeverification.