Video Editing
AI-assisted editing of real footage. Not generating from prompts. Quickly edit existing videos.
When to Activate
- User wants to edit, cut, or structure video footage
- Convert long recordings into short-form video content
- Build vlogs, tutorials, or demo videos from raw material
- Add overlays, captions, music, or voiceovers to existing videos
- Reframe videos for different platforms (YouTube, TikTok, Instagram)
- User says "edit video," "cut this footage," "make a vlog," or "video workflow"
Core Thesis
AI video editing is useful when you stop asking it to create entire videos and start using it to compress, structure, and enhance real footage. Value is not in generation. Value is in compression.
Pipeline
Screen Studio / Raw Footage
→ Claude / Codex
→ FFmpeg
→ Remotion
→ ElevenLabs / fal.ai
→ Descript or CapCut
Each layer has a specific job. Don't skip layers. Don't try to make one tool do everything.
Layer 1: Capture (Screen Studio / Raw Footage)
Collect source material:
- Screen Studio: Polished screen recordings for app demos, coding sessions, browser workflows
- Raw camera footage: Vlog material, interviews, event recordings
- Desktop capture via VideoDB: Session recordings with live context (see
videodbskill)
Output: Raw files ready to be organized.
Layer 2: Organize (Claude / Codex)
Use Claude Code or Codex to:
- Transcribe and tag: Generate transcriptions, identify topics and themes
- Plan structure: Decide what to keep, what to cut, what order makes sense
- Identify dead zones: Find pauses, tangents, repeated takes
- Generate edit decision list: Timestamps for cuts, segments to keep
- Scaffold FFmpeg and Remotion code: Generate commands and compositions
Example prompt:
"Here is the transcript of a 4-hour recording. Find the 8 strongest clips
for a 24-minute vlog. Give me FFmpeg cut commands for each clip."
This layer is about structure, not final creative taste.
Layer 3: Deterministic Cutting (FFmpeg)
FFmpeg handles the boring but critical work: splitting, trimming, concatenating, and preprocessing.
Extracting segments by timestamp
ffmpeg -i raw.mp4 -ss 00:12:30 -to 00:15:45 -c copy segment_01.mp4
Batch cutting from an edit decision list
#!/bin/bash
# cuts.txt: start,end,label
while IFS=, read -r start end label; do
ffmpeg -i raw.mp4 -ss "$start" -to "$end" -c copy "segments/${label}.mp4"
done < cuts.txt
Concatenating segments
# Create file list
for f in segments/*.mp4; do echo "file '$f'"; done > concat.txt
ffmpeg -f concat -safe 0 -i concat.txt -c copy assembled.mp4
Creating proxy files for faster editing
ffmpeg -i raw.mp4 -vf "scale=960:-2" -c:v libx264 -preset ultrafast -crf 28 proxy.mp4
Extracting audio for transcription
ffmpeg -i raw.mp4 -vn -acodec pcm_s16le -ar 16000 audio.wav
Normalizing audio levels
ffmpeg -i segment.mp4 -af loudnorm=I=-16:TP=-1.5:LRA=11 -c:v copy normalized.mp4
Layer 4: Programmatic Compositing (Remotion)
Remotion turns editing problems into composable code. Use it for things traditional editors struggle with:
When to use Remotion
- Overlays: Text, images, branding, lower thirds
- Data visualization: Charts, stats, animated numbers
- Motion graphics: Transitions, explainer animations
- Composable scenes: Reusable templates across videos
- Product demos: Annotated screenshots, UI highlights
Basic Remotion composition
import { AbsoluteFill, Sequence, Video, useCurrentFrame } from "remotion";
export const VlogComposition: React.FC = () => {
const frame = useCurrentFrame();
return (
<AbsoluteFill>
{/* Main footage */}
<Sequence from={0} durationInFrames={300}>
<Video src="/segments/intro.mp4" />
</Sequence>
{/* Title overlay */}
<Sequence from={30} durationInFrames={90}>
<AbsoluteFill style={{
justifyContent: "center",
alignItems: "center",
}}>
<h1 style={{
fontSize: 72,
color: "white",
textShadow: "2px 2px 8px rgba(0,0,0,0.8)",
}}>
AI Editing Stack
</h1>
</AbsoluteFill>
</Sequence>
{/* Next segment */}
<Sequence from={300} durationInFrames={450}>
<Video src="/segments/demo.mp4" />
</Sequence>
</AbsoluteFill>
);
};
Rendering output
npx remotion render src/index.ts VlogComposition output.mp4
See Remotion docs for detailed patterns and API reference.
Layer 5: Generate Assets (ElevenLabs / fal.ai)
Only generate what you need. Don't generate the entire video.
Voiceover with ElevenLabs
import os
import requests
resp = requests.post(
f"https://api.elevenlabs.io/v1/text-to-speech/{voice_id}",
headers={
"xi-api-key": os.environ["ELEVENLABS_API_KEY"],
"Content-Type": "application/json"
},
json={
"text": "Your voiceover text",
"model_id": "eleven_turbo_v2_5",
"voice_settings": {"stability": 0.5, "similarity_boost": 0.75}
}
)
with open("voiceover.mp3", "wb") as f:
f.write(resp.content)
Music and sound effects with fal.ai
Use the fal-ai-media skill for:
- Background music generation
- Sound effects (ThinkSound model for video-to-audio)
- Transition sound effects
Generative vision with fal.ai
For B-roll, thumbnails, or footage that doesn't exist:
generate(app_id: "fal-ai/nano-banana-pro", input_data: {
"prompt": "Tech vlog professional thumbnail, dark background, code on screen",
"image_size": "landscape_16_9"
})
VideoDB audio generation
If VideoDB is configured:
voiceover = coll.generate_voice(text="Voiceover here", voice="alloy")
music = coll.generate_music(prompt="Lo-fi background music for coding vlog", duration=120)
sfx = coll.generate_sound_effect(prompt="Subtle whoosh transition sound")
Layer 6: Final Polish (Descript / CapCut)
The last layer is human work. Use traditional editors for:
- Pacing: Adjust cuts that feel too fast or too slow
- Captions: Auto-generate, then manually proofread
- Color grading: Basic correction and mood
- Final audio mix: Balance voice, music, and sound effect levels
- Export: Platform-specific formats and quality settings
Taste lives here. AI removes repetitive work. You make the final decisions.
Social Media Reframing
Different platforms need different aspect ratios:
| Platform | Aspect Ratio | Resolution |
|---|---|---|
| YouTube | 16:9 | 1920x1080 |
| TikTok / Reels | 9:16 | 1080x1920 |
| Instagram Feed | 1:1 | 1080x1080 |
| X / Twitter | 16:9 or 1:1 | 1280x720 or 720x720 |
Reframing with FFmpeg
# 16:9 to 9:16 (center crop)
ffmpeg -i input.mp4 -vf "crop=ih*9/16:ih,scale=1080:1920" vertical.mp4
# 16:9 to 1:1 (center crop)
ffmpeg -i input.mp4 -vf "crop=ih:ih,scale=1080:1080" square.mp4
Reframing with VideoDB
from videodb import ReframeMode
# Smart reframing (AI-guided subject tracking)
reframed = video.reframe(start=0, end=60, target="vertical", mode=ReframeMode.smart)
Scene Detection and Auto-Cutting
FFmpeg scene detection
# Detect scene changes (threshold 0.3 = medium sensitivity)
ffmpeg -i input.mp4 -vf "select='gt(scene,0.3)',showinfo" -vsync vfr -f null - 2>&1 | grep showinfo
Silence detection for auto-cutting
# Find silent segments (useful for cutting dead air)
ffmpeg -i input.mp4 -af silencedetect=noise=-30dB:d=2 -f null - 2>&1 | grep silence
Highlight extraction
Use Claude to analyze transcript + scene timestamps:
"Based on this timestamped transcript and these scene change points,
identify 5 best 30-second clips for social media."
What Each Tool Does Best
| Tool | Strengths | Weaknesses |
|---|---|---|
| Claude / Codex | Organize, plan, code generation | Not a creative taste layer |
| FFmpeg | Deterministic cuts, batch processing, format conversion | No visual editing UI |
| Remotion | Programmatic overlays, composable scenes, reusable templates | Learning curve for non-developers |
| Screen Studio | Instant polished screen recordings | Screen capture only |
| ElevenLabs | Voice, voiceover, music, sound effects | Not a workflow hub |
| Descript / CapCut | Final pacing, captions, polish | Manual, not automatable |
Key Principles
- Edit, don't generate. This workflow is for cutting real footage, not creating from prompts.
- Structure before style. Get the story right in Layer 2 before touching any visuals.
- FFmpeg is the backbone. Boring but critical. Long footage becomes manageable here.
- Remotion for repeatability. If you'll do the same thing multiple times, make it a Remotion component.
- Selective generation. Only use AI generation for assets that don't exist, not for everything.
- Taste is the last layer. AI removes repetitive work. You make the final creative decisions.
Related Skills
fal-ai-media— AI image, video, and audio generationvideodb— Server-side video processing, indexing, and streamingcontent-engine— Platform-native content distribution