remotion-sfx

SFX (Executable Pipeline)

Position: A hands-on skill. The true source of rules is in remotion-assembly/references/audio.md (loudness chain, mastering, pauses, BGM rulings). In case of conflict, that document takes precedence. Hard-learned lessons (do not repeat): ffmpeg white noise synthesis → rejected (cheap, sandpaper-like); sampled soundtrack audio → rejected (thin after removing vocals). The correct approach = professional SFX library + user audition selection (Mistake #21).

0. First, Check the Finalized Library (Most Cases End Here)

public/sfx/ The ten active sound colors (audition-finalized on 2026-07-16 + source-switched from the approved pool on 2026-07-21; numbering/source-switching records are in audio.md rule 8; parentheses indicate §4 event positions):

glass (evidence card reveal) (A) / pop chip (light tap) (B, source switched to fav20 crisp bright tick, replacing the old one criticized as 'crackly') / tick2 (B variant, fav33—used for consecutive B events in the same segment to avoid repetition) / pop2 (nameplate soft plop placement) (G, source switched to fav14 dull plop with volume) / stamp (heavy seal hammer) (D, ≤1 per entire video) / rise (digital scroll continuous roll) (C, fav29 remade, BASE_VOL independently lowered to 0.1) / whoosh (clean soft sweep) (F) / shutter (photo snap freeze) / bright (highlight underline) (fav02 bright ding—the sole bright family sound in the reference video's full spectrum) / lowthud (physical impact, falling) (fav13 low-frequency long rumble, specifically for invasion beats; prone to being too heavy, awaiting segment final review calibration). Old pop/pop2 are stored in spare/prev-*. trans and ding were abandoned after user audition on 2026-07-20 (E-position defaults to silence; official source replacement follows §1-2). Spares are in public/sfx/spare/. The approved pool public/sfx/jy-fav/ (user auditioned 42 clips from their Jianying favorites on 2026-07-21, approved 19 usable ones): fav number + hash + level grouping are documented in SOURCES.md within the directory; all have passed the user's ears. For source replacement/supplement for B/G/A/E positions, prioritize from this pool; role assignment is completed during segment final review. Still, adhere to the A-G table restraint for active sound colors in the same video (user calibration: only about a dozen common sound colors are needed; SFX must not overshadow the content). When switching to a new video, reuse this library directly; only follow steps 1-3 if a new character or new texture is needed.

1. Source Selection (When New SFX is Needed)

Priority: a. Jianying local cache library (user-designated primary: ~/Library/Containers/com.lemon.lvpro/Data/Movies/JianyingPro/User Data/Cache/music/ etc. Contains downloaded SFX library caches; use find ... -iname "*.mp3" + ffprobe duration ≤5s to filter SFX + md5 deduplication; files are hash-named without titles, claimed via audition board. To expand the pool: the user taps a desired sound in Jianying, it lands in the cache, re-run the library scanning script—this is the standard action for user library expansion) → b. @remotion/sfx (d.ts is the direct link list; usable items: whip/whoosh/switch/mouse-click/page-turn/ding/shutter-modern) → c. Mixkit free commercial (category page grep assets.mixkit.co/active_storage/sfx/<id>/<id>-preview.mp3) → d. Pixabay / kenney.nl CC0 → e. Commercial packs (Artlist/Epidemic). Copyright note: Jianying asset terms nominally restrict use to within Jianying; common practice in the domestic creator ecosystem is user discretion; for external commercial distribution, prioritize b-e.

2. Audition Board (Mandatory Before Any New SFX Appears in a Video, Cannot Be Skipped)

AI has zero judgment on sound texture; it can only filter spectral flaws. Workflow: Normalize candidates to -3dB → Render each with a black background and numbered name card (drawtext) into a small segment → Concat into one audition video for the user → Only allow on-screen after the user reports the number. Critical joining rules (pitfall encountered 2026-07-16): ① Each segment's duration must be locked to be equal—audio apad=whole_dur=2.1,atrim=0:2.1, video -t 2.1, disable -shortest (aac priming causes minor duration differences); ② Disable -c copy for concatenation; must re-encode the entire stream—otherwise audio-video drift accumulates, causing later number labels and sounds to mismatch (user tested 40+ numbers all scrambled), making all user-reported numbers invalid and requiring rework. ③ Validate each segment before delivery (accident 0715-02: silent segments at positions 4/8 were delivered to the user as a detector): After assembling the board, run ffmpeg -ss <segment start> -t <segment length> -af volumedetect on each segment to check max_volume; any segment below -60dB is silent—first run volumedetect on that source file (to check for corrupted files in the library, e.g., rise.wav -91dB), then check the join command, fix it, and re-join; do not deliver a silent board to the user.

All sound color assignment relies on the audition board, not AI guessing (settled in 0715-02 final review): Which sound color matches a visual event is dictated by the user reviewing the audition board—AI has zero judgment on sound color. In 0715-02, AI guessed wrong sound colors twice; the correct one was only found via the audition board (stamp). Whenever a new video or new sound color doesn't match an old library sound, output an audition board first and let the user claim the roles. Prohibit AI from directly assigning sound colors to a video.

3. Processing & Library Integration

• Peak normalize to -3dBFS (use volumedetect to measure max → volume=(-3-max)dB), 48kHz mono.
• Trim long samples: for impact types, take the first 1.0-1.6s + afade out.
• Consecutive roll synthesis (digital scrolling/batch lighting): select tick, trim 40ms + -stream_loop 29 + volume ramp volume='0.35+0.65*t/1.2':eval=frame—the reference video's "scrolling sound" is essentially tick hits, not a pitch riser.
• New characters: add to sfx-<video-name>.ts as a union type of SfxName + add to SfxTrack BASE_VOL synchronously.

4. Cue Sheet Mounting: A-G Sound Assignment Syntax (2026-07-16 spectral decoding of 11 anchors finalized + 2026-07-21 full statistics upgrade of two reference videos [221 actual events]; true source = this section's A-G table + audio.md rule 8; finalization evidence: assembly/0715-01/分段工坊/音效设计谱.md + docs/基准片配声谱v2.md)

Visual Event → Sound Color Mapping Table (look up per beat, prohibit improvisation):

Event	Sound Color	Library File
A Evidence card/screenshot reveal	Glass chime (short, pitched, UI-like; reference video measured reveal sound is duller bright -13.5—do not pick the brightest tier; bright ding candidates fav02/25/19 must have brightness calibrated before on-screen use)	glass.wav (Jianying J08, audition board #106)
B Chip/small object entry, dot-by-dot lighting	Short crisp tick; for batches, fire at 0.15-0.3s intervals	pop.wav (source switched to fav20 on 2026-07-21); for consecutive use in the same segment, switch to tick2.wav to avoid repetition
C Digital scrolling	Tick roll covering the entire scroll duration; no heavy impact on final value (reference video syntax)	rise.wav (1.5s version)
D Stamp/judgment slam	Short dull hit, solid but not boomy	stamp.wav (if too heavy, check spare/1143)
E Chapter transition/takeover entry	Long airflow riser, only for chapter-level	trans.wav abandoned 2026-07-20; reference video measured all 20 chapter transitions/takeovers used short switch sounds (decay ≤0.1s, no long airflow riser)—supplement source with <0.2s short sounds, prohibit risers; maintain silence by default (0715-02 finalized practice, user ruling for this style); if necessary, use a light whoosh (vol ≤0.5, use a lower step than F's practical value to ensure 'exit is lighter than entry'); official source supplement follows §1-2 audition
F Cleanup/exit	Decaying soft sweep, lighter than entry by one notch (vol ≤0.5; 'lighter by one notch' is a style restraint ruling—reference video showed exit not being lighter than entry; the difference is a slightly longer decay tail; choose items with 'decay tail' over lower volume)	whoosh.wav
G Nameplate/badge/chip placement	Soft pop	pop2.wav (source switched to fav14 dull plop on 2026-07-21) or spare/interface-1109

Extended event positions (2026-07-18 generalization stress test supplement, grouped by family without adding new sound sources): hero text/conclusion lockup → D light (vol ≤0.5, does not consume Stamp heavy quota); timeline card/chart container entry → A; timeline node lighting, data bar growth, line draw-on → B (batch follows consecutive fire rule); judgment small note/ZeroTag placement → B; highlight underline / emphasis → bright.wav (dedicated bright ding, sole bright family in reference video full spectrum +3.0); physical impact/fall/squeeze → lowthud.wav (light vol reduce as needed, prone to being heavy); punch-in zoom → medium strength short sound (D light or lowthud light); micro-chapter downgrade—for paginated segments <8s, E transition only at the very beginning and end of the whole group; intervals use B instead (prevent transition sound fatigue; E position currently temporarily uses whoosh, see §0).

Approved pool mapping (2026-07-21 feature measurement by family; primary selections were formalized and integrated into the library on the same day, see §0; final hearing calibration occurs during segment final review): Formalized—pop←fav20, pop2←fav14, tick2←fav33, bright←fav02, lowthud←fav13. Remaining in pool as backups: B backup fav27/24, G backup fav11/06, A backup fav25/19 (bright ding, if bright is too bright, downgrade to this), F backup fav16 (extremely light breath), D light backup fav05, physical backup fav08, E position candidate fav17/00 (still defaults to silence). During segment final review, swap any that are disliked; 1 minute to return to pool and swap source. Full characteristic table in jy-fav/SOURCES.md.

First, ask about the semantic family (set 2026-07-21, to prevent 'pick the one that sounds best'): Bright ding = emphasize/highlight, the viewer should look here (reference video shows the only bright family is highlight underline +3.0; evidence card reveal measured duller at -13.5, belongs to dull-short family, use glass mid-tier); Dry tick = small parts in place / lighting step by step (forward momentum); Soft plop = identity/conclusion settled (weight); Low frequency = physical actions impact/fall/squeeze (causality); Breath = departure (cleanup feeling). Determine the semantic family of the visual event, then pick sound from that family—if the family doesn't match, it's better not to play a sound.

Three meta-rules (more important than sound color itself, root cause of the v6 derailment): ① Sound color set consistency—the entire video must use one family of UI-type sounds; prohibit mixing photographic, office-type, or cinematic-style sounds. ② Volume restraint—no sound should overpower the voiceover; rely on density, not loudness (reference video measured 4-5s/shot across the whole video; '2-3s/shot' is for dense segments, not the average, see v2 audio spectrum). ③ One event, one sound—do not layer two sounds; if two events occur on the same beat, stagger them by ≥0.1s.

Hook / Opening Sound General Rule (the correct answer to question ⑩ 'Does the opening need an SFX?'): The opening is not a special case of 'whether to add sound'; it follows the same general rule of 'main element gets sound + A-G look-up table'—the hook main visual should play a sound when it lands: typewriter text = per-character tick (B), hook heavy digital = C scroll, evidence card reveal = A, chapter start/takeover entry = E. The only thing rejected is 'decorative introductory sound without a visual anchor' (whoosh/sandpaper opening, filler sound for atmosphere)—these don't correspond to any visual event and are pure noise outside meta-rules ① and ②. Criterion: every sound in the opening must be attached to a main element that is landing. If it can't be attached, don't add it; don't force a sound just to have sound at the start.

Already rejected: opening whoosh (decorative intro, see above), heavy slam for number finalization, heavy hammer throughout the old video (≤2 per video, as of 2026-07-21 tightened to crackle-type sounds ≤1 per video, hard line see §4.4; reference video measured stamp 1/2 spots, sound profile = dullest bright -18.4 + heaviest rise to 30dB, 'dull not boomy' confirmed), BGM (reference video measured has: continuous low-frequency bed, constant through head/middle/tail, estimated relative to voiceover ≤-15dB—the absence of BGM is a user-stated style difference, not reference video syntax; when revisiting, start with -15dB constant pad), trans chapter riser and ding ending (abandoned after 2026-07-20 audition, see §0).

4.4 Full Video SFX Validation Gate (settled at 0715-02 final review: after mounting cues, must run; if lines exceeded, re-edit and re-render)

• Density: Hard lower limit ≥2.5s/shot (incident: 1.95s/shot was rejected at final review as 'too noisy'); target band 3.5-5s/shot (reference video measurements: v5=5.0/v6=4.0; 0715-02 cut to 3.05s still felt dense, not the endpoint). Silence is also syntax: 20-30s continuous no-cue is normal (measured max silence 28.7s); dry voiceover segments must not add sound to reach density. Density should undulate using a 30s window comparison (dense window 9-14 fires vs sparse window 2-3 fires), not uniform distribution. Coverage criterion takes precedence over average density (2026-07-21 0721-01 final review): Finalized 42 fires at 3.02s/shot (below the old target band lower bound), user still pointed out 'many MG are missing SFX'—density average meeting the target doesn't mean coverage is adequate; audit uses 'every MG main element landing has a sound, or an explicit reason for silence (pure voiceover/takeover entry/semantic family mismatch)' as the standard. The target band 3.5-5 is downgraded to a reference line for preventing noise; when coverage requires it, density below 3.5 does not trigger rework (at 0721-01, only 7 active sound colors were the main bottleneck for coverage; library expansion is already done, see tension section below).
• Single sound color proportion ≤35% (incident: pop 114/220=52%, listening sensation 'camera shutter burst'—short tick type consecutive hits create a shutter feeling). Reference video dry tick family measured exactly 35% (78/221), this line aligns with measurement, hard line not to be relaxed.
• Batch lighting defaults to 'count as one fire for the whole group'; 0.15-0.3s consecutive firing only used for ≤4 items' semantic stagger, overriding §4 B position's old 'batch consecutive fire' default.
• BASE_VOL baseline ≈0.25 (0.2-0.3), heavy hammer upper limit 0.55—user ruling on 2026-07-21: 'Sound effects' basic volume should be around -15dB': library peak -3dBFS × 0.25 ≈ effective peak -15dBFS (overriding the 2026-07-20 range of 0.3-0.55; further back, 0.8-1.0 was the 'too loud' old version, do not revert).
• Use stamp sound with caution (settled at 0715-02 final review): The user perceived dense stamp sounds as 'camera shutter clicking'—not every judgment/name/green stamp needs a stamp sound; rely on visual silence instead. Suggested ≤ single digits for stamp (including heavy hammer) across the whole video. This stacks with 'heavy hammer ≤2 per video': heavy hammer is a strong subset of stamp; first limit heavy hammer to ≤2, then limit total stamp sound count to ≤ single digits.
• Crackle-type sounds ≤1 across the whole video (user ruling 2026-07-21 0721-01, tightening the previous stamp ≤ single digits): The user is extremely sensitive to the crackle (stamp/shutter) listening sensation; dense judgment crackle = camera burst, harsh; keep only 1 heavy hammer stamp for the climax, change all other judgments/conclusion placements to soft sounds (pop2 soft plop / rise growth), don't stamp every judgment.
• Iconic sound (rise growth sound) also restrained, more importantly prevent local clustering (lesson from 0721-01): rise ≤4-5 across whole video and prohibit ≥2 fires within a 30s window (0721-01 first version had rise 10 times, 111-122s clustered 4, user judged 'same sound effect repeating'—single sound color 22% didn't exceed 35% line but still considered too many; local clustering is more lethal than overall proportion); among adjacent growth animations, only pick the most iconic 1 beat for rise, leave the rest silent or use pop.
• Continuous tick-type sounds (rise) BASE_VOL independently lowered further: rise is ~1.2s continuous tick, cumulative loudness much higher than single pop/pop2; even with same peak -3dB, it sounds 'bigger'; user ruling 0721-01: rise independently lowered to 0.1 (one notch below the regular 0.25)—continuous/sustained sounds should all be one notch lower than single fire.
• Tension between MG sound coverage and sound library size (0721-01 unresolved, awaiting library expansion): User wants 'every MG to have a sound' but library only has 7 sound colors; forcefully increasing coverage = repeatedly using the same few sounds → both noisy and repetitive (this was the root cause of crackle/rise being considered too many). Coverage and 'not noisy, not repetitive' are contradictory with a small library; the cure = expand the library. Already implemented on 2026-07-21: Approved pool formalized 3 new characters (tick2/bright/lowthud) + B/G source switch, ten active sound colors (§0); if still feels monotonous, pull from the pool (19 items already passed user ears, mapping table in SOURCES.md) or user can collect more to expand the pool.
• Audit action: grep -c to count total cues and per-sound distribution; if any line is exceeded, first cut batch positions, then reduce proportion; stamp sound counted separately; if exceeds single digits, rework.

4.5 Automated Closed Loop (User Ruling: No More Per-Version Volume Adjustments)

The user only intervenes at two points; everything else is automatic:

1. One-time library definition (already in progress): Scan library / select sources → full audition board → user reports numbers → assign roles per A-G syntax → finalize public/sfx/ permanent library. After that, no more sound selection for new videos.
2. Segment final review: Listen to the completed segment as a whole; report time points for replacement (return to library/spare, swap source, new version in 1 minute). Completed segments for final review must always be delivered with SFX—cue automatic mapping does not rely on visual first finalization; prohibit delivering a version without SFX and then re-rendering (lesson from 0715-02: three segments rendered twice, wasting one rendering cycle). Intermediate steps zero manual: During each assembly segment, automatically assign cues based on 'component type → event type → A-G lookup'—Chip/Grid point=B, QuoteDoc/ShotCard reveal=A, Stamp=D, BigNumber scroll=C, Takeover/Chapter start=E, Exit cleanup=F, Nameplate/Badge=G; cue at directly takes element's enterAt converted to seconds. Cue draft is already scripted: node scripts/beat-to-cues.mjs <video-name> mechanically generates a sfx-<video-name>.ts draft from the assembly plan table (with built-in §4.4 pre-check report; highlights exceeding lines in red but does not delete them); the sound editor only reviews, does not edit. The mapping table inside this script is a copy of this section; rule changes take this SKILL as the source of truth. Prohibit searching for new sound effects for a single segment; only go through §1-2 to supplement the library if a role is missing.

5. Quantitative Validation (Must Run After Each Segment Render)

• For each cue point, run ffmpeg -ss <t> -t 0.5 -af volumedetect—the full-band peak should be significantly higher than the adjacent pure voiceover window. For bright texture items, additionally test highpass=f=8000: ≥ voiceover sibilance baseline +6dB (this threshold is pending recalibration from 2026-07-20 onward—after BASE_VOL downgrade, rely on per-cue audibility + user final review; do not raise volume to meet dB targets, see audio.md rule 1).
• Audible event count: highpass=f=12000,silencedetect=noise=-25dB event count ≈ cue count.
• False peak identification: Voiceover plosives and BGM kick drum hits can also create peaks (dual video decoding eliminated 39% of 360 candidates); finger snaps and hand claps recorded live can reach rise 40dB—when listening to cue point validation, first exclude these live sounds; do not use them as benchmarks for SFX.
• Final criterion = the user's ears: Deliver the completed segment for the user to listen to; report time points for replacement. Return to audition board/spare parts to swap sources; a new version in 1 minute.