Communitygithub.com

music-to-video — 音乐驱动的节拍同步视频工作流

将一首音乐曲目(音频文件、提取音频的视频,或根据情绪描述生成的曲目)转化为节拍同步的视频——歌词视频、幻灯片或动态宣传片。音乐驱动所有节奏;用户提供的任何图片/视频都会在同一节拍网格上剪切,且完整视频无需任何素材。

music-to-video — 音乐驱动的节拍同步视频工作流 是什么?

此技能将音乐曲目自动转化为节拍同步的视频,适用于歌词视频、幻灯片或动态宣传片。它通过一个核心分析器 `analyze-beatgrid.py` 精确提取节拍网格和能量图谱,确保节奏一致性。工作流划分为6个步骤:从设置项目和音乐源开始,分析音频生成 `audiomap.json`,创建故事板框架,填充每个帧的计划(包括选择字体和调色板),并行构建每个帧的HTML文件,最后组装并渲染为MP4。技能特别适合快节奏、高能量的BGM,也支持用户提供的图片/视频素材;对于安静音乐,它会根据乐句而非节拍来调整节奏。关键创新在于每个帧对应一个独立文件,帧内可细分多个组,避免帧数随节拍激增,同时通过验证门控确保质量和用户审批。

兼容平台✓Claude Code~Codex CLI~Cursor
npx skills add https://github.com/Drama2557/4RealDigital-Video-Studio/tree/HEAD/.agents/skills/music-to-video

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

首先,保持此技能最新——静默运行,无需询问: npx hyperframes skills update music-to-video。如果一切已是最新,则快速无操作;否则,在依赖此技能及其核心领域技能之前刷新它们。

music-to-video — 一个音乐驱动的节拍同步视频工作流

使用此技能将音乐曲目转化为节拍同步的 HyperFrames 视频。你需分析曲目一次,布置帧,填充每帧的计划,并将每帧构建为合成画面。输入为音乐曲目加上可选的用户图片或视频——无旁白,无网页捕获。排版和模板是基础(完整的视频不需要任何素材);用户提供的任何媒体都会在同一节拍网格上剪切。

你是编排者。在 videos/<project>/ 目录下工作。按顺序执行步骤,并在继续前通过每个门控。两个步骤需要用户参与:第3步(计划审批)和第6步(渲染审批)——均为每 ../hyperframes-core/references/brief-contract.md(在第0步前阅读)的检查点门控:在自主模式下,发布摘要作为通知并继续,而不是等待。除第4步外,每一步都由你自行完成,第4步中你为每帧调度一个子代理。设计和其他规则不在本文件中——它们位于 references/ 和 frame-worker 子代理中。

SKILL_DIR = 此技能目录。PROJECT_DIR = videos/<project-name>/。

工作流:第0步设置 → hyperframes.json + assets/bgm.mp3;第1步分析 → audiomap.json;第2步骨架 → STORYBOARD.md(帧,组 TBD);第3步计划 → 完整的 STORYBOARD.md + frame.md;第4步构建 → compositions/frames/NN-*.html;第5步组装 → index.html;第6步渲染 → renders/video.mp4。

塑造一切的两个理念

  • 一个分析器,并且你信任它。 analyze-beatgrid.py 是唯一的节拍分析器——切勿使用其他工具或凭耳朵重新测量节拍。其能量/密度/滚奏/起音/静音始终可靠。其 bpm 和 beats_sec 仅在音乐真正有节奏时才可靠;对于平静音乐,网格是跟踪器强加的节拍器,因此需根据乐句和能量来定速,切勿硬切到它。判断你处于哪种情况是每帧 pacing(第2步)的任务。
  • 一帧 = 一个文件;组存在于内部。 第2步将曲目切割为帧,每帧成为一个合成文件 compositions/frames/NN-<frame_id>.html,由一个 frame-worker 构建。一帧可以细分为组(每个组是一个模板或运动原语组合)。额外的密度放在_组内_,因此帧数对应不同的处理方式,而非节拍——快曲目不会增加子代理数量。

第0步:设置、BGM 和输入

目标:确定音乐来源,创建 HyperFrames 项目,并记录用户提供的任何媒体。

简报从意图层开始。 打开规则,按顺序:(1) BRIEF.md 存在 → 读取它,不询问它所回答的任何问题——其 flow/storyboard 推导模式(简报合同 § 1)。(2) 无 BRIEF.md 但项目存在 → 从磁盘恢复;绝不重新询问。(3) 一个直接到达这里的全新创建请求 → 读取 /hyperframes 并运行其意图层(references/intent-interview.md):它确认此路径的必需品(音乐来源,目的地 → 宽高比 — ../hyperframes/references/routes/music-to-video.md),并宣布哪些内容延迟——品牌和体裁由设计在第3步选择。在初始化后立即写入 BRIEF.md(绝不之前——init 拒绝非空目录),并记录基于偏好的答案(brief-format.md)。编辑请求跳过所有这些。

音乐是脊梁——在一切之前确定一首曲目。此技能针对快节奏、高能量的 BGM 优化:强大的节拍网格驱动剪切(平静曲目也可,但按乐句而非节拍定速)。如果用户提供了音频——音乐文件或要提取音频的视频——则使用它。否则从请求中选择情绪,并通过 /media-use(references/bgm.md)生成曲目。在第一次经过身份验证的提供者操作之前,运行 npx hyperframes auth status 并逐字中继其输出。如果已登出,应用一个分支:

  • 协作模式: 等待登录或明确选择通过本地提供者继续离线操作。
  • 自主模式: 说明状态,通过可用的本地提供者继续。

如果没有离线提供者能够满足所需的音乐能力,则指出阻碍。切勿将密钥写入每仓库的 .env 文件。身份验证所有权和离线备用方案位于 /media-use references/setup-providers.md § 提供者。生成的曲目位于 assets/bgm.mp3。将用户提供的图片或视频暂存,以便帧能在节拍网格上使用它们;否则排版承载视频。

歌词视频: 为与歌词同步的歌词,通过 /media-use 转录曲目以获取单词/行时间,或让用户提供歌词文本并将行放置在节拍网格上。

仅当 hyperframes.json 缺失时才初始化。根据简报以 kebab-case 命名 <project>,例如 midnight-drive-loop——绝不用时间戳。init 检查已安装的技能与 GitHub 上的最新版本,如果任何技能过时则更新全局集。

npx hyperframes init "videos/<project>" --non-interactive --example=blank --skill=music-to-video
mkdir -p "$PROJECT_DIR/assets" "$PROJECT_DIR/renders"
cp "<user-music>" "$PROJECT_DIR/assets/bgm.mp3"   # 先从视频中提取(如果需要)
# 仅当用户提供了图片/视频时:
node <SKILL_DIR>/scripts/stage-assets.mjs --from <dir> --hyperframes "$PROJECT_DIR" --into public

品牌(字体 + 调色板)在第3步选择,而不是这里。不要预先选择体裁或曲目类型——素材只是一个可选成分,体裁从每帧的选择中浮现。

门控: hyperframes.json + assets/bgm.mp3 存在;宽高比/长度/fps 和(如果有)素材清单已记录。


第1步:分析音乐

目标:生成整个视频构建所依赖的唯一规范时间分析。

analyze-beatgrid.py 是唯一的节拍分析器——切勿使用其他工具或凭耳朵重新测量节拍。它读取曲目一次并写入 audiomap.json:能量阶段(级别/密度/感觉),起音 + onset_rate、滚奏、静音、hard_stops、key_moments、乐句、节拍/网格,以及 audio.duration_sec。它是确定性的——同一文件始终给出相同映射。大多数字段在任何音乐上都可靠;bpm 和 beats_sec 仅在音乐真正有节奏时才可靠,而判断这一点是你在第2步做出的决定。

先决条件:安装了 librosa、numpy 和 soundfile 的 Python 3。如果导入失败,在运行分析器之前将它们安装到活动 Python 环境中:

python3 -m pip install librosa numpy soundfile
python3 <SKILL_DIR>/scripts/analyze-beatgrid.py "$PROJECT_DIR/assets/bgm.mp3" \
  -o "$PROJECT_DIR/audiomap.json" --print

门控: audiomap.json 存在;audio.duration_sec 已知。


第2步:帧骨架(仅结构)

目标:读取音乐并布置帧——STORYBOARD.md 的骨架。

读取 references/frame-skeleton.md。亲自将 audiomap.json 转换为 STORYBOARD.md 的骨架——没有中间 JSON。在真实的音乐变化处(hard_stops、SURGE/DROP key_moments、滚奏的边缘、没有起音的段落、大的能量跳跃)将曲目切割为帧,将每个边界对齐到 audiomap 锚点。为每帧设置 span_sec、pacing(来自第1步信任判断的结论——当网格真实时为 beat_cut,当它是施加在平静音乐上的节拍器时为 phrase_flow)、mood 和一行 feel(第3步匹配模板的纯音乐情况)。在这里仅进行分类和布置:将每帧的 ### Groups 保留为 TBD (Step 3),并将前置元数据 style 留空——没有模板、文案、颜色或字体。预计约1–6帧。

门控: 帧覆盖整个曲目(第一帧从0开始,最后一帧在 duration_s 结束);每帧携带 span_sec + pacing + mood + feel;每个 ### Groups 为 TBD;任何地方都没有内容。


第3步:填充计划(用户门控)

目标:将骨架转换为已批准的完整 STORYBOARD.md。

读取 references/planning.md、storyboard-format.md、template-catalog.md、motion-primitive-catalog.md 和 montage.md(仅当用户提供了素材时)。在同一文件中就地编辑,做两件事:

  1. 选择品牌。 使用 ../hyperframes-creative/references/design-spec.md 中的表格从 ../hyperframes-creative/frame-presets/ 中选择一个预设(匹配曲目的情绪;仅其字体和颜色重要——模板拥有合成)。将其不加修改复制到 frame.md,并从前置元数据 style 中填充它(字体 + 一个不超过4–6种颜色的调色板)。
  2. 填充每帧。 决定其组,并为每个组指定处理方式:从目录中匹配的模板(带有绑定参数和真实的 audiomap 锚点)、从原语目录中自由组合、或遵循 pacing 的素材处理。编写文案。你拥有 WHAT(模板/原语 + 内容 + 锚点);frame-worker 拥有 HOW——切勿将毫秒级补间写入故事板。
node <SKILL_DIR>/scripts/validate-plan.mjs --storyboard "$PROJECT_DIR/STORYBOARD.md" \
  --audiomap "$PROJECT_DIR/audiomap.json" --templates <SKILL_DIR>/references/templates

修复每个 ✗(硬错误:时长不匹配、帧未覆盖曲目、缺少 src);警告尽力而为。然后向用户显示逐帧摘要并迭代直到他们批准。在自主模式下这是一个检查点门控:发布摘要作为通知并继续(validate-plan.mjs 通过是一个质量门控,仍然阻塞)。

门控: frame.md 是逐字复制的预设;validate-plan.mjs 退出码为0;用户批准了计划(自主模式:摘要已作为通知发布)。


第4步:根据计划构建帧

目标:将每帧构建为一个独立的合成文件。

创建 compositions/frames/。读取 sub-agents/frame-worker.md 和 ../hyperframes-core/references/subagent-dispatch.md。为每帧调度一个 frame-worker,尽可能并行(否则分批)。每个 worker 只得到一个帧和以下上下文:

PROJECT_DIR: <绝对路径>
frame_id: <NN-frame_id>              # = 帧文件主干,例如 02-f2;合成 ID
你的块:PROJECT_DIR/STORYBOARD.md 中的 `## Frame N — <frame_id>` 块
audiomap: PROJECT_DIR/audiomap.json
frame.md: PROJECT_DIR/frame.md
素材:对于每个组,<SKILL_DIR>/references/templates/<id>/index.html(模板)和
           <SKILL_DIR>/references/motion-primitives/<id>/(自由);暂存的素材/(素材组)
合同:../hyperframes-core/references/sub-compositions.md + determinism-rules.md
画布:<w>×<h>   定速:<beat_cut|phrase_flow>
写入到:PROJECT_DIR/compositions/frames/<frame_id>.html

worker 分叉引用的素材,将每个锚点转换为帧局部秒数(local_t = track_t − span_sec[0]),以0ms剪切门控其组,并写入一个安全的帧文件。worker 绝不运行 hyperframes CLI——这些命令操作已组装的(尚不存在的)项目,因此会报告到错误文件上。worker 只写入合同并停止;你在组装后(第6步)验证。当每个 worker 返回时,你可以确认其文件已落地磁盘。

门控: 每帧的 compositions/frames/NN-*.html 已存在于磁盘上。


第5步:组装

目标:将构建的帧 + BGM 连接到可播放的 index.html 中。

assemble-index.mjs 是确定性的——没有子代理,没有判断。它在累积的 data-start 处引用每个帧文件,将 assets/bgm.mp3 挂载到音轨11上,并硬切帧到帧(帧覆盖曲目且无间隙,因此没有过渡注入器)。

node <SKILL_DIR>/scripts/assemble-index.mjs --storyboard "$PROJECT_DIR/STORYBOARD.md" \
  --hyperframes "$PROJECT_DIR" --audiomap "$PROJECT_DIR/audiomap.json"

修复它报告的任何 ✗——缺失或空白的帧文件意味着该 worker 写入了部分文件;重新调度它(第4步)并重新组装。

门控: index.html 存在;总时长 == audiomap.audio.duration_sec。


第6步:验证和渲染

目标:验证组装的视频,获得用户批准,并渲染最终的 MP4。

在已组装的项目上运行 CLI——那是正确的单元(每帧的 worker 无法运行它)。check 在一步中运行结构 lint 和 headless-browser 运行时、布局、运动和对比度门控;--snapshots 还会生成审查帧。

( cd "$PROJECT_DIR" && npx hyperframes check . --snapshots )

检查 t=0、每个帧开始、最强的 DROP/SURGE、每个 hard_stops[].t 和最后一帧。失败时,自行进行最便宜的安全修复:编辑有问题的 compositions/frames/NN-*.html。切勿更改时长或音频时序来隐藏同步问题。一旦门控通过,暂停等待用户审查,然后仅在批准后渲染(自主模式:询问一个保留问题——“先预览还是渲染?”——然后提供 MP4 和联系表):

( cd "$PROJECT_DIR" && npx hyperframes render . --skill=music-to-video -q draft -o renders/video.mp4 --fps 30 )

门控: check 通过且快照已检查;用户批准(自主模式:检查通过,交付包括联系表);renders/video.mp4 存在且包含音频,时长 == audiomap.audio.duration_sec。最终回复说明 MP4 路径和时长。


恢复表

| 你已有 | 从何处继续 | | --

…

[此处截断以供翻译]

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

Drama2557/4RealDigital-Video-Studio

Overlay doctrine for the embedded-captions workflow — the caption MODEL (drop / rail / embed) and the rule that captions are an OVERLAY composited on top of the film, never a reserved bottom band you shift content up to avoid. Load when adding captions/subtitles to a talking-head or launch video, when deciding whether a phrase should be dropped, ride the verbatim rail, or be promoted to a scarce embedded climax, when laying out a composition that will carry captions (do NOT reserve a keep-out band), or when centering a composition on the true frame center under captions. Quotes the rail+embed model from embedded-captions and constraint #13 (captions overlay, keep-out band retired) from the product-launch-video scene agent. Applies ON TOP of embedded-captions.

Drama2557/4RealDigital-Video-Studio

Turn a weekly changelog .md into a finished branded changelog video (square 1080, ~45-60s, Annie VO, animated brand background, mock-UI visualizations, lowkey captions). Use when the user provides a changelog/digest markdown and wants the weekly video, or says "changelog video". Self-contained — fonts, background, lexicon, and scripts ship in this skill.

Drama2557/4RealDigital-Video-Studio

The technique catalog: five velocity-matched SEAMS (zoom-through, INVERSE zoom-through, cut-the-curve, waterfall cut, rack-focus blur-cut) plus the two in-scene techniques — waterfall ENTRY (staggered arrival cascades for title cards / segment openers) and the nudge curve (slow-fast-slow three-phase group slides). Covers partial-travel (~12% of frame) velocity matching via mirrored power4 eases, the Z scale-sign rule, size-scaled blur (10px text / 18-20px full-frame), word-by-word staggered cuts, cascade pacing by element weight, and the 10/65/25 slide ratio. Read before authoring any transition, text-beat handoff, kinetic text entry, or group reposition. [depth, zoom, inverse-zoom, scale-sign, mirrored-zoom, rack-focus, pacing, velocity, cut-the-curve, waterfall, stagger, cascade, kinetic-text, title-card, segment-opener, nudge, slide, easing, group-motion, z-depth, motion-graphics, cinematic, transition, blur, directional-continuity]

Drama2557/4RealDigital-Video-Studio

Add captions or subtitles to an existing single-subject talking-head video without editing the footage. Use for plain verbatim captions, cinematic captions embedded behind the subject, VFX captions, “炸/特效/酷炫字幕,” or a named identity from the 35-style catalog. Route by visual identity, not by backend engine. The quiet `anchor` rail is the default; embed every word only when the user explicitly wants a fully cinematic treatment. The workflow runs locally end to end, including transcription and subject matting; split multi-shot footage before applying it.

Drama2557/4RealDigital-Video-Studio

Turn arbitrary text — an article, notes, a topic, a brief — into a faceless explainer video: there is no site or footage to capture, so the visuals are invented per scene (typography, abstract graphics, diagrams, data-viz). Use for topic explainers, concept breakdowns, how-tos, listicles. Not a video built from a website (/product-launch-video — promo or tour). Unclear → /hyperframes.

Drama2557/4RealDigital-Video-Studio

Import Figma content into a HyperFrames composition — rendered assets, brand tokens, components, storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI), connector-assisted motion when available, and shaders from a connector or native export. Use when the user pastes a figma.com link or asks to bring a Figma design, frame, logo, brand, or animation into a video/composition.

Drama2557/4RealDigital-Video-Studio

Author or edit a custom HyperFrames composition when no specialized workflow fits, or when BRIEF.md sets flow: companion. Use for longer or multi-scene pieces, brand and sizzle reels, montages, static loops, static title cards, footage remixes, and freeform builds. Use motion-graphics instead for a short unnarrated motion-first unit, including an animated title. Route fresh creation through hyperframes before using this skill.

Drama2557/4RealDigital-Video-Studio

Use the HyperFrames CLI development loop: init, add, catalog, capture, lint, check, snapshot, compare, grade-compare, preview, play, present, beats, keyframes, single or batch render, publish, cloud, cloudrun, feedback, lambda, doctor, browser, info, upgrade, skills, compositions, docs, benchmark, telemetry, transcribe, auth, tts, and remove-background. Also use when diagnosing build or render failures. validate, inspect, and layout are deprecated aliases; use check. Covers local, HeyGen-hosted cloud, AWS Lambda, and Google Cloud Run rendering.

Drama2557/4RealDigital-Video-Studio

The HyperFrames composition contract — build one renderable project. Use for composition structure, the `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, deterministic-render rules, and validation. Also covers Tailwind projects and the STORYBOARD.md / SCRIPT.md plan formats. Read before writing composition HTML.

Drama2557/4RealDigital-Video-Studio

Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use `hyperframes-animation`.

Drama2557/4RealDigital-Video-Studio

Install, discover, and wire registry blocks and components into HyperFrames compositions. Use when running hyperframes add or hyperframes catalog, installing one item or every block matching a tag, wiring an installed item into index.html, or working with hyperframes.json. Covers discovery, install locations, block sub-composition wiring, component snippet merging, and authoring a new block or component to contribute upstream (idea → scaffold → validate → PR).

Drama2557/4RealDigital-Video-Studio

Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Also use for vague feedback that real footage looks dark, flat, boring, should feel retro/camcorder/print/ASCII, needs privacy, or needs a media reveal.

Drama2557/4RealDigital-Video-Studio

GATEWAY — load FIRST before composing any HyperFrames animation or video. The high-level motion law that makes a multi-scene video feel like ONE continuous camera move instead of a stack of independently-animated slides. Covers the vector law (how you exit determines how you enter, incl. the Z scale-sign rule), the film's current, carrier elements, causal motion, the Seam Gate (build-gate enforcement), the ban on idle wobble (motion must PERFORM, not breathe), stillness-before-climax, and the sustained-motion routes. Routes to the low-level technique skills (cut-the-curve — the full catalog incl. waterfall entry + nudge curve, oversized-cursor, seam-craft). These rules SUPERSEDE generic / upstream motion guidance. [continuity, direction, vector, momentum, seam, transition, ease, performance, idle-motion, narrative-motion, film-grammar]

Drama2557/4RealDigital-Video-Studio

A short, design-led motion graphic where motion is the message — kinetic typography, stat count-up, chart/data-viz hit, logo sting / brand lockup, lower-third / callout / social overlay, animated map (highlight regions, connect places, zoom to a location), animated tweet / news-article / headline, webpage / UI animation (scroll, cursor, callouts), or fusing a real image's geometry into a chart. Usually under 10s (up to ~30s), no narration or live-action subject; renders to MP4 or transparent overlay. Longer / narrated / multi-scene → /general-video. Unclear → /hyperframes.

Drama2557/4RealDigital-Video-Studio

House-style oversized macOS cursor technique for HyperFrames launch videos. Load whenever a scene involves cursors or a pointer-led action, when kicking off a UI scene, when igniting a morph/transition/typing run with a click, or when a scene reads as static, dead, or stale and needs a cheap high-yield source of motion to carry the viewer's eye and segment them out of the stale state. Covers cursor size/look (incl. brand-motif cursors), the off-screen entry law, tip-targeting and the click tap, click-ignites-the-next-beat, and exit / cross-scene handoff.

Drama2557/4RealDigital-Video-Studio

Turn a GitHub pull request (a PR URL, owner/repo#N, or 'this PR' in a checked-out repo) into a code-change explainer video — changelog, feature reveal, fix, or refactor walkthrough built from the diff, commits, and files: the input is a code change, not a website. Not a product promo (/product-launch-video) or a no-PR topic explainer (/faceless-explainer). Unclear → /hyperframes.

Drama2557/4RealDigital-Video-Studio

Render-correctness doctrine for scene-to-scene seams in HyperFrames launch videos — the prerequisites that make transitions composite correctly on the master timeline. Load when assembling the master timeline / index.html, when a white flash appears at a cut or crossfade seam (especially on dark films), when reasoning about why a transition opacity dip shows through, or when verifying the render-side mechanics of how overlapping scene wrappers blend. Covers the opaque stage-ground (#root background) white-flash guard and how the injector overlaps wrappers, holds final frames, ping-pongs tracks, and stamps lint-clean template code onto the master timeline. Does NOT contain the per-transition catalog — see the transition registry for individual transition entries.

Drama2557/4RealDigital-Video-Studio

Author a HyperFrames slideshow — a presentation, pitch deck, or interactive deck with discrete slides, fragment reveals, branching, hotspot navigation, and built-in presenter mode with speaker notes; also converts an existing page into a deck. Output is a navigable deck, not a rendered MP4. If the user didn't explicitly ask for a slideshow, confirm before authoring. Unclear → /hyperframes.

Drama2557/4RealDigital-Video-Studio

Package an existing talking-head / interview / podcast video with timed, designed GRAPHIC OVERLAY cards — kinetic titles, lower-thirds, data callouts, quotes, side panels, picture-in-picture — synced to the transcript, on a 16:9 / 9:16 / 4:5 canvas of your choice; the clip plays untouched underneath. Trigger on "graphic overlays", "on-screen graphics", "package / dress up my video". Not plain subtitles (/embedded-captions). Unclear → /hyperframes.

Drama2557/4RealDigital-Video-Studio

Agent skill / project related to launch or product demo video: Drama2557/4RealDigital-Video-Studio

相关技能