Communitygithub.com

music-to-video — 一个以音乐为根基、节拍同步的视频工作流

将音乐曲目(音频文件、视频提取的音频或根据情绪概要生成的曲目)转换为节拍同步的视频——歌词视频、幻灯片或动态宣传片。音乐驱动所有节奏;用户提供的任何图片/视频都会被裁剪到同一节拍网格上,且完整视频无需任何素材。

music-to-video — 一个以音乐为根基、节拍同步的视频工作流 是什么?

该技能将任何音乐曲目转化为节拍同步的视频,支持歌词视频、幻灯片和动态宣传片等多种格式。它通过一个统一的音频分析脚本 `analyze-beatgrid.py` 生成音乐的节拍网格、能量阶段和关键事件,然后基于这些分析将视频切成多个帧(frames),每个帧作为一个独立的合成文件构建。用户可以选择预置模板或自由组合运动原语,并且所有用户提供的图片/视频都会被精准地裁切到节拍网格上。该技能在 Claude Code 和 HyperFrames 生态中运行,采用单子代理模式:主代理协调工作流并委派帧级构建任务给子代理。其核心设计哲学是信任单一节拍分析器,并将设计规则与执行逻辑分离,确保结果的可重复性和确定性。

兼容平台✓Claude Code~Codex CLI~Cursor
npx skills add https://github.com/1999AZZAR/designer-mcp/tree/HEAD/skills/video/engine/music-to-video

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

首先,保持此技能最新——运行前与用户确认: npx hyperframes skills update music-to-video。当一切为最新时,这是一个快速的无操作;否则它会在你依赖它们之前刷新此技能及其依赖的核心领域技能。

music-to-video — 一个以音乐为根基、节拍同步的视频工作流 是做什么的?

使用此技能将音乐曲目转换为节拍同步的 HyperFrames 视频。你分析一次曲目,布置帧,填充每帧计划,并将每帧构建为一个合成。输入是一首音乐曲目加上用户可选的图片或视频——没有旁白,也没有网站捕获。排版和模板是基础(完整视频无需任何素材);用户提供的任何媒体都会被裁切到同一节拍网格上。

你是编排者。在 videos/<project>/ 下工作。按顺序运行步骤并通过每个关卡后再继续。两个步骤需要用户:步骤 3(计划审批)和步骤 6(渲染审批)——两者都是每 ../hyperframes/references/brief-contract.md(步骤 0 之前阅读它)的检查点关卡:在自主模式下,将摘要作为提醒发布并继续,而不是等待。除步骤 4 之外,你自己完成每一步,其中你为每帧分配一个子代理。设计规则和运动规则不要放在此文件中——它们位于 references/ 和 frame-worker 子代理中。

SKILL_DIR = 此技能目录。PROJECT_DIR = videos/<project-name>/。

工作流:步骤 0 设置 → hyperframes.json + assets/bgm.mp3;步骤 1 分析 → audiomap.json;步骤 2 骨架 → STORYBOARD.md(帧,组标记为 TBD);步骤 3 计划 → 完成 STORYBOARD.md + frame.md;步骤 4 构建 → compositions/frames/NN-*.html;步骤 5 组装 → index.html;步骤 6 渲染 → renders/video.mp4。

塑造一切的两个理念

  • 一个分析器,并信任它。 analyze-beatgrid.py 是唯一的节拍分析器——绝不使用其他工具或凭耳朵重新测量节拍。其能量/密度/滚奏/起始点/静音始终可靠。其 bpm 和 beats_sec 仅在音乐真正有节奏时才可靠;在平静的音乐上,网格是追踪器强加的节拍器,因此需根据乐句和能量调整节奏,绝不硬切到它。判断你处于哪种情况是每帧的 pacing(步骤 2)。
  • 一帧一个文件;组位于内部。 步骤 2 将曲目切成帧,每帧成为一个合成文件 compositions/frames/NN-<frame_id>.html,由一个帧工作者构建。帧可细分为组(每个组是一个模板或运动原语组合)。额外的密度放在_组内_,因此帧数跟踪不同的处理方式,而非节拍——快节奏曲目不会增加子代理的数量。

步骤 0:设置、BGM 和输入

目标:确定音乐来源,创建 HyperFrames 项目,并记录用户提供的任何媒体。

简报从意图层开始。 顺序的开启规则:(1) 存在 BRIEF.md → 阅读它,并且不要问它已经回答的问题——其 flow/storyboard 衍生出模式(简报合同 § 1)。(2) 没有 BRIEF.md 但项目存在 → 从磁盘上的内容恢复;绝不重新询问。(3) 直接到达此处的全新创作请求 → 阅读 /hyperframes 并运行其意图层(references/intent-interview.md):它确认此路径的必备条件(音乐来源,目的地 → 宽高比 — ../hyperframes/references/routes/music-to-video.md)并宣布哪些内容被延迟——品牌和类型按设计在步骤 3 选择。初始化后立即写入 BRIEF.md(绝不在此之前——init 拒绝非空目录)并记录基于偏好的答案(brief-format.md)。编辑请求跳过所有这些。

音乐是主干——在任何其他事情之前确定一首曲目。此技能针对快速、高能量的 BGM 进行调整:强节拍网格驱动剪辑(平静曲目也能工作,但按乐句而不是节拍调整节奏)。如果用户提供了音频——一个音乐文件或一个要从中提取音频的视频——则使用它。否则根据请求选择情绪并通过 /media-use(references/bgm.md)生成曲目。在第一次经过身份验证的提供商操作之前,运行 npx hyperframes auth status 并逐字中继其输出。如果未登录,则应用一个分支:

  • 协作模式: 等待登录或明确选择继续使用本地提供商。
  • 自主模式: 说明状态并通过可用的本地提供商继续。

如果没有离线提供商能够满足所需的音乐能力,则说明障碍。切勿将密钥写入每仓库的 .env。验证所有权和离线回退位于 /media-use references/setup-providers.md § 提供商。生成的曲目标记为 assets/bgm.mp3。暂存提供的图片或视频,以便帧可以在节拍网格上使用它们;否则排版带动视频。

歌词视频: 对于与歌声同步的歌词,通过 /media-use 转录曲目获得单词/行时间,或向用户索要歌词文本并将行放在节拍网格上。

仅当缺少 hyperframes.json 时才初始化。从简报中以 kebab-case 命名 <project>,例如 midnight-drive-loop——绝不要时间戳。init 检查已安装的技能是否与 GitHub 上的最新版本一致,如果有任何过时则更新全局集合。

npx hyperframes init "videos/<project>" --non-interactive --example=blank --skill=music-to-video
mkdir -p "$PROJECT_DIR/assets" "$PROJECT_DIR/renders"
cp "<user-music>" "$PROJECT_DIR/assets/bgm.mp3"   # 如果需要,先从视频中提取
# 仅当用户提供了图片/视频:
node <SKILL_DIR>/scripts/stage-assets.mjs --from <dir> --hyperframes "$PROJECT_DIR" --into public

品牌(字体 + 调色板)在步骤 3 选择,而不是在这里。不要预先选择类型或曲目类型——素材只是一个可选成分,类型从每帧的选择中浮现。

关卡: 存在 hyperframes.json + assets/bgm.mp3;记录了宽高比/长度/帧率以及(如果有的话)素材清单。


步骤 1:分析音乐

目标:生成整个视频所依赖的规范时间分析。

analyze-beatgrid.py 是唯一的节拍分析器——绝不使用其他工具或凭耳朵重新测量节拍。它读取一次曲目并写入 audiomap.json:能量阶段(级别/密度/感觉)、起始点 + onset_rate、滚奏、静音、hard_stops、key_moments、乐句、节奏/网格以及 audio.duration_sec。它是确定性的——同一文件始终给出同一映射。大多数字段在任何音乐上都可靠;bpm 和 beats_sec 仅在音乐真正有节奏时才可靠,判断这一点是你在步骤 2 中做出的决定。

前提条件:Python 3 具有可用的 librosa、numpy 和 soundfile。如果导入失败,在运行分析器之前将它们安装到活动的 Python 环境中:

python3 -m pip install librosa numpy soundfile
python3 <SKILL_DIR>/scripts/analyze-beatgrid.py "$PROJECT_DIR/assets/bgm.mp3" \
  -o "$PROJECT_DIR/audiomap.json" --print

关卡: 存在 audiomap.json;已知 audio.duration_sec。


步骤 2:帧骨架(仅结构)

目标:读取音乐并布局帧——STORYBOARD.md 的骨架。

阅读 references/frame-skeleton.md。你自己将 audiomap.json 转化为 STORYBOARD.md 的骨架——不存在中间 JSON。在真实的音乐变化处(hard_stops、SURGE/DROP key_moments、滚奏的边缘、没有起始点的段落、大的能量跳跃)将曲目切成帧,将每个边界锚定到一个音频图锚点。为每帧设置 span_sec、pacing(来自步骤 1 信任判断的结论——当网格真实时为 beat_cut,当它是平静音乐上强加的节拍器时为 phrase_flow)、mood 和一行 feel(步骤 3 用于匹配模板的纯音乐情况)。在此处仅进行分类和布局:将每帧的 ### Groups 留作 TBD (Step 3),并将前部的 style 留空——不涉及模板、副本、颜色或字体。预期约 1-6 帧。

关卡: 帧覆盖整个曲目(第一帧在 0 处,最后一帧在 duration_s 处);每帧携带 span_sec + pacing + mood + feel;每个 ### Groups 都是 TBD;任何地方都没有内容。


步骤 3:填充计划(用户关隘)

目标:将骨架转化为一个经过批准的、完整的 STORYBOARD.md。

阅读 references/planning.md、storyboard-format.md、template-catalog.md、motion-primitive-catalog.md 和 montage.md(仅在用户提供了素材时)。在原地编辑同一文件,执行两个操作:

  1. 选择品牌。 使用 ../hyperframes-creative/references/design-spec.md 中的表格,从 ../hyperframes-creative/frame-presets/ 中选择一个预设(匹配曲目的情绪;仅其字体和颜色重要——模板拥有构图)。将其未修改地复制到 frame.md 中,并从其中填充前部 style(字体 + 最多 4-6 个色板调色板)。
  2. 填充每帧。 决定其组,并为每个组指定处理方式:来自目录的匹配模板(带有绑定参数和真实的音频图锚点)、来自原语目录的自由构图、或遵循 pacing 的素材处理方式。在自由构图一个命名外观之前,先搜索实时目录: 对于用户要求的每个外观、效果、处理或过渡——“CRT 扫描线”、“故障”、“胶片颗粒”、“闪光扫过”——运行 npx hyperframes catalog --query "<外观,用纯英文描述>" --json 并阅读前几个结果。template-catalog.md 和 motion-primitive-catalog.md 仅列出此技能自身的本地素材;搜索对整个托管注册表(约 400 个块和组件)进行排名,并且无需安装任何东西——无需项目、无需事先 add、无需账户。仅当搜索没有找到合适的匹配后才自由构图一个外观。编写副本。你掌握 WHAT(模板/原语 + 内容 + 锚点);帧工作者掌握 HOW——绝对不要将毫秒级补间写入故事板。
node <SKILL_DIR>/scripts/validate-plan.mjs --storyboard "$PROJECT_DIR/STORYBOARD.md" \
  --audiomap "$PROJECT_DIR/audiomap.json" --templates <SKILL_DIR>/references/templates

修复每个 ✗(硬错误:持续时间不匹配、帧未覆盖曲目、缺少 src);警告是尽力而为。然后在聊天中将逐帧摘要作为提案呈现(../hyperframes/references/review-loop.md § 1)并根据用户的回复进行迭代,直到他们批准;对于 storyboard: yes,还将其编写为 storyboard.html(../hyperframes-creative/references/storyboard-recipe.md § 3),供他们打开。在自主模式下,这是一个检查点关卡:将摘要作为提醒发布并继续(validate-plan.mjs 通过是一个质量关卡,仍然会阻塞)。

关卡: frame.md 是逐字预设副本;validate-plan.mjs 以退出码 0 退出;用户批准了计划(自主模式:摘要已作为提醒发布)。


步骤 4:根据计划构建帧

目标:将每帧构建为独立的合成文件。

创建 compositions/frames/。阅读 sub-agents/frame-worker.md 和 ../hyperframes/references/subagent-dispatch.md。为每帧分配一个帧工作者,尽可能并行(否则分批)。每个工作者恰好获得一帧和此上下文:

PROJECT_DIR: <绝对路径>
frame_id: <NN-frame_id>              # = 帧文件名主干,例如 02-f2;合成 id
你的块:在 PROJECT_DIR/STORYBOARD.md 中的 `## Frame N — <frame_id>` 块
audiomap: PROJECT_DIR/audiomap.json
frame.md: PROJECT_DIR/frame.md
素材:对于每个组,<SKILL_DIR>/references/templates/<id>/index.html(模板)和
          <SKILL_DIR>/references/motion-primitives/<id>/(自由);暂存的 assets/(素材组)
合同:../hyperframes-core/references/sub-compositions.md + determinism-rules.md
画布:<宽>×<高>   节奏:<beat_cut|phrase_flow>
写入:PROJECT_DIR/compositions/frames/<frame_id>.html

工作者派生引用的素材,将每个锚点转换为帧本地秒数(local_t = track_t − span_sec[0]),用 0ms 剪辑门控其组,并写入一个 seek-safe 的帧文件。工作者从不运行 hyperframes CLI ——这些命令操作已组装的项目,该项目尚不存在,因此它们会报告错误的文件。工作者仅根据合同写入内容并停止;你在组装后(步骤 6)进行验证。每当一个工作者返回时,你可以确认其文件已落地到磁盘。

关卡: 每帧都有其 compositions/frames/NN-*.html 存在于磁盘上。


步骤 5:组装

目标:将构建的帧 + BGM 连成可播放的 index.html。

assemble-index.mjs 是确定性的——没有子代理,没有判断。它以其累积的 data-start 引用每个帧文件,将 assets/bgm.mp3 安装在轨道 11 上,并硬切帧 → 帧(帧覆盖曲目且无间隙,因此没有过渡注入器)。

node <SKILL_DIR>/scripts/assemble-index.mjs --storyboard "$PROJECT_DIR/STORYBOARD.md" \
  --hyperframes "$PROJECT_DIR" --audiomap "$PROJECT_DIR/audiomap.json"

修复其报告的任何 ✗ ——缺失或空白的帧文件意味着该工作者写入了部分文件;重新分发它(步骤 4)并重新组装。

关卡: 存在 index.html;总持续时间 == audiomap.audio.duration_sec。


步骤 6:验证并渲染

目标:验证组装的视频,获得用户批准,并渲染最终的 MP4。

在已组装的项目上运行 CLI —— 那是正确的单元(每帧工作者无法运行它)。check 在一次通过中运行结构 lint 以及无头浏览器运行时、布局、运动和对比度检查;--snapshots 还会生成审查帧。

( cd "$PROJECT_DIR" && npx hyperframes check . --snapshots )

检查 t=0、每帧开始处、最强的 DROP/SURGE、每个 hard_stops[].t、

……

[已截断用于翻译]

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

1999AZZAR/designer-mcp

SOTA animation and physics heuristic guide using motion.dev

1999AZZAR/designer-mcp

Add captions or subtitles to an existing single-subject talking-head video without editing the footage. Use for plain verbatim captions, cinematic captions embedded behind the subject, VFX captions, “炸/特效/酷炫字幕,” or a named identity from the 35-style catalog. Route by visual identity, not by backend engine. The quiet `anchor` rail is the default; embed every word only when the user explicitly wants a fully cinematic treatment. The workflow runs locally end to end, including transcription and subject matting; split multi-shot footage before applying it.

1999AZZAR/designer-mcp

Turn arbitrary text — an article, notes, a topic, a brief — into a faceless explainer video: there is no site or footage to capture, so the visuals are invented per scene (typography, abstract graphics, diagrams, data-viz). Use for topic explainers, concept breakdowns, how-tos, listicles. Not a video built from a website (/product-launch-video — promo or tour). Unclear → /hyperframes.

1999AZZAR/designer-mcp

Import Figma content into a HyperFrames composition — rendered assets, brand tokens, components, storyboard sections → reconstructed motion (frames read as states, not slides) (REST/CLI), connector-assisted motion when available, and shaders from a connector or native export. Use when the user pastes a figma.com link or asks to bring a Figma design, frame, logo, brand, or animation into a video/composition.

1999AZZAR/designer-mcp

Author or edit a custom HyperFrames composition when no specialized workflow fits, or when BRIEF.md sets flow: companion. Use for longer or multi-scene pieces, brand and sizzle reels, montages, static loops, static title cards, footage remixes, and freeform builds. Use motion-graphics instead for a short unnarrated motion-first unit, including an animated title. Route fresh creation through hyperframes before using this skill.

1999AZZAR/designer-mcp

All animation knowledge for HyperFrames — atomic motion rules, multi-phase scene blueprints, scene transitions, broader motion-design techniques, AND the seven runtime adapters (GSAP default, plus Lottie, Three.js, Anime.js, CSS keyframes, Web Animations API, TypeGPU). Use for any motion or animation task: pick 2-4 rules and compose, or load a blueprint, or look up runtime-specific API (e.g. GSAP eases / Lottie player / Three.js mixer). Also covers auditing an existing composition's choreography (animation map) and 24 named text-animation effects. HyperFrames-native: single paused timeline, seek-safe, deterministic.

1999AZZAR/designer-mcp

Use when audio already placed in a HyperFrames composition needs to be mixed: fade-in/fade-out, crossfade, track gain or volume, volume automation, ducking, a music bed that fights a voiceover (voiceover carve), effects on a track (EQ, compressor, limiter, gate, saturation, delay, reverb, chorus, phaser, bitcrush), automation envelopes drawn on a track's volume or any effect parameter, or one submix bus carrying a chain, a fader and an automation clock for several tracks at once (`<hf-audio-group>`). Don't use for sourcing or generating audio — finding BGM, SFX, or making a voiceover is `/media-use`. Don't use for clip timing or track layout, which is `/hyperframes-core`.

1999AZZAR/designer-mcp

Use the HyperFrames CLI development loop: init, add, catalog, capture, lint, check, snapshot, compare, grade-compare, preview, play, present, beats, keyframes, single or batch render, publish, cloud, cloudrun, feedback, lambda, doctor, browser, info, upgrade, skills, compositions, timeline, docs, benchmark, telemetry, transcribe, auth, tts, and remove-background. Also use when diagnosing build or render failures. validate, inspect, and layout are deprecated aliases; use check. Covers local, HeyGen-hosted cloud, AWS Lambda, and Google Cloud Run rendering.

1999AZZAR/designer-mcp

The HyperFrames composition contract — build one renderable project. Use for composition structure, the `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, deterministic-render rules, and validation. Read before writing composition HTML.

1999AZZAR/designer-mcp

Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use `hyperframes-animation`.

1999AZZAR/designer-mcp

Use when a HyperFrames composition needs a punch-in, punch-out, zoom, reframe, Ken Burns treatment, camera move, visual match/whip handoff, or other seek-safe 2D/3D keyframes; also for GSAP, CSS keyframes, Anime.js, WAAPI, FLIP, paths, masks, SVG morph/draw, text trails, 3D depth, or `hyperframes keyframes` diagnostics. Don't use for broad scene strategy, brand design, media sourcing, captions, or general video planning.

1999AZZAR/designer-mcp

Search, install, and wire registry blocks and components into HyperFrames compositions. Use BEFORE hand-building any named visual — whenever a brief, a user, or a storyboard names a look, effect, treatment, or transition such as CRT scanlines, glitch, chromatic aberration, film grain, a shimmer sweep, a chart, a code or terminal window, a map, or a confetti burst — because roughly 400 hosted items already cover many of them and the search ranks all of them with nothing installed, no project, and no account. Also use when running hyperframes add or hyperframes catalog, installing one item or every block matching a tag, wiring an installed item into index.html, or working with hyperframes.json. Covers discovery, install locations, block sub-composition wiring, component snippet merging, and authoring a new block or component to contribute upstream (idea → scaffold → validate → PR).

1999AZZAR/designer-mcp

Mandatory entry point: read this first for any request to make, create, edit, animate, or render a video, animation, or motion graphic, including a promo, explainer, captioned clip, title card, overlay, slideshow or interactive deck, Remotion port, or any HyperFrames HTML composition. Also use it to inspect, diagnose, validate, preview, publish, or batch-render an existing HyperFrames project. Inputs may be a website URL, GitHub PR, Figma design or URL, text or brief, existing footage, or music. It resumes project state, captures intent when applicable, selects and installs the owning workflow, and routes domain capabilities. HyperFrames is the default output framework unless the user explicitly chooses another framework for the deliverable or asks only to record a browser session.

1999AZZAR/designer-mcp

Use when building or editing a HyperFrames project that people open in Studio: how the timeline should be laid out so it reads well (one caption track, one element kind per track, every scene a sub-composition) and where captions and key content may sit (safe zones). Don't use for how to perform an individual edit (split, trim, retime, volume, copy, swap): that is `creator-editing-recipes.md` in `/hyperframes-core`.

1999AZZAR/designer-mcp

Agent Media OS, the single skill for every media need in a HyperFrames project. Resolve BGM, SFX, image, icon, brand logo, voice, color grade, or LUT into a frozen local file or paste-ready block + ledger record (one verb, `resolve`); generate via TTS / music / image models when the catalog misses; produce voiceover, transcription, captions, and background removal through one shared audio engine; operate on media (cut / reframe / transform); and reuse assets across projects. Also use for vague feedback that real footage looks dark, flat, boring, should feel retro/camcorder/print/ASCII, needs privacy, or needs a media reveal.

1999AZZAR/designer-mcp

Turn a GitHub pull request (a PR URL, owner/repo#N, or 'this PR' in a checked-out repo) into a code-change explainer video — changelog, feature reveal, fix, or refactor walkthrough built from the diff, commits, and files: the input is a code change, not a website. Not a product promo (/product-launch-video) or a no-PR topic explainer (/faceless-explainer). Unclear → /hyperframes.

1999AZZAR/designer-mcp

Port an existing Remotion (React) composition''s source to HyperFrames HTML. Use ONLY on an explicit ask to port/convert/migrate/translate a Remotion source — one-way, Remotion-only. A passing Remotion mention, reference-only code, or "make something like my Remotion video" is a fresh build (/general-video). Unclear → /hyperframes.

1999AZZAR/designer-mcp

Author a HyperFrames slideshow — a presentation, pitch deck, or interactive deck with discrete slides, fragment reveals, branching, hotspot navigation, and built-in presenter mode with speaker notes; also converts an existing page into a deck. Output is a navigable deck, not a rendered MP4. If the user didn't explicitly ask for a slideshow, confirm before authoring. Unclear → /hyperframes.

1999AZZAR/designer-mcp

Package an existing talking-head / interview / podcast video with timed, designed GRAPHIC OVERLAY cards — kinetic titles, lower-thirds, data callouts, quotes, side panels, picture-in-picture — synced to the transcript, on a 16:9 / 9:16 / 4:5 canvas of your choice; the clip plays untouched underneath. Trigger on "graphic overlays", "on-screen graphics", "package / dress up my video". Not plain subtitles (/embedded-captions). Unclear → /hyperframes.

1999AZZAR/designer-mcp

All animation knowledge for HyperFrames — atomic motion rules, multi-phase scene blueprints, scene transitions, broader motion-design techniques, AND the seven runtime adapters (GSAP default, plus Lottie, Three.js, Anime.js, CSS keyframes, Web Animations API, TypeGPU). Use for any motion or animation task: pick 2-4 rules and compose, or load a blueprint, or look up runtime-specific API (e.g. GSAP eases / Lottie player / Three.js mixer). Also covers auditing an existing composition's choreography (animation map) and 24 named text-animation effects. HyperFrames-native: single paused timeline, seek-safe, deterministic.

相关技能