Communitygithub.com

HyperFrames Media

为 HyperFrames 作品创建音频和媒体资源 —— 语音合成(TTS)、背景音乐、音效、转录、字幕、抠图,并将这些数据用于 HTML 动画。

HyperFrames Media 是什么?

这是一个专注于音频和媒体资源生成的 AI 技能,为 HyperFrames 创作流程提供支持。核心是统一的音频引擎 `scripts/audio.mjs`,它基于 `audio_request.json` 请求文件生成 `audio_meta.json` 输出及对应资源。引擎支持多渠道 TTS(HeyGen、ElevenLabs 或本地 Kokoro)、背景音乐(HeyGen 检索或本地 Lyria/MusicGen 生成)以及音效(HeyGen 检索或内置 21 个音效库)。它还集成了 Whisper 转录、抠图和字幕创作功能。关键设计是:所有能力依据 HeyGen 凭证是否有无自动切换,无凭证时优雅降级到本地引擎,但强制先运行 `npx hyperframes auth status` 让用户授权再生成,而不是静默降级。该技能适用于 Claude Code、Cursor、Codex 及其他 AI 代理平台,工作流必须通过统一的音频引擎而非自行实现,以保证一致性和可维护性。

兼容平台~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/1345AbhishekKumar/starter-nextjs/tree/HEAD/.agents/skills/hyperframes-media

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

HyperFrames Media 是做什么的?

为作品创建所需的音频和媒体资源 —— 语音合成(TTS)、背景音乐 + 音效、转录、字幕、抠图 —— 然后在 HTML 中消耗和动画化这些数据。如需将资源放置到作品中,请参阅 hyperframes-core。

音频引擎 —— TTS · BGM · SFX 的一站式来源

工作流不应自行实现音频或使用第三方副本。只有一个引擎 —— scripts/audio.mjs —— 它接受一个中性的 audio_request.json 并写入 audio_meta.json(以及 assets/voice|bgm|sfx 下的资源):

# <MEDIA_DIR> = 本技能的目录
node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json

所有三个能力通过一个开关降级 —— 即是否存在 HeyGen 凭证(从 $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY / ~/.heygen 解析,不是 CLI):

能力HeyGen 凭证存在不存在
TTSHeyGen Starfish REST(原生逐词时间戳)→ ElevenLabs → Kokoro(链式调用 transcribe 获取词)
BGMHeyGen 音乐检索Lyria → MusicGen 本地生成(分离模式)
SFXHeyGen 音效检索(min_score 0.4)内置 21 文件库 (assets/sfx/)
  • 请求文件 (audio_request.json):{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }。id 将每行关联回调用的模型(帧号、场景 ID 等)。bgm.mode = retrieve | generate | none;省略则自动(有凭证时检索,否则生成)。显式的 retrieve 是严格的 —— 它跳过而不是启动分离生成(适用于没有 wait-bgm 步骤的调用者)。
  • 输出 (audio_meta.json, 以 id 为键):{ tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }。
  • --only tts,bgm,sfx 运行子集并合并到现有的 --out 中(例如,先做 TTS+BGM,待线索存在后再做 SFX)。
  • BGM 生成以分离方式启动(bgm_pending: true)—— 在组装前运行 scripts/wait-bgm.mjs。
  • scripts/heygen-tts.mjs 是基于相同代码的单次 CLI(一段文本 → wav + 词),适用于只需 HeyGen TTS 而不需要请求文件的情况。

完整标志列表和 audio_meta.json 模式位于 scripts/audio.mjs 的头部注释中。以下参考资料涵盖每个能力背后的提供商细节和边缘情况。

预检 —— 在任何音频生成前显示登录状态

在生成语音或 BGM 之前务必运行此项 —— 无论是在整个工作流中还是单次“给我 BGM/语音解说”的请求中。 没有 HeyGen 凭证不是静默降级到本地引擎的理由:首先建议登录,让用户决定。运行共享预检并逐字输出其内容——不要自己编造“缺少密钥”的提示,也不要提议将密钥写入仓库特定的 .env:

npx hyperframes auth status
  • 已登录 → 打印账户信息,继续。
  • 未登录(exit 1 是预期的 —— “未登录”是正常状态,并非失败)→ 打印注册优先的指导。建议登录:npx hyperframes auth login 是浏览器 OAuth —— 它登录并创建账户(始终可通过此仓库的 CLI 使用)。要使用现有的 HeyGen API 密钥(来自 app.heygen.com/settings/api),运行 npx hyperframes auth login --api-key —— 它保存到共享的 ~/.heygen(无仓库特定的 .env)。输出还会列出语音/BGM 将降级到的本地引擎以及依赖缺失时的 pip 提示。原样输出这些内容 —— 不要用你自己的措辞改述。 然后停止并等待用户选择 —— 登录,或说“go”/“local”以继续离线 —— 在生成任何内容之前。这是一个真实的决策点,不是随口一提:不要将其合并到另一个问题中,也不要自行绕过。(例外:在自主/非交互模式下,记录状态并继续离线。)
  • npx hyperframes auth status --json 返回 { configured, recommended_action, offline_engines },用于确定性分支。
  • 如果 CLI 无法运行(不在 PATH 中且 npx 无法获取)→ 仍然建议登录(npx hyperframes auth login)并停止等待用户选择 —— 不要将“没有凭证”视为本地生成的静默绿灯。

凭证解析、完整密钥优先级和本地依赖列表见 references/requirements.md。

提供商链(引擎背后的细节)

TTS — 首个可用提供商获胜(引擎或 npx hyperframes tts "..."):

顺序提供商检测条件逐词时间戳
1HeyGen(Starfish)$HEYGEN_API_KEY / hyperframes auth login是,原生 —— 传递 --words narration.words.json 获取
2ElevenLabs$ELEVENLABS_API_KEY 已设置否 —— 之后链式调用 transcribe
3Kokoro-82M(本地,54 个语音)始终(无需密钥)否 —— 之后链式调用 transcribe

已发布的 hyperframes tts CLI 通常是仅本地构建(其 --help 显示“Kokoro-82M”,没有 --provider/--words),即使 $HEYGEN_API_KEY 已设置也会静默降级到 Kokoro。这就是为什么引擎的 HeyGen 路径是自包含的 scripts/heygen-tts.mjs(REST),而不是 CLI;CLI 仅用于 Kokoro 路径。详见 references/tts.md。

BGM 和 SFX — 默认从 HeyGen 音频库(/v3/audio/sounds)检索,凭证与 HeyGen TTS 相同,无凭证降级依据上述开关:

资源HeyGen type存放位置降级(无凭证)
BGMmusicassets/bgm/track.mp3(检索) · track.wav(生成)Lyria / MusicGen 生成
SFXsound_effects(min_score 0.4)assets/sfx/<slug>.mp3内置 21 文件库(assets/sfx/* + manifest.json)

详见 references/bgm.md 和 references/sfx.md。

路由

任务参考
音频引擎 —— 请求/元数据模式、--only、开关scripts/audio.mjs(头部注释)
npx hyperframes tts / heygen-tts.mjs —— 提供商、语音、词references/tts.md
BGM —— HeyGen 检索 + 本地 Lyria / MusicGen 生成references/bgm.md
SFX —— HeyGen 检索(min_score 0.4)+ 内置本地库references/sfx.md
npx hyperframes transcribe —— Whisper、模型规则、输出形状references/transcribe.md
npx hyperframes remove-background —— 透明抠图references/remove-background.md
TTS → 转录 → 字幕(无录制语音解说)references/tts-to-captions.md
字幕创作 —— 样式检测、布局、词分组、出口references/captions/authoring.md
转录处理 —— 输入格式、质量门控、清理、APIreferences/captions/transcript-handling.md
字幕动画 —— 卡拉 OK、标记效果、音频响应references/captions/motion.md
模型缓存、系统依赖、故障排除references/requirements.md

不可协商的规则

  • 一个引擎,无第三方副本。 通过 scripts/audio.mjs(或用于单次 HeyGen TTS 的 heygen-tts.mjs)生成音频。不要在工作流中重新实现 TTS/BGM/SFX —— 编写 audio_request.json 适配器并调用引擎。
  • “HeyGen 可用” = 可解析的凭证,而非 CLI。 整个开关依赖于 heygenCredential();已发布的 hyperframes tts 可能仅限 Kokoro,并且根本没有 hyperframes bgm / hyperframes sfx 命令。
  • 语音 ID 是提供商特定的。 am_michael 仅限 Kokoro;HeyGen UUID 在 Kokoro 上不工作。如果你传递 --voice,同时固定 --provider 以避免用户环境变化时提供商漂移。
  • 始终向 transcribe 传递 --model。 CLI 默认值 small.en 会静默翻译非英语音频。见 references/transcribe.md → “Language Rule”。
  • HeyGen 返回逐词时间戳;ElevenLabs / Kokoro 不返回。 引擎自动为后两者链式调用 transcribe;独立使用时,向 HeyGen 传递 --words 或对音频文件运行 transcribe。
  • 字幕使用平坦词数组格式,包含 { id, text, start, end }。见 references/transcribe.md → “Output Shape”。
  • remove-background --background-output 是抠洞,非修复。 对于“没有人的场景”,需要不同的工具。见 references/remove-background.md → “When NOT the right tool”。
  • BGM/SFX 默认使用 HeyGen 检索;无凭证降级是生成(BGM)或内置库(SFX)。 /audio/sounds 按文本查询排序 —— 确切命名效果(glass shatter,而不是 dramatic sound);无匹配则跳过,绝不阻塞渲染。SFX 在语音 + BGM 下音量约为 0.35。见 references/sfx.md / references/bgm.md。
  • 将工作流字幕 HTML 视为生成的输出。 对于基于预设的视频,可复用的皮肤源代码位于 .hyperframes/caption-skin.html,工作流脚本写入 compositions/captions.html;不要编辑生成的 compositions/captions.html 来修复皮肤。通过工作流的 captions.mjs 重建,或使用该工作流显式的覆盖机制(如果存在)。

Individual skills in this repo

This repo contains 13 individual skills — each has its own dedicated page.

1345AbhishekKumar/starter-nextjs

Add captions to a talking-head video. ONE catalog (CATALOG.md) of 32 visual identities behind two engines: column-flow (captions composited INTO the scene — matte occlusion + mix-blend; cream/ink/editorial/keynote/documentary/loud/neon/glitch/chrome/velocity) and themed constitutions (anchor/ordnance/terminal/neonsign/stardust/stomp/scoreboard/transit/vhs/arcade/dossier/laser/thunder/hologram/biolume/aurora/spectrum/papercut/popup/chalkboard/graffiti/brush/inkwater/ransom/lastpage/nightcity — e.g. a glyph-decode climax, a neon sign WRITTEN stroke by stroke, or the quiet `anchor` rail default). Route by identity, never by mode. Trigger on "captions/subtitles", "embed/cinematic captions", "VFX captions", "炸/特效/酷炫字幕", a named identity, or top-tier motion-graphics asks. Embedding every word is wrong for most talking-head content — `anchor` is the verbatim default. Pipeline: transcription → hyperframes remove-background matting → HTML render → ffmpeg overlay. Requires hyperframes and a single-subject clip.

1345AbhishekKumar/starter-nextjs

The fallback workflow for authoring custom HyperFrames video compositions at any length or format — longer or multi-scene pieces, brand / sizzle reels, montages, title cards, static loops, and freeform compositions. Input- and length-agnostic. If a specialized workflow clearly fits the input — a marketed product, a website, a topic explainer, a GitHub PR, existing footage, a short motion graphic, or a Remotion port — prefer it (see /hyperframes); use this only as the general fallback when none fit.

1345AbhishekKumar/starter-nextjs

All animation knowledge for HyperFrames — atomic motion rules, multi-phase scene blueprints, scene transitions, broader motion-design techniques, AND the seven runtime adapters (GSAP default, plus Lottie, Three.js, Anime.js, CSS keyframes, Web Animations API, TypeGPU). Use for any motion or animation task: pick 2-4 rules and compose, or load a blueprint, or look up runtime-specific API (e.g. GSAP eases / Lottie player / Three.js mixer). HyperFrames-native: single paused timeline, seek-safe, deterministic.

1345AbhishekKumar/starter-nextjs

HyperFrames CLI dev loop. Use when running npx hyperframes init, add, catalog, capture, lint, validate, inspect, layout, snapshot, preview, play, render, publish, lambda, doctor, browser, info, upgrade, skills, compositions, docs, benchmark, telemetry, transcribe, tts, or remove-background, or when troubleshooting the HyperFrames build/render environment. Entry point for AWS Lambda cloud rendering (`hyperframes lambda deploy / render / progress / destroy / policies`).

1345AbhishekKumar/starter-nextjs

The HyperFrames composition contract — build one renderable project. Use for composition structure, the `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, deterministic-render rules, and validation. Read before writing composition HTML.

1345AbhishekKumar/starter-nextjs

Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use `hyperframes-animation`.

1345AbhishekKumar/starter-nextjs

Install and wire registry blocks and components into HyperFrames compositions. Use when running hyperframes add, installing a block or component, wiring an installed item into index.html, or working with hyperframes.json. Covers the add command, install locations, block sub-composition wiring, component snippet merging, registry discovery, and authoring a new block or component to contribute upstream (idea → scaffold → validate → PR).

1345AbhishekKumar/starter-nextjs

Use when the user wants a short, design-led motion graphic where motion is the message: kinetic typography, stat or number count-up, chart/data-viz hit, logo sting, brand lockup, lower-third, callout, social overlay, animated headline/tweet/news item, motion poster, or quick captured-page highlight. Usually under 10s and up to ~30s, with no narration arc, voice-over, or live-action subject. Can render to MP4 or transparent overlay. Not for longer, multi-scene, narrated, or brand-reel pieces (use general-video), narrated website videos (website-to-video), topic explainers (faceless-explainer), product promos (product-launch-video), PR videos (pr-to-video), or captions on existing footage (embedded-captions). When unsure whether it's a quick motion-first piece or a longer / narrated treatment, see /hyperframes.

1345AbhishekKumar/starter-nextjs

Use when the user has a music track (an audio file, or a video to pull audio from) and wants a beat-synced HyperFrames video, calm to hard-hitting. The music drives everything: one analyzer reads it once, the orchestrator lays out the frames and fills a per-frame plan, and one sub-agent builds each frame. Typography and templates are the floor — a complete video needs zero assets — but any images or videos the user supplies are cut into the frames on the same beat grid (beat-cut / ken-burns). The genre (lyric video, slideshow, kinetic promo) falls out of the per-frame choices; the pipeline never branches on it.

1345AbhishekKumar/starter-nextjs

turn a product or marketing URL, pasted script, or brief into a product launch video, including SaaS promos, feature reveals, app launches, company promos, and product marketing videos. Use this skill when the user wants to market, launch, promote, or reveal a product. Do not use it for general non-launch website tours, non-product topic explainers, GitHub pull requests, captioning existing footage, or short unnarrated motion graphics. If the intent is unclear, route through /hyperframes first. This is the new shot-sequence architecture: every visual frame is authored as a time-coded shot sequence picked from a menu of golden blueprints, so frames develop over their full duration instead of freezing after entrance.

1345AbhishekKumar/starter-nextjs

turn a GitHub pull request (a PR URL like github.com/<owner>/<repo>/pull/<N>, an <owner>/<repo>#<N> ref, or 'this PR' in a checked-out repo) into a code-change explainer video, up to ~3 min (sweet spot 30-90s) — changelog, feature reveal, fix, or refactor walkthrough, rendered from the diff / commits / files. The input is a CODE CHANGE read via the gh CLI; there is no website capture. Use this skill for a GitHub PR. Do not use it for a product launch/promo (use /product-launch-video), a tour of a real website (use /website-to-video), a topic explainer with no PR (use /faceless-explainer), captions on existing footage (use /embedded-captions), or a short unnarrated motion graphic (use /motion-graphics). If the intent is unclear, route through /hyperframes first.

1345AbhishekKumar/starter-nextjs

Port an existing Remotion (React) composition to HyperFrames HTML. Use ONLY when the user explicitly asks to port/convert/migrate/translate a Remotion source. Do NOT use: (a) authoring a new HyperFrames composition; (b) Remotion mentioned in passing; (c) Remotion code shared as reference only; (d) "same video as my Remotion one" without explicit migrate request — treat as fresh build. Doubt → `/general-video`. One-way, Remotion-only: no reverse export (HyperFrames→Remotion or any framework), no non-Remotion source (After Effects, Framer Motion, plain React/CSS) → out of scope, re-create via `/general-video`. Flags unsupported patterns (useState, useEffect, async calculateMetadata, third-party React libs, `@remotion/lambda`) and recommends runtime interop over lossy translation. Unsure whether to port vs. build fresh, or only a passing Remotion mention? → /hyperframes.

1345AbhishekKumar/starter-nextjs

Capture a general website/URL and turn it into a HyperFrames video (site tour, showcase, or social clip from the site's own visuals). Uses headless Chrome screenshots + brand assets. Use when intent is general — portfolio/blog/landing-page showcase or social clip from the site. NOT for: product/SaaS launch or promo (→ /product-launch-video, even from a URL); topic explainer with no site (→ /faceless-explainer); GitHub PR (→ /pr-to-video); adding captions to existing video (→ /embedded-captions); short unnarrated page-highlight motion graphic (→ /motion-graphics). Unclear launch-vs-general-site? Ask one question or start at /hyperframes.

相关技能