Communitygithub.com

9018/mychat

音频与媒体资源创建技能,用于 HyperFrames 合成,由统一音频引擎驱动 —— 提供多供应商 TTS、背景音乐、音效、转写、背景去除及字幕制作功能。

mychat 是什么?

该技能为 HyperFrames 合成系统提供完整的音频与媒体资源创建能力,由统一的音频引擎 (`scripts/audio.mjs`) 驱动。它支持多供应商语音合成(TTS),包括 HeyGen、ElevenLabs 和本地的 Kokoro,并且会根据是否配置 HeyGen 凭证自动选择最佳供应商。背景音乐和音效默认从 HeyGen 音频库检索,没有凭证时则回退到本地生成(Lyria/MusicGen)或捆绑的音效库。此外,该技能还提供 Whisper 转写、背景去除、字幕制作(支持逐词样式、卡拉OK效果)等功能,并允许用户选择语音、供应商和音乐情绪。其核心设计强调“单一引擎,无供应商拷贝”,所有音频生成都必须通过引擎脚本,避免工作流中重复实现。

兼容平台~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/9018/mychat/tree/HEAD/OpenMontage/.agents/skills/hyperframes-media

在你喜欢的 AI 中提问

打开一个已预加载此 Agent Skill 的新对话。

文档

HyperFrames 媒体

创建合成所需的音频和媒体资源 —— 配音(TTS)、背景音乐 + 音效、转写、字幕、背景去除 —— 然后在 HTML 中使用和动画化这些数据。要将资源放入合成中,请查看 hyperframes-core。

音频引擎 —— TTS · BGM · SFX 的统一来源

工作流不应当手动处理音频或供应商拷贝。只有一个引擎 —— scripts/audio.mjs —— 它接受一个中性的 audio_request.json 并写入 audio_meta.json(以及 assets/voice|bgm|sfx 目录下的资源):

# <MEDIA_DIR> = 本技能的目录
node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json

所有三种能力都依赖于一个开关 —— 是否存在 HeyGen 凭证(通过 $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY / ~/.heygen 解析,不是 CLI):

能力HeyGen 凭证存在不存在
TTSHeyGen Starfish REST(原生单词时间戳)→ ElevenLabs → Kokoro(链式 transcribe 获取单词)
BGMHeyGen 音乐检索Lyria → MusicGen 本地生成(分离启动)
SFXHeyGen 音效检索(min_score 0.4)捆绑的 21 文件库(assets/sfx/)
  • 请求(audio_request.json):{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }。id 将每行内容连接回调用者的模型(帧编号、场景 ID 等)。bgm.mode = retrieve | generate | none;省略则为自动(有凭证时检索,否则生成)。显式的 retrieve 是严格的 —— 如果没有凭证则跳过,而不是启动一个分离的生成任务(适用于没有 wait-bgm 步骤的调用者)。
  • 输出(audio_meta.json,以 ID 为键):{ tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }。
  • --only tts,bgm,sfx 运行子集并合并到已有的 --out 中(例如,提前运行 TTS+BGM,等操作提示存在后再运行 SFX)。
  • BGM 生成以分离方式启动(bgm_pending: true)—— 在组装前运行 scripts/wait-bgm.mjs。
  • scripts/heygen-tts.mjs 是基于相同代码的单次 CLI(一段文本 → wav + 单词),适用于不需要请求文件时仅使用 HeyGen TTS 的场景。

完整的标志列表和 audio_meta.json 模式位于 scripts/audio.mjs 的开头注释中。下面的参考文献涵盖了每种能力背后的供应商细节和边界情况。

预检 —— 在任何音频之前显示登录状态

在生成配音或背景音乐之前始终运行此步骤 —— 无论是在完整工作流内部还是单独的“生成一段背景音乐/配音”请求中。 没有 HeyGen 凭证不是静默回退到本地引擎的理由:首先建议登录,让用户决定。运行共享预检并逐字输出其结果 —— 不要自己编写“缺少密钥”的提示,也不要提议写入每个仓库的 .env:

npx hyperframes auth status
  • 已登录 → 打印账户信息;继续。
  • 未登录(exit 1 在此是预期的 —— “未登录”是正常状态,不是失败) → 打印注册优先的指导信息。建议登录:npx hyperframes auth login 是浏览器 OAuth —— 它会登录并创建账户(通过此仓库的 CLI 始终可用)。要使用现有的 HeyGen API 密钥(来自 app.heygen.com/settings/api),运行 npx hyperframes auth login --api-key —— 它会保存到共享的 ~/.heygen 中(不创建每个仓库的 .env)。输出还会列出本地引擎在语音/背景音乐方面的回退选项,以及依赖缺失时的 pip 提示。原样输出此结果 —— 不要用自己的措辞重新阐述。 然后停止并等待用户选择 —— 登录,或者说“GO”/“local”以离线继续 —— 然后才开始生成任何内容。 这是一个真正的决策点,不是路过提示:不要把它折叠到另一个问题中,也不要自行继续。(例外:在自治/非交互模式下,记录状态并离线继续。)
  • npx hyperframes auth status --json 返回 { configured, recommended_action, offline_engines },用于确定性分支。
  • 如果 CLI 无法运行(不在 PATH 中且 npx 无法获取)→ 仍然建议登录(npx hyperframes auth login)并等待用户选择 —— 不要将“没有凭证”视为静默允许本地生成的绿灯。

凭证解析、完整密钥优先级和本地依赖项列表位于 references/requirements.md。

供应商链(引擎背后的细节)

TTS —— 第一个可用的供应商胜出(引擎,或 npx hyperframes tts "..."):

顺序供应商检测条件单词时间戳
1HeyGen(Starfish)$HEYGEN_API_KEY / hyperframes auth login是,原生 —— 传递 --words narration.words.json 捕获
2ElevenLabs已设置 $ELEVENLABS_API_KEY否 —— 之后链式 transcribe
3Kokoro-82M(本地,54 个语音)始终可用(无需密钥)否 —— 之后链式 transcribe

已发布的 hyperframes tts CLI 通常是仅本地构建(其 --help 显示“Kokoro-82M”,没有 --provider/--words 选项),即使设置了 $HEYGEN_API_KEY 也会静默回退到 Kokoro。这就是引擎的 HeyGen 路径是独立的 scripts/heygen-tts.mjs(REST)而不是 CLI 的原因;CLI 仅用于 Kokoro 路径。参见 references/tts.md。

BGM 与 SFX —— 默认从 HeyGen 音频库检索(/v3/audio/sounds),与 HeyGen TTS 使用相同的凭证,如果未设置凭证则使用上面开关指定的回退:

资源HeyGen type存放位置回退(无凭证)
BGMmusicassets/bgm/track.mp3(检索)· track.wav(生成)Lyria / MusicGen 生成
SFXsound_effects(min_score 0.4)assets/sfx/<slug>.mp3捆绑的 21 文件库(assets/sfx/* + manifest.json)

参见 references/bgm.md 和 references/sfx.md。

路由

任务阅读
音频引擎 —— 请求/元数据模式、--only、开关scripts/audio.mjs(开头的注释)
npx hyperframes tts / heygen-tts.mjs —— 供应商、语音、单词references/tts.md
BGM —— HeyGen 检索 + 本地 Lyria / MusicGen 生成references/bgm.md
SFX —— HeyGen 检索(min_score 0.4)+ 捆绑的本地库references/sfx.md
npx hyperframes transcribe —— Whisper、模型规则、输出形状references/transcribe.md
npx hyperframes remove-background —— 透明裁切references/remove-background.md
TTS → 转写 → 字幕(无录制配音)references/tts-to-captions.md
字幕制作 —— 样式检测、布局、单词分组、退出references/captions/authoring.md
转录文本处理 —— 输入格式、质量门限、清理、APIreferences/captions/transcript-handling.md
字幕动画 —— 卡拉OK、标记效果、音频响应references/captions/motion.md
模型缓存、系统依赖、故障排除references/requirements.md

不可协商的规则

  • 单一引擎,无供应商拷贝。 通过 scripts/audio.mjs(或单次 HeyGen TTS 使用 heygen-tts.mjs)生成音频。不要在工作流中重新实现 TTS/BGM/SFX —— 编写 audio_request.json 适配器并调用引擎。
  • “HeyGen 可用” = 可解析的凭证,不是 CLI。 整个开关取决于 heygenCredential();发布的 hyperframes tts 可能只是 Kokoro 版本,并且根本没有 hyperframes bgm / hyperframes sfx 命令。
  • 语音 ID 是供应商特定的。 am_michael 仅 Kokoro 使用;HeyGen UUID 在 Kokoro 上不工作。如果传递 --voice,也请同时明确指定 --provider,以避免用户环境变化时供应商静默偏移。
  • 始终向 transcribe 传递 --model。 CLI 默认 small.en 会静默翻译非英语音频。参见 references/transcribe.md → “语言规则”。
  • HeyGen 返回单词时间戳;ElevenLabs / Kokoro 不返回。 引擎会自动为后两者链式运行 transcribe;单独使用时,向 HeyGen 传递 --words 或对音频文件运行 transcribe。
  • 字幕使用扁平单词数组格式,包含 { id, text, start, end }。参见 references/transcribe.md → “输出形状”。
  • remove-background --background-output 是挖洞,不是修复。 对于“场景中没有人物”,需要不同的工具。参见 references/remove-background.md → “何时不是合适的工具”。
  • BGM/SFX 默认从 HeyGen 检索;无凭证回退是生成(BGM)或捆绑的库(SFX)。 /audio/sounds 按文本查询排序 —— 给效果起具体名称(glass shatter,而不是 dramatic sound);未匹配则跳过,永不阻塞渲染。在语音 + BGM 下,SFX 音量约为 ~0.35。参见 references/sfx.md / references/bgm.md。
  • 将工作流中的字幕 HTML 视为生成的输出。 对于基于预设的视频,可重用的皮肤源位于 .hyperframes/caption-skin.html,工作流脚本写入 compositions/captions.html;不要编辑生成的 compositions/captions.html 来修复皮肤。通过工作流的 captions.mjs 重新构建,或者使用该工作流中存在的显式覆盖机制。

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

9018/mychat

Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax, Gemini Omni). Supports gateways: HeyGen API, fal.ai API, Kling official direct API, and the Gemini API (Gemini Omni Flash).

9018/mychat

Create AI avatar videos with precise control over avatars, voices, scripts, scenes, and backgrounds using HeyGen's v2 API. Use when: (1) Choosing a specific avatar and voice for a video, (2) Writing exact scripts for an avatar to speak, (3) Building multi-scene videos with different backgrounds per scene, (4) Creating transparent WebM videos for compositing, (5) Using talking photos as video presenters, (6) Integrating HeyGen avatars with Remotion, (7) Batch video generation with exact specs, (8) Brand-consistent production videos with precise control.

9018/mychat

Use p5.js/canvas for local procedural character effects: particles, weather, squash/stretch, walk cycles, and environmental motion.

9018/mychat

Review local character animation with schema checks, Playwright browser previews, frame sampling, and FFmpeg/ffprobe final output checks.

9018/mychat

Create videos from a text prompt using HeyGen's Video Agent. Use when: (1) Creating a video from a description or idea, (2) Generating explainer, demo, or marketing videos from a prompt, (3) Making a video without specifying exact avatars, voices, or scenes, (4) Quick video prototyping or drafts, (5) One-shot prompt-to-video generation, (6) User says "make me a video" or "create a video about X".

9018/mychat

Generate Mandarin and multilingual narration with Volcengine Doubao Speech 2.0. Use when creating Chinese voiceovers, when the user prefers Doubao/Volcengine/火山引擎/豆包 TTS, or when narration needs character-level timestamp metadata for subtitles.

9018/mychat

Video and audio processing with FFmpeg. Use for format conversion, resizing, compression, audio extraction, and preparing assets for Remotion. Triggers include converting GIF to MP4, resizing video, extracting audio, compressing files, or any media transformation task.

9018/mychat

Use when implementing Disney's 12 animation principles with Framer Motion in React applications

9018/mychat

All animation knowledge for HyperFrames — atomic motion rules, multi-phase scene blueprints, scene transitions, broader motion-design techniques, AND the seven runtime adapters (GSAP default, plus Lottie, Three.js, Anime.js, CSS keyframes, Web Animations API, TypeGPU). Use for any motion or animation task: pick 2-4 rules and compose, or load a blueprint, or look up runtime-specific API (e.g. GSAP eases / Lottie player / Three.js mixer). HyperFrames-native: single paused timeline, seek-safe, deterministic.

9018/mychat

HyperFrames CLI dev loop. Use when running npx hyperframes init, add, catalog, capture, lint, validate, inspect, layout, snapshot, preview, play, render, publish, lambda, doctor, browser, info, upgrade, skills, compositions, docs, benchmark, telemetry, transcribe, tts, or remove-background, or when troubleshooting the HyperFrames build/render environment. Entry point for AWS Lambda cloud rendering (`hyperframes lambda deploy / render / progress / destroy / policies`).

9018/mychat

The HyperFrames composition contract — build one renderable project. Use for composition structure, the `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, deterministic-render rules, and validation. Read before writing composition HTML.

9018/mychat

Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use `hyperframes-animation`.

9018/mychat

Install and wire registry blocks and components into HyperFrames compositions. Use when running hyperframes add, installing a block or component, wiring an installed item into index.html, or working with hyperframes.json. Covers the add command, install locations, block sub-composition wiring, component snippet merging, registry discovery, and authoring a new block or component to contribute upstream (idea → scaffold → validate → PR).

9018/mychat

Use when the user wants a short, design-led motion graphic where motion is the message: kinetic typography, stat or number count-up, chart/data-viz hit, logo sting, brand lockup, lower-third, callout, social overlay, animated headline/tweet/news item, motion poster, or quick captured-page highlight. Usually under 10s and up to ~30s, with no narration arc, voice-over, or live-action subject. Can render to MP4 or transparent overlay. Not for longer, multi-scene, narrated, or brand-reel pieces (use general-video), narrated website videos (website-to-video), topic explainers (faceless-explainer), product promos (product-launch-video), PR videos (pr-to-video), or captions on existing footage (embedded-captions). When unsure whether it's a quick motion-first piece or a longer / narrated treatment, see /hyperframes.

9018/mychat

Use when the user has a music track (an audio file, or a video to pull audio from) and wants a beat-synced HyperFrames video, calm to hard-hitting. The music drives everything: one analyzer reads it once, the orchestrator lays out the frames and fills a per-frame plan, and one sub-agent builds each frame. Typography and templates are the floor — a complete video needs zero assets — but any images or videos the user supplies are cut into the frames on the same beat grid (beat-cut / ken-burns). The genre (lyric video, slideshow, kinetic promo) falls out of the per-frame choices; the pipeline never branches on it.

9018/mychat

Toolkit-specific Remotion patterns — custom transitions, shared components, and project conventions. For core Remotion framework knowledge (hooks, animations, rendering, etc.), see the `remotion-official` skill.

9018/mychat

Port an existing Remotion (React) composition to HyperFrames HTML. Use ONLY when the user explicitly asks to port/convert/migrate/translate a Remotion source. Do NOT use: (a) authoring a new HyperFrames composition; (b) Remotion mentioned in passing; (c) Remotion code shared as reference only; (d) "same video as my Remotion one" without explicit migrate request — treat as fresh build. Doubt → `/general-video`. One-way, Remotion-only: no reverse export (HyperFrames→Remotion or any framework), no non-Remotion source (After Effects, Framer Motion, plain React/CSS) → out of scope, re-create via `/general-video`. Flags unsupported patterns (useState, useEffect, async calculateMetadata, third-party React libs, `@remotion/lambda`) and recommends runtime interop over lossy translation. Unsure whether to port vs. build fresh, or only a passing Remotion mention? → /hyperframes.

9018/mychat

Animate SVG character rigs with GSAP, CSS transforms, Remotion frame control, and HyperFrames-compatible browser previews.

9018/mychat

Three.js animation - keyframe animation, skeletal animation, morph targets, animation mixing. Use when animating objects, playing GLTF animations, creating procedural motion, or blending animations.

9018/mychat

Download video and audio from YouTube and 1000+ sites using yt-dlp. No API keys needed. Use when: (1) Downloading a video from YouTube or other sites, (2) Extracting audio from a video URL, (3) Downloading subtitles/captions from a video, (4) Getting video metadata without downloading.

相关技能