Communitygithub.com

9018/mychat

為 HyperFrames 合成作品創建音訊和多媒體資產——配音(TTS)、背景音樂 + 音效、轉錄、字幕、去背——然後在 HTML 中使用這些數據。

mychat 是什麼?

此技能提供一個統一的音訊引擎(`scripts/audio.mjs`),用於處理 HyperFrames 作品所需的所有音訊和多媒體資產。它支援多供應商的文字轉語音(TTS),依序嘗試 HeyGen(含原生文字時間戳)、ElevenLabs 和本機的 Kokoro-82M。背景音樂(BGM)和音效(SFX)預設從 HeyGen 的音訊庫擷取,若無憑證時則優雅降級為本地 Lyria/MusicGen 生成或內建的 21 個 SFX 檔案庫。該引擎也整合了 Whisper 轉錄、去背和進階字幕創作(含卡拉 OK 樣式),並產出結構化的 `audio_meta.json`。一個關鍵的飛航前檢查會在使用者不具備 HeyGen 憑證時提示登入,確保在降級為離線引擎前獲得知情同意。此技能專為 Claude Code、Cursor 和 Codex 等 AI 代理設計,使其能夠以程式化方式將語音、音樂和音效整合到 HTML 作品中。

相容平台~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/9018/mychat/tree/HEAD/OpenMontage/.agents/skills/hyperframes-media

在你喜歡的 AI 中提問

開啟一個已預先載入此 Agent Skill 的新對話。

說明文件

HyperFrames Media

建立作品所需的音訊和多媒體資產——配音(TTS)、背景音樂 + 音效、轉錄、字幕、去背——然後在 HTML 中消費並動畫處理這些數據。如需將資產放入作品,請參閱 hyperframes-core。

音訊引擎——TTS · BGM · SFX 的單一來源

工作流程不會自行製作音訊或外包複製品。只有一個引擎——scripts/audio.mjs——它接收中性的 audio_request.json 並寫出 audio_meta.json(以及 assets/voice|bgm|sfx 下的資產):

# <MEDIA_DIR> = 此技能目錄
node <MEDIA_DIR>/scripts/audio.mjs --request ./audio_request.json --hyperframes . --out ./audio_meta.json

所有三種能力根據一個開關降級——是否存在 HeyGen 憑證(從 $HEYGEN_API_KEY / $HYPERFRAMES_API_KEY / ~/.heygen 解析,不是 CLI):

能力HeyGen 憑證存在不存在
TTSHeyGen Starfish REST(原生文字時間戳)→ ElevenLabs → Kokoro(連結 transcribe 以取得文字)
BGMHeyGen 音樂擷取Lyria → MusicGen 本機生成(分離執行)
SFXHeyGen 音效擷取(min_score 0.4)內建 21 個檔案庫(assets/sfx/)
  • 請求(audio_request.json):{ provider?, lang?, speed?, lines: [{ id, text, sfx?: [names] }], bgm: { mode?, query?, prompt? } }。id 將每一行連結回呼叫者的模型(影格編號、場景 ID 等)。bgm.mode = retrieve | generate | none;省略則自動(有憑證時擷取,否則生成)。明確的 retrieve 是嚴格的——它會跳過而非啟動分離的生成(適用於沒有 wait-bgm 步驟的呼叫者)。
  • 輸出(audio_meta.json,以 id 為鍵):{ tts_provider, voice_id, bgm, bgm_pending, …, voices: [{ id, path, duration_s, words }], sfx: [{ id, name, file, source, offset_s, duration_s, volume }], total_duration_s }。
  • --only tts,bgm,sfx 執行子集並合併到現有的 --out(例如先 TTS+BGM,待提示存在後再 SFX)。
  • BGM 生成以分離方式啟動(bgm_pending: true)——在組裝前執行 scripts/wait-bgm.mjs。
  • scripts/heygen-tts.mjs 是基於相同程式碼的單發 CLI(一段文字 → wav + 文字),適用於只需 HeyGen TTS 而無需請求檔案的情況。

完整旗標清單 + audio_meta.json 結構位於 scripts/audio.mjs 的檔頭註解中。以下是每種能力背後供應商細節和邊際情況的參考。

飛航前檢查——在任何音訊產生前顯示登入狀態

在產生語音或 BGM 之前始終執行此操作——無論是完整工作流程還是單次「為我產生 BGM/配音」請求。 沒有 HeyGen 憑證不是默默回退到本機引擎的理由:首先建議登入並讓使用者決定。執行共享的飛航前檢查並逐字轉發其輸出——不要自創「缺少密鑰」的提示,也不要提議將密鑰寫入每個儲存庫的 .env:

npx hyperframes auth status
  • 已登入 → 它會列印帳戶資訊;繼續。
  • 未登入(exit 1 是預期的——「未登入」是正常狀態,不是失敗)→ 它會列印註冊優先的引導。建議登入:npx hyperframes auth login 是瀏覽器 OAuth——它會登入並建立帳戶(始終可透過此儲存庫的 CLI 使用)。若要使用現有的 HeyGen API 金鑰(來自 app.heygen.com/settings/api),請執行 npx hyperframes auth login --api-key——它會儲存到共享的 ~/.heygen(無每個儲存庫的 .env)。輸出也會列出語音/BGM 將回退到的本機引擎,以及依賴項缺失時的 pip 提示。按原樣轉發此輸出——不要用自己的措詞解釋。 然後停止並等待使用者選擇——登入,或說「go」/「local」以繼續離線——然後再產生任何內容。 這是一個真正的決策點,不是一個隨意的註記:不要將它摺疊到另一個問題中,也不要自行繼續執行。(例外:在自主/非互動模式中,記錄狀態並繼續離線。)
  • npx hyperframes auth status --json 會回傳 { configured, recommended_action, offline_engines } 以供確定性分支。
  • 如果 CLI 無法執行(不在 PATH 上且 npx 無法取得)→ 仍然建議登入(npx hyperframes auth login)並為了使用者的選擇而停止——不要將「沒有憑證」視為本地生成的默示綠燈。

憑證解析、完整金鑰優先級和本地依賴項清單位於 references/requirements.md。

供應商鏈(引擎背後的細節)

TTS——第一個可用的供應商獲勝(引擎,或 npx hyperframes tts "…"):

順序供應商偵測時機文字時間戳
1HeyGen(Starfish)$HEYGEN_API_KEY / hyperframes auth login是,原生——傳遞 --words narration.words.json 以捕獲
2ElevenLabs設定了 $ELEVENLABS_API_KEY否——之後連結 transcribe
3Kokoro-82M(本機,54 種語音)始終可用(無需金鑰)否——之後連結 transcribe

已發布的 hyperframes tts CLI 通常是僅限本機的建置(其 --help 顯示「Kokoro-82M」,沒有 --provider/--words),即使設定了 $HEYGEN_API_KEY,也會靜默回退到 Kokoro。這就是為什麼引擎的 HeyGen 路徑是自包含的 scripts/heygen-tts.mjs(REST),而不是 CLI;CLI 僅用於 Kokoro 路徑。請參閱 references/tts.md。

BGM 和 SFX——預設從 HeyGen 音訊庫擷取(/v3/audio/sounds),使用與 HeyGen TTS 相同的憑證,並使用上方開關的無憑證回退:

資產HeyGen type存放位置回退(無憑證)
BGMmusicassets/bgm/track.mp3(擷取)· track.wav(生成)Lyria / MusicGen 生成
SFXsound_effects(min_score 0.4)assets/sfx/<slug>.mp3內建 21 個檔案庫(assets/sfx/* + manifest.json)

請參閱 references/bgm.md 和 references/sfx.md。

路由

任務閱讀
音訊引擎——請求/中繼資料結構、--only、開關scripts/audio.mjs(檔頭註解)
npx hyperframes tts / heygen-tts.mjs——供應商、語音、文字references/tts.md
BGM——HeyGen 擷取 + 本機 Lyria / MusicGen 生成references/bgm.md
SFX——HeyGen 擷取(min_score 0.4)+ 內建本機庫references/sfx.md
npx hyperframes transcribe——Whisper、模型規則、輸出形狀references/transcribe.md
npx hyperframes remove-background——透明去背references/remove-background.md
TTS → 轉錄 → 字幕(無錄製的配音)references/tts-to-captions.md
字幕創作——樣式偵測、佈局、文字分組、結束references/captions/authoring.md
逐字稿處理——輸入格式、品質門檻、清理、APIreferences/captions/transcript-handling.md
字幕動畫——卡拉 OK、標記效果、音訊反應式references/captions/motion.md
模型快取、系統依賴項、疑難排解references/requirements.md

不可協商的規則

  • 一個引擎,無外包複製品。 透過 scripts/audio.mjs(或 heygen-tts.mjs 用於單發 HeyGen TTS)產生音訊。不要在工作流程中重新實作 TTS/BGM/SFX——編寫一個 audio_request.json 配接器並呼叫引擎。
  • 「HeyGen 可用」= 可解析的憑證,不是 CLI。 整個開關取決於 heygenCredential();已發布的 hyperframes tts 可能僅為 Kokoro,而且根本沒有 hyperframes bgm / hyperframes sfx 指令。
  • 語音 ID 是供應商特定的。 am_michael 僅限 Kokoro;HeyGen UUID 在 Kokoro 上無法使用。如果你傳遞 --voice,也要鎖定 --provider,以避免在使用者環境變更時發生靜默的供應商漂移。
  • 始終將 --model 傳遞給 transcribe。 CLI 預設值 small.en 會靜默地翻譯非英文音訊。請參閱 references/transcribe.md →「語言規則」。
  • HeyGen 會回傳文字時間戳;ElevenLabs / Kokoro 不會。 引擎會自動為後兩者連結 transcribe;單獨使用時,請將 --words 傳遞給 HeyGen 或針對音訊檔案執行 transcribe。
  • 字幕使用扁平文字陣列格式,包含 { id, text, start, end }。請參閱 references/transcribe.md →「輸出形狀」。
  • remove-background --background-output 是挖洞,不是修補。 對於「沒有人的場景」,需要不同的工具。請參閱 references/remove-background.md →「何時不是正確工具」。
  • BGM/SFX 預設為 HeyGen 擷取;無憑證回退是生成(BGM)或內建庫(SFX)。 /audio/sounds 根據文字查詢排序——具體命名效果(glass shatter,而不是 dramatic sound);無匹配則跳過,絕不阻擋渲染。SFX 在語音 + BGM 下的音量約為 ~0.35。請參閱 references/sfx.md / references/bgm.md。
  • 將工作流程字幕 HTML 視為已產生的輸出。 對於基於預設範本的影片,可重複使用的面板原始檔位於 .hyperframes/caption-skin.html,工作流程腳本會寫入 compositions/captions.html;不要編輯已產生的 compositions/captions.html 來修復面板。透過工作流程的 captions.mjs 重建,或在使用時使用該工作流程的明確覆蓋機制。

Individual skills in this repo

This repo contains 20 individual skills — each has its own dedicated page.

9018/mychat

Generate AI videos from text prompts using multiple provider gateways. Use when: (1) Generating videos from text descriptions, (2) Creating AI-generated video clips for content production, (3) Image-to-video generation with a reference image, (4) Choosing between video generation providers (VEO, Kling, Sora, Runway, Seedance, MiniMax, Gemini Omni). Supports gateways: HeyGen API, fal.ai API, Kling official direct API, and the Gemini API (Gemini Omni Flash).

9018/mychat

Create AI avatar videos with precise control over avatars, voices, scripts, scenes, and backgrounds using HeyGen's v2 API. Use when: (1) Choosing a specific avatar and voice for a video, (2) Writing exact scripts for an avatar to speak, (3) Building multi-scene videos with different backgrounds per scene, (4) Creating transparent WebM videos for compositing, (5) Using talking photos as video presenters, (6) Integrating HeyGen avatars with Remotion, (7) Batch video generation with exact specs, (8) Brand-consistent production videos with precise control.

9018/mychat

Use p5.js/canvas for local procedural character effects: particles, weather, squash/stretch, walk cycles, and environmental motion.

9018/mychat

Review local character animation with schema checks, Playwright browser previews, frame sampling, and FFmpeg/ffprobe final output checks.

9018/mychat

Create videos from a text prompt using HeyGen's Video Agent. Use when: (1) Creating a video from a description or idea, (2) Generating explainer, demo, or marketing videos from a prompt, (3) Making a video without specifying exact avatars, voices, or scenes, (4) Quick video prototyping or drafts, (5) One-shot prompt-to-video generation, (6) User says "make me a video" or "create a video about X".

9018/mychat

Generate Mandarin and multilingual narration with Volcengine Doubao Speech 2.0. Use when creating Chinese voiceovers, when the user prefers Doubao/Volcengine/火山引擎/豆包 TTS, or when narration needs character-level timestamp metadata for subtitles.

9018/mychat

Video and audio processing with FFmpeg. Use for format conversion, resizing, compression, audio extraction, and preparing assets for Remotion. Triggers include converting GIF to MP4, resizing video, extracting audio, compressing files, or any media transformation task.

9018/mychat

Use when implementing Disney's 12 animation principles with Framer Motion in React applications

9018/mychat

All animation knowledge for HyperFrames — atomic motion rules, multi-phase scene blueprints, scene transitions, broader motion-design techniques, AND the seven runtime adapters (GSAP default, plus Lottie, Three.js, Anime.js, CSS keyframes, Web Animations API, TypeGPU). Use for any motion or animation task: pick 2-4 rules and compose, or load a blueprint, or look up runtime-specific API (e.g. GSAP eases / Lottie player / Three.js mixer). HyperFrames-native: single paused timeline, seek-safe, deterministic.

9018/mychat

HyperFrames CLI dev loop. Use when running npx hyperframes init, add, catalog, capture, lint, validate, inspect, layout, snapshot, preview, play, render, publish, lambda, doctor, browser, info, upgrade, skills, compositions, docs, benchmark, telemetry, transcribe, tts, or remove-background, or when troubleshooting the HyperFrames build/render environment. Entry point for AWS Lambda cloud rendering (`hyperframes lambda deploy / render / progress / destroy / policies`).

9018/mychat

The HyperFrames composition contract — build one renderable project. Use for composition structure, the `data-*` timing attributes, `class="clip"`, tracks, sub-compositions, variables, framework-owned media playback, deterministic-render rules, and validation. Read before writing composition HTML.

9018/mychat

Non-animation creative direction for HyperFrames videos. Use for design spec (frame.md / design.md) handling, palettes, typography, narration, beat planning, audio-reactive visuals, composition patterns, and brand / style decisions. For atomic motion patterns and scene blueprints, use `hyperframes-animation`.

9018/mychat

Install and wire registry blocks and components into HyperFrames compositions. Use when running hyperframes add, installing a block or component, wiring an installed item into index.html, or working with hyperframes.json. Covers the add command, install locations, block sub-composition wiring, component snippet merging, registry discovery, and authoring a new block or component to contribute upstream (idea → scaffold → validate → PR).

9018/mychat

Use when the user wants a short, design-led motion graphic where motion is the message: kinetic typography, stat or number count-up, chart/data-viz hit, logo sting, brand lockup, lower-third, callout, social overlay, animated headline/tweet/news item, motion poster, or quick captured-page highlight. Usually under 10s and up to ~30s, with no narration arc, voice-over, or live-action subject. Can render to MP4 or transparent overlay. Not for longer, multi-scene, narrated, or brand-reel pieces (use general-video), narrated website videos (website-to-video), topic explainers (faceless-explainer), product promos (product-launch-video), PR videos (pr-to-video), or captions on existing footage (embedded-captions). When unsure whether it's a quick motion-first piece or a longer / narrated treatment, see /hyperframes.

9018/mychat

Use when the user has a music track (an audio file, or a video to pull audio from) and wants a beat-synced HyperFrames video, calm to hard-hitting. The music drives everything: one analyzer reads it once, the orchestrator lays out the frames and fills a per-frame plan, and one sub-agent builds each frame. Typography and templates are the floor — a complete video needs zero assets — but any images or videos the user supplies are cut into the frames on the same beat grid (beat-cut / ken-burns). The genre (lyric video, slideshow, kinetic promo) falls out of the per-frame choices; the pipeline never branches on it.

9018/mychat

Toolkit-specific Remotion patterns — custom transitions, shared components, and project conventions. For core Remotion framework knowledge (hooks, animations, rendering, etc.), see the `remotion-official` skill.

9018/mychat

Port an existing Remotion (React) composition to HyperFrames HTML. Use ONLY when the user explicitly asks to port/convert/migrate/translate a Remotion source. Do NOT use: (a) authoring a new HyperFrames composition; (b) Remotion mentioned in passing; (c) Remotion code shared as reference only; (d) "same video as my Remotion one" without explicit migrate request — treat as fresh build. Doubt → `/general-video`. One-way, Remotion-only: no reverse export (HyperFrames→Remotion or any framework), no non-Remotion source (After Effects, Framer Motion, plain React/CSS) → out of scope, re-create via `/general-video`. Flags unsupported patterns (useState, useEffect, async calculateMetadata, third-party React libs, `@remotion/lambda`) and recommends runtime interop over lossy translation. Unsure whether to port vs. build fresh, or only a passing Remotion mention? → /hyperframes.

9018/mychat

Animate SVG character rigs with GSAP, CSS transforms, Remotion frame control, and HyperFrames-compatible browser previews.

9018/mychat

Three.js animation - keyframe animation, skeletal animation, morph targets, animation mixing. Use when animating objects, playing GLTF animations, creating procedural motion, or blending animations.

9018/mychat

Download video and audio from YouTube and 1000+ sites using yt-dlp. No API keys needed. Use when: (1) Downloading a video from YouTube or other sites, (2) Extracting audio from a video URL, (3) Downloading subtitles/captions from a video, (4) Getting video metadata without downloading.

相關技能