影片理解技能
此技能使用 z-ai-web-dev-sdk 套件提供專業的影片理解功能,讓 AI 模型能夠分析、描述並從影片內容中提取資訊,包括動作、時間序列及場景變化。
技能路徑
技能位置:{project_path}/skills/video-understand
此技能位於你專案中的上述路徑。
參考腳本:在 {Skill Location}/scripts/ 目錄中提供了範例測試腳本,供快速測試與參考。請參閱 {Skill Location}/scripts/video-understand.ts 取得可運作的範例。
概述
影片理解專注於影片內容分析,提供以下功能:
- 影片場景理解與描述
- 動作與行為偵測
- 時間序列分析
- 影片中的事件偵測
- 影片內容摘要
- 場景變化偵測
- 跨影格的人物與物體追蹤
- 視聽內容分析(適用時)
重要:z-ai-web-dev-sdk 必須僅在後端程式碼中使用。切勿在客戶端程式碼中使用。
先決條件
z-ai-web-dev-sdk 套件已安裝。請依下方範例所示方式匯入。
命令列介面使用方式(適用於簡單任務)
對於快速的影片分析任務,您可以使用 z-ai 命令列介面替代編寫程式碼。這適用於簡單的影片描述、測試或自動化。
基本影片分析
# 從 URL 分析影片
z-ai vision --prompt "Summarize what happens in this video" --image "https://example.com/video.mp4"
# 注意:影片 URL 也請使用 --image 旗標
z-ai vision -p "Describe the key events" -i "https://example.com/presentation.mp4"
分析本機影片
# 分析本機影片檔案
z-ai vision -p "What activities are shown in this video?" -i "./recording.mp4"
# 將回應儲存至檔案
z-ai vision -p "Provide a detailed summary" -i "./meeting.mp4" -o summary.json
進階影片分析
# 包含思考過程的複雜場景理解
z-ai vision \
-p "Analyze this video and identify: 1) Main events, 2) People and their actions, 3) Timeline of key moments" \
-i "./event.mp4" \
--thinking \
-o analysis.json
# 動作偵測
z-ai vision \
-p "Identify all actions performed by people in this video" \
-i "./sports.mp4" \
--thinking
串流輸出
# 串流影片分析結果
z-ai vision -p "Describe this video content" -i "./video.mp4" --stream
命令列參數
--prompt, -p <text>:必要 - 關於影片的問題或指令--image, -i <URL or path>:選用 - 影片 URL 或本機檔案路徑(雖然名稱是 image,但也適用於影片)--thinking, -t:選用 - 啟用鏈式思考推理以進行複雜分析(預設:停用)--output, -o <path>:選用 - 輸出檔案路徑(JSON 格式)--stream:選用 - 即時串流回應
支援的影片格式
- MP4 (.mp4) - 最廣泛支援的格式
- AVI (.avi) - Audio Video Interleave
- MOV (.mov) - QuickTime 格式
- WebM (.webm) - 網頁最佳化格式
- MKV (.mkv) - Matroska 格式
- FLV (.flv) - Flash Video 格式
何時使用命令列介面 vs 軟體開發套件
使用命令列介面時機:
- 快速影片摘要
- 一次性影片分析
- 測試影片理解能力
- 簡單自動化腳本
- 產生影片描述
使用軟體開發套件時機:
- 多輪對話討論影片
- 複雜的影片處理流程
- 具錯誤處理的正式環境應用
- 自訂整合影片處理邏輯
- 批次影片處理與自訂工作流程
建議做法
為獲得更好的本機影片效能與可靠性,請考慮:
- 將影片上傳至 CDN 並使用 URL
- 對於較短的影片,將關鍵影格轉換為圖片以加速分析
- 對於較長的影片,考慮分段或以間隔取樣
基本影片理解實作
單一影片分析
import ZAI from 'z-ai-web-dev-sdk';
async function analyzeVideo(videoUrl, prompt) {
const zai = await ZAI.create();
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: prompt
},
{
type: 'video_url',
video_url: {
url: videoUrl
}
}
]
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
}
// 使用範例
const summary = await analyzeVideo(
'https://example.com/presentation.mp4',
'Summarize the key points presented in this video'
);
const actionDetection = await analyzeVideo(
'https://example.com/sports.mp4',
'Identify and describe all athletic actions performed in this video'
);
影片場景理解
import ZAI from 'z-ai-web-dev-sdk';
async function understandVideoScenes(videoUrl) {
const zai = await ZAI.create();
const prompt = `Analyze this video and provide:
1. Overall summary of the video content
2. Main scenes or segments (with approximate timestamps if possible)
3. Key people or characters and their roles
4. Important actions or events in chronological order
5. Setting and environment description
6. Overall mood or tone`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'enabled' } // 啟用以進行詳細分析
});
return response.choices[0]?.message?.content;
}
// 使用
const sceneAnalysis = await understandVideoScenes(
'https://example.com/documentary.mp4'
);
動作與行為偵測
import ZAI from 'z-ai-web-dev-sdk';
async function detectActions(videoUrl, specificAction = null) {
const zai = await ZAI.create();
const prompt = specificAction
? `Identify all instances of "${specificAction}" in this video. For each instance, describe when it occurs and provide details about how it's performed.`
: 'Identify and describe all significant actions and movements in this video. Include who is performing them and when they occur.';
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'enabled' }
});
return response.choices[0]?.message?.content;
}
// 使用
const runningActions = await detectActions(
'https://example.com/sports.mp4',
'running'
);
const allActions = await detectActions(
'https://example.com/activity.mp4'
);
事件時間線提取
import ZAI from 'z-ai-web-dev-sdk';
async function extractTimeline(videoUrl) {
const zai = await ZAI.create();
const prompt = `Create a detailed timeline of events in this video:
- Identify key moments and transitions
- Note approximate timing (beginning, middle, end or specific timestamps if visible)
- Describe what happens at each key point
- Identify any cause-and-effect relationships between events
Format as a chronological list.`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'enabled' }
});
return response.choices[0]?.message?.content;
}
影片內容分類
import ZAI from 'z-ai-web-dev-sdk';
async function classifyVideo(videoUrl) {
const zai = await ZAI.create();
const prompt = `Classify this video content:
1. Primary category (e.g., educational, entertainment, sports, news, tutorial)
2. Sub-category or genre
3. Target audience
4. Content style (professional, casual, documentary, etc.)
5. Key themes or topics
6. Suggested tags (10-15 keywords)
Format your response as structured JSON.`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'disabled' }
});
const content = response.choices[0]?.message?.content;
try {
return JSON.parse(content);
} catch (e) {
return { rawResponse: content };
}
}
進階使用案例
多輪影片對話
import ZAI from 'z-ai-web-dev-sdk';
class VideoConversation {
constructor() {
this.messages = [];
}
async initialize() {
this.zai = await ZAI.create();
}
async loadVideo(videoUrl, initialQuestion) {
this.messages.push({
role: 'user',
content: [
{ type: 'text', text: initialQuestion },
{ type: 'video_url', video_url: { url: videoUrl } }
]
});
return this.getResponse();
}
async askFollowUp(question) {
this.messages.push({
role: 'user',
content: [
{ type: 'text', text: question }
]
});
return this.getResponse();
}
async getResponse() {
const response = await this.zai.chat.completions.createVision({
messages: this.messages,
thinking: { type: 'disabled' }
});
const assistantMessage = response.choices[0]?.message?.content;
this.messages.push({
role: 'assistant',
content: assistantMessage
});
return assistantMessage;
}
}
// 使用
const conversation = new VideoConversation();
await conversation.initialize();
const initial = await conversation.loadVideo(
'https://example.com/lecture.mp4',
'What is the main topic of this lecture?'
);
const followup1 = await conversation.askFollowUp(
'Can you explain the key concepts mentioned?'
);
const followup2 = await conversation.askFollowUp(
'What examples were used to illustrate these concepts?'
);
影片品質評估
import ZAI from 'z-ai-web-dev-sdk';
async function assessVideoQuality(videoUrl) {
const zai = await ZAI.create();
const prompt = `Assess the quality of this video:
1. Visual quality (resolution, clarity, lighting) - Rate 1-10
2. Audio quality (if audio is present) - Rate 1-10
3. Camera work (stability, framing, composition) - Rate 1-10
4. Production value (editing, transitions, effects) - Rate 1-10
5. Content clarity (is the message clear?) - Rate 1-10
6. Pacing (too fast, too slow, just right)
7. Technical issues (artifacts, blur, audio sync, etc.)
8. Overall rating - 1-10
9. Specific recommendations for improvement
Provide detailed feedback for each criterion.`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'enabled' }
});
return response.choices[0]?.message?.content;
}
影片內容審核
import ZAI from 'z-ai-web-dev-sdk';
async function moderateVideo(videoUrl) {
const zai = await ZAI.create();
const prompt = `Review this video for content moderation:
1. Check for any inappropriate or sensitive content
2. Identify any potential safety concerns
3. Note any content that might violate common community guidelines
4. Assess age-appropriateness
5. Identify any copyrighted material visible (logos, brands, music)
6. Overall safety rating: Safe / Caution / Review Required
Provide specific examples for any concerns identified.`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'enabled' }
});
return response.choices[0]?.message?.content;
}
影片逐字稿生成(視覺描述)
import ZAI from 'z-ai-web-dev-sdk';
async function generateVisualTranscript(videoUrl) {
const zai = await ZAI.create();
const prompt = `Generate a detailed visual transcript of this video:
- Describe what's happening in each scene
- Note any text that appears on screen
- Describe important visual elements
- Mention any scene changes or transitions
- Include descriptions of people's actions and expressions
Format as a time-based narrative (e.g., "At the beginning...", "Then...", "Finally...").`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
}
運動影片分析
import ZAI from 'z-ai-web-dev-sdk';
async function analyzeSportsVideo(videoUrl, sport = null) {
const zai = await ZAI.create();
const prompt = sport
? `Analyze this ${sport} video in detail:
1. Identify players and their positions
2. Describe key plays and strategies
3. Note scoring events or important moments
4. Assess player performance
5. Identify any rule violations or fouls
6. Describe the pace and flow of the game`
: `Analyze this sports video:
1. Identify the sport being played
2. Describe the key actions and plays
3. Note any scoring or significant events
4. Describe player movements and strategies
5. Overall assessment of the game or match`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'enabled' }
});
return response.choices[0]?.message?.content;
}
教學影片分析
import ZAI from 'z-ai-web-dev-sdk';
async function analyzeEducationalVideo(videoUrl) {
const zai = await ZAI.create();
const prompt = `Analyze this educational video:
1. Subject matter and difficulty level
2. Key learning objectives
3. Teaching methods used (lecture, demonstration, animation, etc.)
4. Clarity of explanations
5. Visual aids and their effectiveness
6. Logical flow of information
7. Suggested prerequisites for viewers
8. Target audience appropriateness
9. Overall educational value (1-10)
10. Suggestions for improvement`;
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'enabled' }
});
return response.choices[0]?.message?.content;
}
影片處理錯誤處理
import ZAI from 'z-ai-web-dev-sdk';
async function analyzeVideoWithFallback(videoUrl, prompt) {
try {
const zai = await ZAI.create();
const response = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: prompt },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
],
thinking: { type: 'disabled' }
});
return response.choices[0]?.message?.content;
} catch (error) {
console.error('Video analysis failed:', error.message);
// 嘗試使用簡化的提示詞重試
try {
console.log('Retrying with simplified prompt...');
const zai = await ZAI.create();
const simplifiedResponse = await zai.chat.completions.createVision({
messages: [
{
role: 'user',
content: [
{ type: 'text', text: 'Describe the video key points' },
{ type: 'video_url', video_url: { url: videoUrl } }
]
}
]
});
return simplifiedResponse.choices[0]?.message?.content;
} catch (retryError) {
return `Analysis failed: ${error.message}. Retry also failed: ${retryError.message}`;
}
}
}
使用影片分析的最佳實踐
- 影片長度:較短的影片(5 分鐘以下)可獲得最佳效果。較長的影片應考慮分段。
- 影片品質:解析度高、光線充足、穩定的影片可產生更準確的分析。
- 明確的提示詞:使用具體、結構化的提示詞以獲得詳細的分析。
- 時間參考:在提示詞中要求時間參考,以取得結構化的回應。
- 結合鍵盤:對於複雜分析,啟用
thinking: { type: 'enabled' }。 - 本地影片:對於本地影片,上傳至 CDN 或將其轉換為較小區塊。
- 錯誤處理:始終實作錯誤處理與重試邏輯,尤其是在正式環境中。
- 效能:在提示詞中要求較短的摘要,以縮短回應時間。