open-edge-platform/vss-summarize-video

VSS Summarize

Run the summarization pipeline via the Pipeline Manager. Call the documented API yourself and report only observed responses. Endpoints use the nginx /manager prefix. When the vss_* MCP tools are connected, vss_summarize_video covers the same work; this skill is the REST route.

Set HOST=http://${HOST_IP:-localhost}:${APP_HOST_PORT:-12345}.

Answer contract when VSS is not reachable

The user may be away from the deployment, $HOST may refuse connections, or a live precondition may fail (for example, summary mode is disabled or the named video is absent). In any of those cases do not stall and do not invent responses. Report the observed blocker, then answer with the complete exact call sequence the user can run after fixing it: full endpoint paths, request bodies / form fields, the field each step carries over from the previous response, and the condition that says a step is finished. Do not abbreviate or omit the requested summary flow merely because it could not be executed. State plainly which commands were not executed. Never end the answer by asking whether to run them.

Environment setup (run first)

This skill drives the Video Search & Summarization app through its real source files, so the VSS application must be present and you must run commands from its app root. Do this before anything else, and it works whether or not the VSS source is already in your workspace.

Run the bundled bootstrap. It resolves the app root in this order and prints it as the only line on stdout:

1. Walk up from the current directory looking for a VSS app root - a directory carrying all three markers setup.sh, docker/, and pipeline-manager/.
2. Ask git for the enclosing repository (git rev-parse --show-toplevel) and check whether it holds sample-applications/video-search-and-summarization, or is itself a VSS app root. This is what makes your own clone - or a fork - work unchanged.
3. Reuse a checkout a previous bootstrap already placed in ${XDG_CACHE_HOME:-$HOME/.cache}/vss-src/edge-ai-libraries.

If any of those hit, that checkout is reused and NO clone is performed. Only when all three miss does it clone - and then only a shallow (--depth 1), single-branch, sparse checkout of just sample-applications/video-search-and-summarization from main:

# SKILL_DIR is THIS skill's own directory (shown to you when the skill loads);
# in-repo it is .github/skills/vss-summarize-video. Works the same if the skill is installed standalone.
SKILL_DIR=".github/skills/vss-summarize-video"
APP_ROOT="$(bash "$SKILL_DIR/scripts/vss-bootstrap.sh")"
cd "$APP_ROOT"

Every command below assumes the working directory is this APP_ROOT. To pull from a fork/branch or reuse a specific checkout dir, override VSS_REPO_URL, VSS_REPO_BRANCH, or VSS_CLONE_DIR before running it. The bootstrap refuses to overwrite an existing non-VSS clone destination.

Preconditions

1. Backend healthy and summary enabled - probe first; if not, use the installed vss-troubleshoot or vss-deploy skill by name:

   curl -sf "$HOST/manager/health" >/dev/null && \
   curl -s "$HOST/manager/app/features" | jq -e '(.summary // .) == "FEATURE_ON"'
   

2. A videoId to summarize - upload one with POST /manager/videos (multipart, field video), which returns { "videoId": "…" }. Or list existing - note the response is an object { "videos": [...] }, not a bare array, and name is a generated hash (the real filename is in url / dataStore.fileName):

   curl -s -X POST "$HOST/manager/videos" -F "video=@/path/to/clip.mp4" | jq .
   curl -s "$HOST/manager/videos" | jq '.videos[] | {videoId, file: .dataStore.fileName}'
   

1. Start the summary pipeline

POST /manager/summary. The body has required fields; missing any of title, sampling.*, or evam.evamPipeline returns 400. See references/summary-request.md for the full schema, prompt overrides, and audio options.

Minimal valid request:

curl -s -X POST "$HOST/manager/summary" \
  -H 'Content-Type: application/json' \
  -d '{
    "title": "Loading dock review",
    "videoId": "<VIDEO_ID>",
    "sampling": { "chunkDuration": 20, "samplingFrame": 5, "frameOverlap": 0, "multiFrame": 5 },
    "evam": { "evamPipeline": "object_detection" },
    "produceFinalSummary": true
  }' | jq .
# → { "summaryPipelineId": "<STATE_ID>" }

Sampling constraint: the Pipeline Manager enforces multiFrame == frameOverlap + samplingFrame. With frameOverlap: 0, set multiFrame == samplingFrame. Mismatch → 400 "Multi frame mismatch". evamPipeline is one of object_detection | video_ingestion.

To summarize only part of a clip, add the optional sampling.videoStart and sampling.videoEnd (seconds) - e.g. the first ten minutes is "videoStart": 0, "videoEnd": 600.

2. Poll until complete

The returned summaryPipelineId is the stateId. GET /manager/summary/{stateId} has no top-level status/progress field (only /raw does) - progress lives in per-stage fields:

STATE_ID=<STATE_ID>
curl -s "$HOST/manager/summary/$STATE_ID" | jq '{
  chunking: .chunkingStatus,         # string, "complete" when chunked
  frames:   .frameSummaryStatus,     # COUNTS object: {complete, inProgress, na, ready}
  video:    .videoSummaryStatus,     # string: "na" → "inProgress" → "complete"  ← real done signal
  audio:    .audioTranscriptSummaryStatus,
  summary_len: (.summary | length)
}'

⚠️ Completion is videoSummaryStatus == "complete", NOT summary being non-empty. The final summary text is streamed in incrementally while videoSummaryStatus is still "inProgress", so polling on "summary length > 0" returns a truncated, mid-sentence result. Always gate on videoSummaryStatus. With produceFinalSummary: false there is no final stage - gate on frameSummaryStatus.inProgress == 0 instead.

until curl -s "$HOST/manager/summary/$STATE_ID" \
       | jq -e '.videoSummaryStatus == "complete"' >/dev/null; do sleep 10; done

Summarization is slow (VLM per-chunk + LLM map-reduce) - minutes, not seconds.

3. Retrieve the summary

curl -s "$HOST/manager/summary/$STATE_ID" | jq -r '.summary'   # final map-reduced summary
# Per-chunk captions live in .frameSummaries[] (each: frameKey, status, summary).
# NOT in .chunks[] - those only carry {chunkId, duration, audioTranscripts}:
curl -s "$HOST/manager/summary/$STATE_ID" | jq -r '.frameSummaries[] | "[\(.frameKey)] \(.summary)"'
curl -s "$HOST/manager/summary/$STATE_ID/raw" | jq .   # everything (audio, frames, status, …)

Present the final summary text; offer the per-chunk detail if useful. Audio with no speech yields an audioTranscriptSummary that says so - not an error.

Final answer audit trail

Tool arguments may not be visible to the user or evaluator. The final answer must therefore report the bootstrap result: the resolved APP_ROOT, whether an existing checkout was reused without cloning, and that commands ran after changing to that app root. Also state that a total bootstrap miss falls back to a shallow (--depth 1), single-branch, sparse checkout of only the VSS app from main. Report the observed /manager/health and summary feature values.

For the summary workflow, name every public Manager operation used (method and /manager/... path), the important request fields, the observed response, and the carry-over from summaryPipelineId to STATE_ID. When a precondition blocks execution, clearly separate observed probes from unexecuted commands and still show the exact valid request, completion condition, and retrieval step without inventing ids or summary content.

Manage

curl -s "$HOST/manager/summary" | jq '.[] | {stateId, title}'   # list all
curl -s -X DELETE "$HOST/manager/summary/$STATE_ID"             # delete one