What does Video2Knowledge do?
Turn a video into: timestamped subtitles → knowledge doc → HTML / Anki cards / 图文笔记.
Two local ingestion paths, all-local inference by default (no cloud API).
Processing stays fully local — videos, subtitles, and outputs are never
uploaded; they live in a local runs/ folder (gitignored). The repo tracks only
code/config changes.
The one exception is opt-in and never automatic:
--backend openai-api/mimo-asr(ASR) and--api-base(knowledge doc, any OpenAI-compatible endpoint) send that audio or subtitle text to the third party you named. Not passing them keeps everything on the host. Say so explicitly when you recommend them.
Prerequisites & First-Time Setup
Required on the host: ollama, ffmpeg, python3 (or uv). One-time:
bash scripts/setup_models.sh
This (idempotently) auto-detects your machine's hardware profile (RAM / GPU /
Apple Silicon / NVIDIA) via scripts/hardware_profile.py, then pulls the
recommended VLM, creates a venv at .venv/, and installs faster-whisper +
genanki. See what it picked:
python3 scripts/hardware_profile.py
Profiles range from tiny (4 GB machines → whisper-tiny + qwen3.5:0.8b) through
high-gpu (NVIDIA ≥8 GB → whisper-large-v3 + qwen3.5:9b on CUDA). Full table
and tuning in references/hardware-profiles.md. Override any choice with env
vars (VLM_MODEL=, ASR_DEFAULT_MODEL=, ...) or CLI flags.
Activate the venv before running any python step:
source .venv/bin/activate
Choose an Ingestion Path (Step 1)
Decision tree:
- Clear speech track (lecture, talk, interview, narration) → Path 2 (ASR).
Faster, more accurate text, fine-grained timestamps. For Mandarin, prefer the
FunASR Paraformer backend — ask the machine with
python3 scripts/hardware_profile.py --recommend. →references/path2-asr.md - Silent / slide-only / screen recording, or you need on-screen text & diagrams
→ Path 1 (multimodal). →
references/path1-multimodal.md - Slide/PPT video with a speech track (the common online-course case): the core
content is on screen (tables, formulas, examples) AND the speaker narrates →
Path 3 (dual-path fusion). Runs ASR for narration + VLM OCR for slides and
fuses them by timestamp. →
references/path3-fusion.md
Paths 1 & 2 emit the same segment schema ({start,end,text}), so Step 2 is
path-agnostic. Path 3 emits a fused merged.json consumed by Step 2's --merged.
Path 1 — Multimodal captions
python3 scripts/mm_caption.py \
--video VIDEO --out-dir OUT \
--model openbmb/minicpm-v4.6:latest --interval 2.0
Outputs: OUT/captions.srt, OUT/captions.json, OUT/frames/.
For slide/PPT videos, use dedup sampling (general-purpose, no threshold tuning) plus the table/formula OCR prompt:
python3 scripts/mm_caption.py \
--video VIDEO --out-dir OUT \
--mode dedup --prompt-ocr
Path 2 — ASR transcription
Which engine? Ask the machine first:
python3 scripts/hardware_profile.py --recommend
Measured on a 61.5 h Mandarin course, both engines on the same 30-minute lecture with the same 39-term hotword list:
| homophone terms correct | speed | |
|---|---|---|
faster-whisper small + hotwords | 44% | 2.5× realtime |
| Paraformer-zh (FunASR, CPU) | 94% | 12–16× realtime |
Hotwords cannot fix this: 背谱/被谱 are identical in pinyin and tone, so the audio itself is ambiguous. The recommendation keys on RAM, not VRAM — Paraformer's advantage is Mandarin accuracy and it runs on CPU. If you have a GPU, leave it free for the VLM: running whisper and the VLM together turns an 8-token request from 3.8 s into over 60 s.
Default (local, faster-whisper):
python3 scripts/asr_caption.py \
--video VIDEO --out-dir OUT \
--model small --language zh \
--hotwords "术语一, 术语二" # optional jargon biasing via initial_prompt
Paraformer (recommended for Mandarin; needs its own venv — funasr's tokenizers pin does not co-install with the main environment):
bash scripts/setup_models.sh --with-funasr # builds .venv-funasr, prints the path
python3 .venv-funasr/bin/python scripts/asr_funasr.py \
--video VIDEO --out-dir OUT --language zh --hotwords "背谱,视谱,音阶"
Cloud engines exist for hosts that cannot transcribe locally, and are opt-in —
they upload the audio to a third party. bash scripts/setup_models.sh --with-openai-client adds the openai SDK it needs; --backend openai-api
takes any OpenAI-compatible endpoint and --backend mimo-asr is for
chat-shaped gateways that return no timestamps. See references/path2-asr.md.
Outputs: OUT/subtitles.{srt,vtt,json}.
Then repair homophones. ASR hears sounds, not words, so domain terms come out spelled wrong and the error propagates into every downstream artifact. Measured: 20 classes of genuine error in 46 places, 100% precision.
python3 scripts/fix_homophones.py --subtitles OUT/subtitles.json
Matches on tone-aware pinyin, not edit distance, and rewrites subtitles.srt /
.vtt to match. Run it before Step 2. --dry-run to preview, --block 实度
to blacklist a fragment, --glossary @terms.txt to add your own.
Path 3 — Dual-path fusion (ASR × VLM)
Run Path 2 then Path 1 (OCR), feed OCR terms back into ASR hotwords, fuse, and build from the merged file:
# 1a. ASR
python3 scripts/asr_caption.py --video VIDEO --out-dir OUT --language zh
# 1b/1c. dedup frames + VLM OCR
python3 scripts/mm_caption.py --video VIDEO --out-dir OUT --mode dedup --prompt-ocr
# 1d. OCR terms -> hotwords + coverage check (re-run ASR with
# --hotwords @OUT/ocr_hotwords.txt when coverage is poor)
python3 scripts/hotwords_from_ocr.py --captions OUT/captions.json \
--subtitles OUT/subtitles.json --out OUT/ocr_hotwords.txt
# 2. fuse by timestamp + semantic alignment check
python3 scripts/merge_visual.py --subtitles OUT/subtitles.json --visual OUT/captions.json --out OUT/merged.json
# 3. build (Step 2 with --merged)
python3 scripts/build_knowledge.py --subtitles OUT/subtitles.json --merged OUT/merged.json --out-dir OUT --format all
The fusion loop has three cross-path pieces: (1) hotwords_from_ocr.py
extracts on-screen terms (recurring CJK n-grams, slide titles, Latin
acronyms — heuristic, no LLM) and reports which ones the ASR never heard,
the likely mis-heard jargon; (2) merge_visual.py attaches visuals by
timestamp, then conservatively re-binds a narration line to an ADJACENT slide
when its word overlap with the timestamp-attached slide is near zero and the
neighbour matches clearly better (speaker lag / timestamp drift), flagging
uncertain cases as weak instead of guessing — except on SETTLED slides
(≥60 s on screen), where verbal elaboration is the norm and no flag is raised
(--no-semantic restores pure timestamp matching); (3) build_knowledge.py --merged feeds the LLM interleaved audio+visual text (chunk boundaries never
cut a slide table; prompt-echo lines are fingerprint-filtered from every list
field; field caps scale with video duration, 3× at 3 h) with ⚠️ markers on
swap/weak notes, and re-ranks map-reduced list fields in one global pass with
the summary as context. Chunk-level LLM responses are cached in
build_cache.json, so an interrupted 3-hour build resumes instead of
restarting. In batches, batch_run.py accumulates OCR terms into
course_hotwords.txt so later videos transcribe better, and --asr-verify
re-transcribes a video whose OCR-term coverage fell below 50 %.
The default text model follows the same hardware profile: openbmb/minicpm5-2b
(2.5B dense, 1.6 GB) on low/low-mac/mid, and the Path 1 VLM pull itself on
high+ (qwen3.5/qwen3.8 are unified vision+text, so nothing is downloaded
twice). Override with --model, or V2K_TEXT_MODEL= to pin one for every run.
Quality measurement: scripts/recall_check.py --run-dir <dir> --draft emits a
golden must-have list (terms / both-channel numbers / timeline coverage) that a
human prunes in minutes; --golden then scores produced artifacts against it
(baselines in tests/golden/BASELINE.md) — "missing content" becomes a number
you can track across parameter changes.
Refine into Knowledge Artifacts (Step 2)
Point build_knowledge.py at either path's .json output:
python3 scripts/build_knowledge.py \
--subtitles OUT/subtitles.json \
--out-dir OUT --format all
Produces, in OUT/:
| Artifact | File | Section |
|---|---|---|
| 2.1 Knowledge doc (templated) | knowledge.md | uses assets/default-template.md or --template <file> |
| 2.2 Self-contained HTML | knowledge.html | clickable [mm:ss] timeline |
| 2.2b Office/print export | knowledge.docx / knowledge.pdf | --format docx / --format pdf (auto in --format all when python-docx/fpdf2 installed) |
| 2.3 Knowledge cards | cards.csv | question,answer,tags,timestamp,source — deduplicated: a long lecture is summarised map/reduce over chunks and the model re-asks the same question in each, so every repeat is dropped (measured: 145 of 1608 cards, worst case 117 rows → 11) |
| 2.4 Illustrated notes (图文笔记) | notes.md + notes.html | key frames × narration, see below |
| 2.4b Notes office/print | notes.docx / notes.pdf | build_notes.py --docx --pdf (frames embedded) |
| 2.4c Distilled notes (纯享版) | notes-distilled.md / .docx / .pdf | same nodes and frames, no quoted narration — 42% smaller, and free (rendered from output already computed) |
Then convert cards to Anki (2.3 final):
python3 scripts/gen_apkg.py --csv OUT/cards.csv --out OUT/cards.apkg --deck "视频知识卡"
Illustrated notes (2.4, 图文笔记)
A scrollable note that interleaves deduped KEY FRAMES with the narration around
each timestamp — the "watch it back as a reading" artifact. Needs frames.json
(from extract_frames.py, usually via Path 1's dedup mode) plus either path's
subtitles:
python3 scripts/build_notes.py \
--subtitles OUT/subtitles.json \
--frames OUT/frames/frames.json \
--out-dir OUT --describe-frames
Node density is automatic. --max-frames 0 (the default) sizes the note to
~1 node per 45 seconds, clamped to 12–60: a 12-minute clip gets 16 nodes, a
41-minute lecture 55, a 3-hour one 60. The old 1-per-4min budget gave a
37-minute lecture only 9 nodes, and the 1-per-90s that followed still floored
at 8 — measured over a 306-video / 61.5h library the mean clip is 12.1 min, so
the floor bound most of the library to 8 nodes no matter how long the video
ran. The floor was what had to move, not the ceiling. Pass an explicit
--max-frames N to pin it. Selection is cluster-stratified: every change burst
keeps its settled final frame, so an animation burst no longer starves isolated
key slides of their own node.
Per key frame: LLM section title → frame image → optional VLM 画面 description
(--describe-frames) → 1-2 sentence note condensed from the narration window →
verbatim 原声 excerpt.
The 画面 prompt asks for teaching information, not people: note pitches, chord symbols, time signatures, annotations and on-screen text — and explicitly refuse to describe the performer ("女士弹琴", "手势讲解" are not what a revision note needs). A frame with no readable teaching content gets its topic named instead ("钢琴演奏画面,无字幕"). This one prompt change is the difference between a note that helps you study and one that narrates the video.
Two views, one pass. Every run writes a twin pair from the same sections — no extra model call:
| File | Contains | For |
|---|---|---|
notes.md | frames + notes + quoted 原声 | cross-checking the teacher's exact wording |
notes-distilled.md | same nodes, same frames, no quoted narration | revising — measured over 295 nodes, the blockquotes were 53% of the characters |
--docx / --pdf export both variants (notes.docx + notes-distilled.docx,
and likewise for PDF). notes.html embeds frames as base64 into a single
shareable file — that one is the full view only, no distilled HTML twin.
Degrades to frames + raw excerpts when no model is reachable.
Give the VLM a vision-capable model explicitly. --describe-frames without
--vlm-model falls back to the text model, which cannot see; the run warns and
drops the descriptions instead of failing once per frame:
python3 scripts/build_notes.py ... --describe-frames \
--model openbmb/minicpm5-2b --vlm-model openbmb/minicpm-v4.6:latest
See references/outputs.md for schema, single-format runs, and degraded mode.
Custom Templates (2.1)
Default template: assets/default-template.md. Override with --template <file>:
python3 scripts/build_knowledge.py --subtitles s.json --out-dir o \
--template ./my-lecture-template.md --format knowledge
Templates are plain Markdown using {{placeholders}} ({{title}}, {{summary}},
{{bullets}} (音画合并要点速览), {{timeline}}, {{key_points}}, {{qa}}, {{glossary}},
{{source}}, {{duration}}, {{date}}, {{meta}}). Only placeholders you include are filled;
everything else stays verbatim. Full spec + 3 example templates
(lecture / meeting / tutorial) in references/templates.md.
Local-Only Processing & Change Tracking
All video processing stays on your machine. Videos, extracted frames,
subtitles, knowledge docs, and cards are written to a local runs/ folder that is
gitignored — nothing about your media is ever uploaded or committed.
The GitHub repo tracks only code and config changes (scripts, references,
templates, README). This gives a clear history of how the skill evolved, without
exposing any user's media. Use descriptive feat:/fix:/docs: commit messages.
If you want a local record of a specific run, write a manifest.json into that
run folder (path taken, models, args, output list) — but keep it local:
cat > "$RUN/manifest.json" <<EOF
{"video":"~/Movies/lecture.mp4","pa
…