Lecture to Notes
Convert a YouTube, Bilibili, or X/Twitter lecture video into a complete, compilable .tex note set and a rendered PDF.
Dependencies
Check before starting (use which). Prompt the user to install any missing tools.
| Tool | Required | Purpose |
|---|---|---|
yt-dlp | Always | Video/subtitle/metadata download (supports YouTube + Bilibili + X/Twitter). Check yt-dlp --version: a build older than ~90 days fails on current Bilibili and YouTube pages |
ffmpeg | Always | Frame extraction, audio extraction |
xelatex | Always | LaTeX compilation (TeX Live + CTeX for Chinese) |
magick | Always | Frame montage and contact sheets |
ffprobe | Always | Duration and frame height (ships with ffmpeg) |
pdftotext | Always | Rendered-page checks in verify_notes.py (poppler) |
python3 | Always | Installed helper scripts and local ASR support |
sherpa-onnx + X ASR model | Optional zh/en ASR | Fast local Chinese/English transcription with token timestamps |
| Whisper backend | ASR fallback / non-zh-en / no-CC | transcribe_whisper.py picks mlx-whisper (macOS arm64) → faster-whisper → openai-whisper |
rapidocr-onnxruntime | Bilibili / any burned-in subtitles | ocr_hardsubs.py reads the subtitle band and overlay geometry |
Pillow + numpy | Always | frame_filter.py overlay and composite-panel crops, talking-head scores |
Install the Python side with pip install rapidocr-onnxruntime Pillow numpy plus one transcription backend: pip install mlx-whisper on Apple silicon, otherwise pip install faster-whisper (or openai-whisper as the slowest fallback). /ABSOLUTE/PATH/TO/lecture-to-notes/assets/smart_crop.py remains an optional experiment.
LaTeX package check (do NOT skip — which xelatex alone is insufficient)
which xelatex passing does not mean the required LaTeX packages are installed. On minimal TeX installs (e.g. MacTeX Basic, TeX Live scheme-basic), the binary exists but ctex, tcolorbox, and other packages are missing. This causes silent failures: long Chinese lines overflow \textwidth (no CJK line breaking) or compilation aborts with File '...' not found.
Check required packages before starting:
MISSING=0
for pkg in ctex tcolorbox environ trimspaces listings hyperref booktabs float subcaption etoolbox; do
if ! kpsewhich "$pkg.sty" >/dev/null 2>&1; then
echo "❌ Missing LaTeX package: $pkg"
MISSING=1
fi
done
if [ "$MISSING" -ne 0 ]; then
echo "Install missing packages:"
echo " tlmgr install ctex tcolorbox environ trimspaces etoolbox"
echo "Or install full TeX distribution:"
echo " macOS: https://www.tug.org/mactex/ (~4 GB, includes everything)"
echo " Linux: sudo apt install texlive-full"
fi
If ctex cannot be installed (e.g. due to l3kernel version conflicts on minimal installs), the notes-template.tex includes a fallback that uses XeTeX's built-in CJK line breaking (\XeTeXlinebreaklocale "zh") with system fonts — no ctex or xeCJK needed.
Cookies and yt-dlp freshness
Every supported platform can reject an unauthenticated download, and the two failures look different:
| Symptom | Platform | Meaning |
|---|---|---|
Sign in to confirm you're not a bot | YouTube | bot detection |
HTTP Error 412 on probe or download | Bilibili | anti-scraping check |
WARNING: Your yt-dlp version (...) is older than 90 days | any | extractors predate the site's current markup |
Fix the version first, because a stale extractor produces the same errors that cookies fix: brew upgrade yt-dlp, pipx upgrade yt-dlp, or pip install -U yt-dlp, then confirm with yt-dlp --version. Homebrew disables the self-updater, so yt-dlp -U reports success without changing anything.
Then supply cookies. video_source.py probe takes them directly:
python3 "/ABSOLUTE/PATH/TO/lecture-to-notes/assets/video_source.py" probe \
--cookies-from-browser chrome "<URL>" > metadata.json
Use --cookies FILE instead when the browser profile is unavailable, for example on a headless host. A probe that fails for an authentication reason now names the flag to retry with, so treat that message as the next action rather than a dead end.
The rest of the workflow calls yt-dlp directly. Rather than editing every command, write a yt-dlp config file once and point XDG_CONFIG_HOME at it, so downloads, subtitle fetches, and thumbnail fetches all pick the cookies up:
mkdir -p "$WORKDIR/.xdg/yt-dlp"
printf -- '--cookies-from-browser chrome\n' > "$WORKDIR/.xdg/yt-dlp/config"
# then prefix every later yt-dlp call in this lecture's workdir:
XDG_CONFIG_HOME="$WORKDIR/.xdg" yt-dlp --no-playlist ... "<URL>"
Browser cookies also unlock Bilibili 1080P+ for logged-in accounts, so the same setup improves frame quality.
Goal
Produce a professional Chinese lecture note from a YouTube, Bilibili, or X/Twitter video URL. The output must:
- use the video's actual teaching content, not just subtitle transcription
- place the video's original cover image on the front page
- include selected full-frame teaching figures chosen by contact-sheet review; when the video places a camera feed and slides side by side, each figure keeps one measured panel so the slide text stays readable
- let a capable first-time reader understand the question, mechanism, evidence, consequence, and boundary without decoding the transcript
- achieve source-fit information density — every figure, box, formula, and paragraph earns its space
- be structurally organized with
\section{}/\subsection{} - end with a synthesis section combining speaker's conclusions and your own distillation
- be a complete
.texfrom\documentclassto\end{document} - compile successfully to PDF
Non-negotiable quality bar (Codex / production mode)
STOP and do not claim completion unless every item below is true. Skeleton outlines, image-heavy PDFs with thin prose, or missing intermediate artifacts are automatic failures.
Mandatory workdir artifacts (must exist on disk before saying "done")
| File | Required content |
|---|---|
metadata.json | From video_source.py probe (or equivalent full dump) |
audio.srt | Final subtitle track used for writing (manual CC, cleaned auto, hard-sub OCR, X ASR, or Whisper) |
audio_corrected.srt | Copy of final track, or glossary/LLM-corrected track when local ASR was used |
hardsub_ocr.srt | ocr_hardsubs.py extract output when ocr_hardsubs.py detect reported burned-in subtitles; absent otherwise |
bands.json | Overlay geometry (navigation strip, subtitle band) from ocr_hardsubs.py detect --geometry or frame_filter.py bands |
layout.json | Panel layout from frame_filter.py layout over the dense sample; always measured, composite: false records a single-picture video |
cover.jpg | Front-page cover |
video.mp4 | Source for frames (may omit only if user forbids download and provides frames) |
frames/ | Dense sample, default 1 frame / 15s |
frame_scores.json | frame_filter.py score over the dense sample; mandatory when the host cannot show you images |
figures/ | Selected figure assets with semantic names (fig_01_topic.jpg, …), overlays cropped via bands.json, composites cropped to one panel via layout.json |
figure_manifest.tsv | Header figure\tframe\tstart\tend\ttopic\tpanel — one row per figure image; panel names the layout.json panel it keeps (main, left, …) or full |
figure_verification.txt | Full stdout of verify_figures.py over all manifest timestamps |
lecture_profile.json | Reader and source-fit profile; required fields are described below |
teaching_atoms.tsv | Header atom\tstatus\tevidence — every teaching atom mapped to the notes |
numerical_claims.tsv | Generated by extract_claims.py extract, filled by extract_claims.py check --write; header-only is valid when the lecture contains no numerical claims |
notes.tex | Complete Chinese lecture notes |
notes.pdf | Two-pass xelatex output |
verify_notes.txt | Full stdout of verify_notes.py, ending in OVERALL PASS |
Source-fit profile and reader contract (hard)
After subtitle correction and contact-sheet review, create lecture_profile.json before outlining the notes:
{
"mode": "technical-slide",
"audience": "capable first-time reader",
"central_question": "What should the reader be able to explain after reading?",
"reader_outcome": "A concrete capability, decision, or mental model",
"visual_teaching_atoms": 24,
"formula_teaching_atoms": 18
}
Choose exactly one mode:
technical-slide: slides, board work, formulas, code, or diagrams carry most teaching content;conceptual-talk: a talk, interview, or discussion carries most content through claims, examples, and reasoning rather than visual mechanisms;mixed: both modes contribute substantial teaching content.
Count visual_teaching_atoms and formula_teaching_atoms from the source, not from the draft. A visual atom is a distinct source visual that helps teach a point. A formula atom is a source equation or derivation worth preserving. Do not inflate these counts to justify more assets, and do not classify a conceptual talk as technical merely to trigger quotas.
For every mode, the hard gates are source fidelity, complete teaching-atom coverage, traceable numerical claims, clear source attribution, and a coherent reader path. The numeric density gates below are source-fit backstops. They never authorize invented equations, low-value talking-head screenshots, repetitive boxes, or synonym padding.
Density and structure gates (source-adaptive)
For a lecture of duration $T$ minutes (from metadata):
Gate vs target (do not confuse): the numbers below are hard gates (fail = no delivery). A high-quality Codex-class note often lands ~25–40% above the CJK gate; that is a target, not a second hard floor. Raising the hard floor to the measured gold value encourages padding, not more teaching atoms. (Validated on CS336 L3: $T\approx 89$ → gate CJK 6246 / figs 25; gold ~8035 CJK / 27 figs / 31 pages.)
-
Chinese character count in the
notes.texbody (CJK unified ideographs only):technical-slide: at least $\max(5000,\ \mathrm{round}(70\times T))$;mixed: at least $\max(3500,\ \mathrm{round}(55\times T))$;conceptual-talk: at least $\max(2500,\ \mathrm{round}(45\times T))$. Do not count English jargon, LaTeX commands, or captions alone. These are completeness floors, not invitations to repeat the same idea.
-
Figures: include every distinct visual teaching atom that materially improves understanding, up to the technical target $\max(20,\ \mathrm{round}(T/3.5))$. A
conceptual-talkwith four real visual atoms should contain four verified figures, not twenty talking-head frames. A slide-led lecture with thirty distinct mechanisms must not stop at twenty. -
Sections for $T\ge 60$: use at least 8 for
technical-slide, 6 formixed, and 5 forconceptual-talk, unless fewer reader questions produce a demonstrably clearer structure. Each major section answers one reader question and ends with\subsection{本章小结}that states the answer and prepares the next question. -
Judgment boxes: use boxes only when they separate a definition, background dependency, decision rule, or failure boundary from the main flow. The technical default is 12 for $T\ge60$;
mixedandconceptual-talkhave no box quota. Never split continuous reasoning into boxes merely to raise a count. -
Teaching atom shape (adapt, do not stamp out a template):
- state what the reader needs to understand and why it matters;
- explain the mechanism or reasoning in the order needed to follow it;
- attach source evidence: a number, example, quotation-level paraphrase, formula, code, table, or verified figure as appropriate;
- state the consequence or boundary when it changes interpretation. Use only the components that the source and the reader need. After every display formula with at least two symbols, add an immediate symbol explanation list.
-
No outline-only sections. If a 5-minute span of the lecture introduces a distinct mechanism, number, or design choice, it must appear as its own subsection or a clearly labeled paragraph with evidence — not a bullet in a summary list.
-
Coverage audit before compile (timeline): walk the timeline in 3–5 minute steps using the SRT; list any gap without corresponding prose or figure; fill gaps before
xelatex. -
Teaching-atom checklist (topic coverage, hard for technical lectures): Before delivery, extract a lecture-specific atom list from SRT + slides (15–40 atoms). Each atom must map to a subsection, a labeled paragraph, or a figure+caption in
notes.tex. Missing atoms → expand prose; do not "pass" on CJK alone. Example atoms for an LLM-architecture lecture: Pre/Post-Norm, LN vs RMSNorm, FLOPs≠runtime, SwiGLU, serial vs parallel block, RoPE, FFN ratio, head dim, width/depth, vocab size, dropout vs weight decay, z-loss, QK-Norm, Prefill vs Decode, MQA/GQA, sliding window / interleaved / hybrid attention. Persist the checklist asteaching_atoms.tsv(atom\tstatus\tevidence). -
Mechanism density (hard — L2 gold comparison 2026-07-12): CJK/fig/box gates alone are not enough. CS336 L2 first Grok pass: 5463 CJK / 8 formulas / 0 code (gate PASS, gold incomplete); after claim-driven补全: ~6450 CJK / 31 formulas / 3 code / 33 pages (matches Codex). Also require:
- Display math: for source formula atoms, preserve at least $\min(N_{\mathrm{formula\ atoms}},\ \max(10,\ \mathrm{round}(T/4)))$ blocks (
\[ ... \]orequation), each with an immediate symbol explanation list. - Code blocks: for source code or pseudocode, preserve at least $\min(N_{\mathrm{code\ atoms}},\ \max(1,\ \mathrm{round}(T/15)))$
lstlistingblocks (orverbatim+color), each with a prose walkthrough. - Tables: for multi-parameter comparisons or dimension specifications, preserve at least $\min(N_{\mathrm{table\ atoms}},\ \max(0,\ \mathrm{round}(T/30)))$
tabularblocks.
- Display math: for source formula atoms, preserve at least $\min(N_{\mathrm{formula\ atoms}},\ \max(10,\ \mathrm{round}(T/4)))$ blocks (
Platform-specific subtitle workflow
| Platform | Preferred track | Fallback |
|---|---|---|
| YouTube | Manual CC (language preference: zh-Hans, zh, en) | Auto-generated CC → Whisper ASR (if auto CC is too sparse or misaligned) |
| Bilibili | Burned-in subtitle OCR via ocr_hardsubs.py | CC API (when available and authorized) → Whisper ASR |
| X/Twitter | Sometimes embedded CC | Whisper ASR only |
After obtaining the subtitle track, always run verify_subtitles.py to check timestamp alignment, misrecognized tokens (e.g. "attention" → "a tension"), and dropped lines.
Verification scripts (always run; fix all failures before delivery)
| Script | Purpose |
|---|---|
verify_metadata.py | Checks metadata.json has all required fields |
verify_subtitles.py | Checks SRT has at least 80% of expected lines, no gaps >15s |
verify_figures.py | Compares every figure timestamp against the dense frame to confirm the timestamp is within the figure's start–end window |
verify_notes.py | Compiles notes.tex, checks output page count, CJK character count, section count, figure count, box count; enforces all hard gates above; exits with OVERALL PASS only when every gate passes |
Workflow summary (implement as sequential steps)
- Probe:
video_source.py probeto get metadata, download options - Download:
video.mp4, cover image, subtitles - Subtitle preparation: OCR, ASR, or CC extraction; correction and verification
- Frame extraction: Dense sample at 1 frame / 15s
- Layout analysis:
frame_filter.py layoutto determine panel geometry - Frame scoring:
frame_filter.py scorefor keyframe selection - Contact-sheet review (human-assisted or LLM-reviewed with frame images): Select teaching figures, edit
figure_manifest.tsv, verify withverify_figures.py - Profile creation: Write
lecture_profile.jsonbased on source analysis - Atom extraction: Create
teaching_atoms.tsvfrom SRT + slides - Note drafting: Write
notes.texfollowing the goal, density gates, and atom checklist - Compilation:
xelatex notes.tex(two passes) - Verification: Run all verification scripts; fix any failures
- Delivery: Confirm
notes.pdfis readable and complete
Shell integration (recommended practice)
Set a workdir for the session and keep all intermediate artifacts there:
WORKDIR="/path/to/lecture-output/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$WORKDIR"
cd "$WORKDIR"
export XDG_CONFIG_HOME="$WORKDIR/.xdg"
Use absolute paths for asset scripts throughout to avoid $PATH resolution surprises: /ABSOLUTE/PATH/TO/lecture-to-notes/assets/video_source.py.
Final note
This skill is designed to produce the highest-quality Chinese lecture notes from online video sources. It demands rigor at every step — from dependency checking and subtitle acquisition to figure selection, prose density, and verification. Follow the workflow sequentially, respect the hard gates, and never deliver before all verification scripts pass.