sansheng-distill -- Book/Video Distillation Engine (v2 Browse Mode)
Input the full electronic text of a book (or a set of videos), run through the Step0-Step7 pipeline (Step2 in two passes), and produce a single-file interactive HTML distillation page that opens directly via file://.
v2 page type = Browse Mode: 'Condensed Map + Detailed Body + In-Page Secondary Views', inheriting the proven five-stage funnel from the previous version:
| Tab | What it Contains |
|---|---|
| ① Book at a Glance | True cover hero + napkin formula + independent core idea modules + clickable mind map + curated quotes (for a 3-minute scan reader) |
| ② Full Book Detail | 800-1500 word detailed paraphrase per chapter, argumentative titles, fully expanded by default, no folding (for a 30-minute reader) |
| ③ Soul of the Book | Visualization of the single most counter-intuitive claim (side-by-side comparison / process chain / curve) |
| ④ Actions & Self-Check | Causal chain + mental models + decision rules + second-person self-check questions (pure browse, no scoring) |
| ⑤ How Much to Trust & Extensions | Critical four-quadrant analysis + internal tensions + book reviews + cross-book links + three in-page sub-view entries (author / similar books / viewpoint comparison) |
Interactivity: Mind map jumps to chapters, chapter detail layers fully expanded by default, book soul visualization, 3 in-page fullscreen sub-views (hash-routed open/close), multiple theme switching.
This is the orchestration file. Read this to align on the pipeline first, then for each step read the corresponding reference / run the corresponding script as indicated in the table below; references are the detailed execution rules, do not rely on memory.
Path and Variable Conventions (defined once throughout)
| Placeholder | Expands to |
|---|---|
$SKILL | The skill directory (after installation: ~/.claude/skills/sansheng-distill) |
$DATA | Book data root directory, specified by environment variable DISTILL_DATA_DIR (default ./distill-data) |
{slug} | ASCII kebab short name for the book (e.g., jinqian-xinlixue); unique site-wide, avoid collision with investment NN/parenting pNN |
{book_dir} | This book's data directory, pure {slug} (e.g., jinqian-xinlixue); no Chinese characters, git/Windows friendly |
Replace placeholders in commands with actual values before execution. The single-book directory is created automatically on first run of Step0.
Data Directory Convention (single-book artifact layout)
$DATA\
knowledge-index.json # Cross-book concept index (shared across library, maintained in Step4, automatic .bak)
{book_dir}\
book.txt # Full text (book=Step0-B; video=Step0-V assembled transcription corpus) -- gitignore
diagnose.json # Book intake diagnosis (book=Step0-B; video=Step0-V video_series variant)
raw\ # Video only: raw transcripts/srt/txt per episode (Step0-V) -- gitignore
series-input.json # Video only: manually written manifest (Step0-V input)
series.json # Video only: normalized manifest (Step0-V output, downstream reads only this)
comments.json # Video only: audience comments (Step0-V, for Step3 enrich.reviews)
distill.json # Main distill object v2 (Step2 two passes: Pass1 skeleton + Pass2 narrative/excerpts)
_pass2_g*.json # Pass2 chunk intermediate states (long books fan-out per chapter groups, merged back into distill) -- gitignore
enrich.json # Online enrichment v2 (Step3: author_page/similar_page/views_page + reviews + cross_book_external, with source URLs, can be entirely degraded)
index-merge.json # 5-tag merge manifest (Step4 intermediate artifact)
{slug}.html # Single-file interactive distill page (Step6 artifact, final deliverable, ≤3MB; true cover base64 inline, no separate cover file)
_verify.png # Step7 verification full-page screenshot -- gitignore
gitignore (recommended to add
.gitignorein the data directory):book.txt/raw//_verify.png/_pass2_*.json/*.bakshould not be committed (copyright original text / temporary artifacts); the rest (distill / enrich / HTML / index-merge / series / comments / diagnose) can be committed.
Pipeline Table (Step0-Step7, Step2 in two passes)
Each step: What to do / Which reference to read / Which command to run / Output / Failure degradation. Follow step-by-step; the output of the previous step is the input to the next.
| Step | What to do | Which reference to read | Which command to run | Output | Failure Degradation |
|---|---|---|---|---|---|
| Step0-B Book Intake Diagnosis + Convert to txt (for distilling books, take this path) | E-book → full text txt + diagnosis (format / extractability / scanned version / garbled rate / TOC recognition) | Script self-sufficient; branching rules see method.md §0 | python $SKILL\scripts\convert_book.py "<book_file>" --outdir "$DATA\{book_dir}" (add --force for re-conversion) | book.txt / diagnose.json | exit 2 = missing dependency / unsupported format / rejects overwrite → install calibre (azw3·mobi) or supplement pip dependencies or add --force; exit 3 = requires OCR / requires manual confirmation → stop and ask the user (see Hard Gate ①), do not force-read or fabricate content |
| Step0-V Video Series Intake (for distilling videos, take this path) | Each video source material transcription → manually write manifest → assemble corpus + fetch comments. Chapters = episode numbers. | method.md §V.0 (source material cascade, fixed process, read first) + §V. Branching: YouTube prioritizes subtitle capture (find human-made subtitles, video-to-subtitle-summary)/ No subtitles or visual semantics needed → Gemini native online (claude-gemini-video, no download)/ Non-YouTube first download then parse | ① Per §V.0, get clean transcription for each video (YouTube human subtitles subtitle.{lang}.vtt can be fed directly to build_series; ⚠️ Do NOT use the scrolling 3× repeated text.txt/auto-generated subtitles), save to $DATA\{book_dir}\raw\{NN}_{id}\; ② Manually write series-input.json (transcript points to the selected clean subtitle file); ③ python $SKILL\scripts\build_series.py --manifest "<series-input.json>" --outdir "$DATA\{book_dir}"; ④ python $SKILL\scripts\fetch_comments.py --series "$DATA\{book_dir}\series.json" --out "$DATA\{book_dir}\comments.json"; ⑤ Grab engagement metadata (views/likes/comments/date for heat bar) per entry via yt-dlp --skip-download --print (can't get with --flat-playlist, must do full extract per entry) | book.txt / series.json / diagnose.json / comments.json | build exit 3 = all videos missing transcript/garbled → stop and ask the user (see Hard Gate ①); fetch exit 2 = all failed or Douyin unsupported → degrade reviews entirely block (enrich.reviews set to null), do not block distillation |
| Step1 Book Type Determination | Read diagnosis + sample full text (beginning/middle/end chapters), determine type: Expository/Narrative/Biographical/Tool, set weighting fields and hard thresholds | method.md §1 | None (read diagnose.json + sample book.txt) | book_type (written into distill.json) | If boundary is fuzzy, adjudicate in order per §1.2, stop when hit; biography emphasizes thinking → Biographical (not Narrative) |
| Step2·Pass1 Skeleton Compression (Condensed Map) | Three rounds of cognitive compression + four grafted elements (decision rules/mental models/internal tensions/critical paragraphs) + v2 extended fields (life analogy/priority/quote commentary/concept misinterpretation/book soul/causal chain/self-check) + per-item anchor and evidence level | method.md §2-§5 (§3 three rounds + §4 four grafted elements + §4.5 Extended Fields + §5 anchors) | None (internalized method distillation, output JSON) | distill.json skeleton layer (all fields except chapters[].narrative/.excerpts) | diagnose=group distillation → follow method.md §8 group then synthesize (forbid one-shot swallowing); after output, self-check against §7 gates G1-G15 (see Hard Gate ②) |
| Step2·Pass2 Detailed Paraphrase (Detailed Body Text) | Two-level search per chapter (first read Pass1 skeleton to preserve structure, then grep back to the original chapter in book.txt for cases/numbers/original text to preserve flesh) → lecture-style narrative (slope opening → opinion + complete case story + data → one sentence connecting to the main line) + select excerpts. Long books: fan-out per chapter in parallel (each group of ≤5 chapters assigns 1 subagent, all Opus, serial within group, master controller merges) | method.md §3.5 (two-level search / lecture template / copyright line / fan-out) | None (internalized; long books produce intermediate _pass2_g*.json per group, master controller backfills into distill.json) | Backfill distill.json's chapters[].narrative (book 800-1500 words/chapter · video segment ≥400) + chapters[].excerpts (book ≥1 per chapter, original text ≤150 characters) | If grep fails to find support, degrade by not writing that point, forbid fabricating cases/numbers from memory; master controller does mechanical check of G9 word count / G14 excerpts + spot-check preservation of signature cases (signature stories must appear as a complete paragraph, not compressed into a tag) |
| Step3 Online Enrichment (Three In-Page Sub-Views + Inline) | Three dedicated search rounds for secondary pages: Author page / Similar books page / Viewpoint comparison page (domestic and international supporters/critics); also produce inline reviews (positive and negative) + cross_book_external (cross-book external positions), all with source URLs. Video series: reviews/views_page data sources include Step0-V comments.json; author_page changes to channel page; see enrich.md §V | enrich.md (§3 three search passes; §4 Douban anti-crawl; video see §V) | Online search (AnySearch → Tavily → WebSearch, tool chain see enrich.md §3.4); video reviews/views read comments.json | enrich.json (five top-level keys: author_page/similar_page/views_page/reviews/cross_book_external) | If a block entirely fails to fetch → set that key to null and hide the entire block (sub-view + entry deleted together); distill body does not depend on the network, enrichment failure does not block |
| Step4 Cross-Book Index Registration + Interlinking | distill.concepts matched semantically one-by-one against existing index, assigned 5-tag (SUPPORTS/REFINES/CONTRADICTS/NEW_SUB_ASPECT/NEW_CONCEPT), register this book's entry + render ⑤ M11 cross-distilled books interlinks | cross-book.md (§2 precise four steps + §3 tag determination) | ① python $SKILL\scripts\update_index.py query --index "$DATA\knowledge-index.json" --names-only → ② Write index-merge.json → ③ ... register --index "$DATA\knowledge-index.json" --merge "$DATA\{book_dir}\index-merge.json" --dry-run (exit 0) → ④ Remove --dry-run and run for real | index-merge.json / updated knowledge-index.json (+.bak) / M11 interlinking data | register exit 1 = validation error → fix index-merge.json per stderr entry by entry, re-run from step ③ (forbid --force to bypass exit 1); exit 2 = same book slug conflict → confirm it's a re-distillation before adding --force, if slug collision then use a unique slug |
| Step5 Design Two-Pass Work Method | For this book, produce token plan + signature decision, pass adversarial self-review; brand lock 80%, book soul 20% | design-craft.md (two-pass work method) + brand-tokens.md (theme token contract) | None (design decisions, internalized to guide Step6 slot filling) | Token/signature tone-setting (no independent file, directly guides Step6) | If signature hits anti-slop blacklist (blue-purple gradient/emoji icon/overuse of rounded corners + left border + card/placeholder data) → change; serif mode must be paired with CJK serif fallback |
| Step6 Generate Single-File HTML | Copy skeleton and fill slots for five tabs + 17 blocks + 3 in-page sub-views (SUB01-03), clean all dummy content, can degrade blocks (M10/M11/sub-views) by deleting the entire block | html-spec.md (§1 five tabs + block spec table / §3 generation flow / §4 mind map md / §5 volume / §6 extract compatibility) | Copy $SKILL\templates\page-skeleton.html to $DATA\{book_dir}\{slug}.html then fill slots one by one (vendor already inlined, do not touch) | $DATA\{book_dir}\{slug}.html (single file, ≤3MB) | If exceeds volume budget, per html-spec.md §5, first delete dummy, then compress excerpts, then compress cover (WebP data URI/reduce size); forbid deleting vendor/required blocks/data-source, forbid compressing narrative below G9 |
| Step7 Factory Verification v2 | Static lint + contract gates + Playwright smoke test: volume ≤3MB / required blocks present / five tab text + panel ids correct / panel-full chapters are not details and narrative meets word count (book ≥800 · video ≥400) / chapter titles are argumentative (blacklist + length) / true book cover img[src^="data:image"] / 3 sub-views consistent with entry and return / no "show source" toggle and .src-note always visible / no standalone quote wall / zero external links / lang="zh" / data-source ≥20 / zero hex outside token blocks / theme switch changes body background | html-spec.md §3 (violation fix priority) | python $SKILL\scripts\verify_page.py "$DATA\{book_dir}\{slug}.html" --distill "$DATA\{book_dir}\distill.json" --screenshot "$DATA\{book_dir}\_verify.png"; echo "Exit code=$?" | Exit code + _verify.png full-page screenshot | exit 1 = has violations, fix each per output and re-run; exit 0 only then Step7 / this book is complete (see Hard Gate ③). Never relax verify or delete check items to 'pass' |
⚠ Video path v2 not yet E2E tested: The video branches of skeleton /
method.md §V/html-spec.md §V/enrich.md §V/verify_page.pyhave been updated along with v2, but have not yet been fully re-distilled and validated with a video sample (book sample "The Psychology of Money" has passed E2E). When distilling a video series, follow §V as instructed; if encountering pitfalls where skeleton/gates don't match the video, record them first then fix. When running scripts that determine success/failure, do not use| tail/| headto grab summaries (piped exit codes take the last segment,tailalways succeeds and will swallow the failure); read the entire end line or append; echo "Exit code=$?".
StepA · Author Evolution Aggregation (Optional, enabled when ≥2 works by the same author have been distilled)
When an author has ≥2 distilled works under $DATA, optionally create a "Thought Evolution Collection" aggregation page; single-book distillation does not involve this, do not generate if <2 works.
- What to do: Only read each book's
distill.json(never re-distill), aggregate intoauthor.json→ render author evolution pageauthor.html(4 views: timeline / motif ribbon / thought shifts / concept evolution diagram). Top of each book's distill page has an "Evolution Entry Card" (SLOT:AUTHOR-ENTRY) linking to it. - Which reference to read:
author-craft.md(§0 fact vs narrative iron law / §2 author.json schema / §4 four-view data contract / §5 block skeleton / §6 shift falsification layer / §7 entry card). - Which command to run:
python $SKILL\scripts\build_author.py --author "<author_name>" --data-root "$DATA" --manual "$DATA\authors\{author_slug}\author.manual.json" --enrich "$DATA\authors\{author_slug}\author.enrich.json" --out "$DATA\authors\{author_slug}\author.json"(if author.json exists and manual is missing, guardrail blocks re-run; add--forceif truly necessary; exit 3 if <2 works, do not generate); then copytemplates\author-page-skeleton.html, replace the#author-dataslot with thisauthor.jsonto generateauthor.html. - Output:
$DATA\authors\{author_slug}\author.json+author.html. - Trigger threshold / degradation: If this author has <2 distilled works →
build_author.pyexit 3, do not generate, do not trigger enrichment search, delete the entire entry card on each book page. - Factory verification:
python $SKILL\scripts\verify_page.py "$DATA\authors\{author_slug}\author.html"; echo "Exit code=$?"(auto-detects it's an author page and runs independent gates: all 4 views present / zero external links / Zero-Hex / lang=zh / em-dashes / deep-link format / shift verdict consistent; exit 0 only then complete).
Hard Gates (three; cannot proceed without passing)
- Step0 exit 3 → stop and ask the user: When diagnosis determines "requires OCR" (scanned version, pure images) or "requires manual confirmation" (gb18030 suspicious fake characters/garbled rate >2%), do not start distillation, do not force-read, do not fabricate content; report the diagnose conclusion for the user to decide (add OCR / swap file / manually verify encoding). Video series similarly:
build_series.pyexit 3 (all videos lack transcript, none captured) → stop and ask the user (add transcript / swap video / verify manifest), do not force-distill with empty corpus. - Step2 two-pass quality gate self-check (G1-G15): After Pass1 skeleton + Pass2 detailed paraphrase output, self-check item by item per
method.md §7(general G1-G7 + book-type specific + v2 detailed/browse layer G8-G15). Hollow praise / anchor-less assertions / unmet hard thresholds / vague formulas / narrative compressed to tags / non-argumentative chapter titles (G8) / insufficient narrative detail (book<800 · video<400 characters · G9) / layman_analogy empty (G10) / soul_module non-compliant (G11) / self_check·action_chain out of bounds (G12/G13) / excerpts missing or exceeding 150-character copyright line (G14) / primary·featured out of bounds (G15) = sent back for re-distillation / patching, do not proceed to Step6 with defects. - Step7 verify v2 exit 0 only then complete: If
verify_page.py(v2, pass--distillto add contract gates) exit code is not 0, it is not a finished product. Fix data/style per output and re-run until exit 0. Never relax verification thresholds or delete check items to fake a pass.
Iron Laws (observe every step)
- Anchoring: Distillation content must be anchored to the original text;
method.md §5.1's six field types (core_ideas / decision_rules / quotes / mental_models.evidence / chapters.excerpts / self_check) each must carry ananchor, go intodata-sourceon the site; assertion without anchor = gate rejection. - Argumentative titles (v2):
chapters[].title/ M04<h3>must be refutable declarative sentences; forbid pure chapter number "Chapter N / Video N" type, forbid generic container words (chapter context / book context / quote wall / summary / overview...); effective length ≥8 characters (G8; verify mechanically blocks blacklist + length, declarative sentence semantics rely on distillation self-check). Mind map second-level nodes (v4 batch A two-layer) are exceptions:topicchanges to concept keyword ≤10 characters as scan layer, drop the ≥8 character minimum (still obey blacklist + forbid container words); refutable declarative sentence sinks totags[0](≥8 characters, verify mechanically blocks missing/too short),tags[1]holds "Chapter N" chapter number chip (see method §4.6 / html-spec §5). - Detail minimum (v2): Per chapter
narrative: book ≥800 characters / video segment ≥400 characters (G9); better to explain fewer points thoroughly than to pad; signature cases / stories must be told as a complete paragraph (time·person·action·turn·outcome), forbid compressing into a single-tag label. - True cover (book, v2): Book M01 cover must be true cover base64 (
data:image); forbid external links, forbid solid-color placeholders; only fall back to placeholder SVG if online network fails to fetch. For video series, use series cover (first episode thumbnail asdata:URI). - URLs: External information (author/reviews/similar books/viewpoint comparison/cross-book external) must include clickable http(s) source URLs; if can't fetch, degrade and hide, do not fabricate.
- No fabrication: Quotes (
<blockquote>) /excerptsmust be verbatim from the original, do not rewrite (verbatim single segment ≤150 characters, blockquote explicitly indicates citation; 30 consecutive characters identical to the original is copyright line requiring rewrite); data that cannot be obtained (ratings/years) must be left empty as-is, do not manufacture false precision. - Em-dashes are always
--: Use two English hyphens in all reader-facing text; forbid full-width——. - "Return to Original Book" statement at bottom: Fixed AI book breakdown as map, not destination disclaimer at the bottom of the page (skeleton already includes it, do not delete).
Batch Mode (distilling multiple books)
- First sample 1 book for manual verification then scale: For multi-book tasks, first fully complete 1 book (Step0-7 + browser review once), you/editor confirms quality and signature are sound before scaling in parallel for the rest.
- 6 books/batch, serial within batch: Small batches to prevent server-side 529 Overloaded; one-time high concurrency will cause large-scale failures.
- Pass2 fan-out is the biggest cost in this round: v2 distillation cost is approximately 2.5-3 times that of v1 (second-pass detailed paraphrase per-chapter grouped parallel + three-page search), all Opus, default no token throttling; when scaling in batch, account for the per-book internal Pass2 fan-out in the concurrency budget, do not start too many at once.
- Trust files on disk: Some "failed" agents may have actually written to disk but got rate-limited when returning metadata → verify actual artifacts in
$DATA\{book_dir}\, do not just trust the workflow's succeeded count. - Serial index registration: Step4's
update_index.py registerwrites to the sameknowledge-index.json; when batch processing, serialize registration (dry-run per book → real run) to avoid concurrent writes overwriting each other; automatic.bakbefore each write.
Environment Dependencies
- Python:
pip install ebooklib beautifulsoup4 pymupdf pillow pytest playwright+playwright install chromium(Step7 requires chromium;pillowused for compression and base64 inlining of true covers / thumbnails). - azw3 / mobi input requires calibre's
ebook-convert(winget install calibre.calibre); epub/pdf/txt does not. - Video series (source material cascade see
method.md §V.0):yt-dlp(YouTube subtitle + comment capture; Bilibili comments use public API with no dependencies). Bilibili/Douyin transcription requires a subtitle/ASR upstream tool (e.g.,video-to-subtitle-summary, reads itsAI_DOUYIN_API_KEY); Bilibili comment fetch via fetch_comments requires no key. No subtitles / need visual semantics → use a Gemini video analysis tool: sibling skill in this reposansheng-gemini-video(reads envGOOGLE_API_KEY); install it and it's ready; not installing does not affect book distillation or subtitled video.