RuijieH/citation-digest

Citation Digest (citation-digest)

Input a paper, output "who cited it + what each citation solves + grouped by category," rendered as markdown or a Notion page.

The pipeline has four steps, already encapsulated in scripts. Do not reinvent:

s2_citations.py  →  citations.json   Who cited it (with citation count/date/DOI/arXiv)
kimi_q1.py       →  kimi.json        Original Kimi Q1 for each paper (structured blocks)
(you judge)      →  classify.json    Classification results
build_content.py  →  content.json + digest.md
notion_publish.py →  Notion page

0. Prerequisites

• S2 key: Prefers $SEMANTIC_SCHOLAR_API_KEY, then --key, then ~/.env/semantic-scholar. It works without a key, but you will see many 429 errors; the script auto-retries (8 times × 12s).
• Notion token: ~/.notion-token or $NOTION_TOKEN; the target page must be shared with this integration first, otherwise search will not find it.

1. Fetch Citations

python3 scripts/s2_citations.py --title "MP-GUI: Modality Perception with MLLMs for GUI Understanding" --out citations.json
# Alternatively: --arxiv 2503.14021 / --paper-id 0e931ba8...

The script has three rules you will certainly encounter:

1. Deduplication: S2 often returns the same arXiv paper twice (e.g., "A Survey on ..." and "A Summary on ...", one with an ArXiv ID, the other with only a DOI 10.48550/arXiv.xxxx). Deduplicate by arXiv ID → DOI → title, keeping the entry with the higher citation count.
2. More than 20 papers: Keep only two top-10 lists: --max-total 20 --top 10, keep top-10-by-citation-count ∪ top-10-by-date. This is enabled by default; do not implement your own filtering.
3. Title matching: The search endpoint selects the result with the highest token overlap. If the score is <0.55, candidates are listed on stderr for user confirmation.

First, check source.citationCount in citations.json – this is the citation count to report.

2. Fetch Kimi Summaries

python3 scripts/kimi_q1.py --in citations.json --out kimi.json --sleep 10 --resume

• Goes to POST https://papers.cool/arxiv/kimi?paper=<arxiv_id> (the underlying implementation of the local papers-cool MCP), requires User-Agent / Origin / Referer headers, otherwise it is blocked.
• Returns HTML FAQ, only extracts Q1 "What problem does this paper attempt to solve?", converts it into structured blocks: ### → heading_3, - → bulleted_list_item, **bold** → bold. The original text is copied verbatim, not compressed or rewritten.
• papers.cool covers more than just arXiv. It also has a venue section: /venue/<CODE>.<YEAR> where CODE ∈ AAAI/ACL/COLM/COLT/CoRL/CVPR/ECCV/EMNLP/ICCV/ICLR/ICML/IJCAI/INTERSPEECH/IWSLT/MICCAI/MLSYS/NAACL/NDSS/NeurIPS/OSDI/UAI. For papers without an arXiv ID, the script automatically falls back to: fetch the list page /venue/<CODE>.<YEAR>?show=5000, match the title via id="title-<pid>" to get a paper ID (e.g., 49@2026@IJCAI; the IJCAI sequence number is the DOI suffix), then POST /venue/kimi?paper=<pid>, returning the same FAQ HTML. The venue list page defaults to 25 entries; you must include ?show=.
• Papers that cannot be fetched via either path (e.g., IEEE MIPR/AIoTC, not indexed by papers.cool) go into missing. For these, fall back to abstract or title, and you must annotate them in the page as "Non-Kimi summary" – build_content.py/notion_publish.py already add a ⚠️ marker for source != "kimi". Add the reason in classify.json under notes.
• --sleep 10 is more stable than 2. For papers without a cached summary, Kimi computes on the fly, potentially taking 60-90s per paper. The script writes to --out after each paper is fetched; use --resume to resume after interruption.
• ⚠️ Do not use pkill -f kimi_q1.py: Multiple Claude sessions running this skill at the same time may kill each other (26 papers that were fetched but not yet written to disk were lost in practice). To run in the background, first run cp scripts/kimi_q1.py <scratchpad>/pcfetch_<session-id>.py to have an independent process name.

3. Classification (You Do It, Not the Script)

Write classify.json:

{
  "categories": [
    {"name": "🧩 Scenario Innovation / Problem Innovation", "desc": "New task, new benchmark, new evaluation dimension, or redefines a problem",
     "papers": ["2503.14021", "10.1109/CVPR52734.2025.02766", "Exact Title"]},
    {"name": "🔧 Technical Improvement", "desc": "Known problem, contribution in method/training pipeline/system/architecture",
     "papers": ["..."]},
    {"name": "📚 Survey Only", "desc": "No new method or task, value lies in context and open problems",
     "papers": ["..."]}
  ],
  "notes": {"2504.20464": "This paper had two duplicate records in S2; merged."}
}

papers can use arXiv IDs, DOIs, or titles (exact or partial match). Papers not covered automatically fall into "❓ Unclassified" – they are not silently dropped.

Criteria (use these three unless the user specifies otherwise):

• Scenario Innovation / Problem Innovation: Introduces a new task, new benchmark, new evaluation dimension (e.g., "looking only at the final state is insufficient; we must observe the process"), or reframes a phenomenon as a problem (e.g., "using a broken tool without switching").
• Technical Improvement: The problem is well-known; the contribution is in method, training pipeline, system implementation, or architecture.
• Survey Only: Survey / Review / position paper that does not propose a novel, testable method.
• Edge cases (does a benchmark count as problem innovation or a separate category; where to place analytical papers) – classify according to the above first, then include one sentence explaining your reasoning in the response. Do not classify silently.

Body tags: For ordinary papers, use "Problem to solve"; for surveys, use "Core insight" – but since the Q1 text is copied verbatim, do not add tags in general; let the original text speak for itself.

4. Compile + Write to Disk

python3 scripts/build_content.py --citations citations.json --kimi kimi.json \
    --classify classify.json --out content.json --md digest.md

5. Publish to Notion

python3 scripts/notion_publish.py --content content.json --page-title "topic paper reading"
# Or --page-id xxx; add --replace to overwrite existing content

The structure is fixed: A root toggle named after the paper → summary paragraph + three category toggles → one toggle per paper (first line: gray meta-info: Venue · Date · Cited N · arXiv:xxx, clickable; below: Q1 source blocks).

Pitfalls (All Encountered in Practice)

Symptom	Cause / Solution
POST /v1/blocks/{id}/children → 400 invalid_request_url	This integration only accepts PATCH to append; notion_publish.py already uses PATCH
Want to "move" an existing block	Notion has no move API; only option is to rebuild: read old blocks → recreate under new parent → delete old blocks
Deleting a block yields Can't edit block that is archived	Use DELETE /v1/blocks/{id} (after PATCH {archived:true} you cannot modify it further)
Single rich_text too long	Each paragraph ≤2000 characters; seg() already splits at 1900 characters automatically (long English abstracts from the fallback route hit this regularly – tested with 2063 characters)
PATCH /blocks/{id}/children → 400 ... start_cursor provided is invalid	Notion eventual consistency jitter: appending immediately after creating a block can cause this. The write usually succeeds anyway. A subsequent GET children may still return 0 children. req() already backs off and retries on this 400. When debugging, don't immediately re-send the whole page; first traverse the root toggles to see the actual structure
Before re-sending the whole page	The script is not idempotent; rerunning adds a duplicate root toggle. First DELETE /v1/blocks/<previous root toggle id> (look for root toggle: in the logs), then rerun
Appending too many items at once	Each request ≤100 blocks (the script splits at 80)
Nested blocks not created	A single request can go at most two levels deep: a toggle with child toggles works; three levels must be appended in separate requests
S2 keeps returning 429	Normal; back off and retry. Parallel requests exacerbate this; serial + sleep is better

Pre-completion Checklist

• citations.json.source.citationCount matches the actual number of listed papers (≤20: list all; >20: union of two top-10 sets)
• Each paper has a meta-info line (citation count required), with clickable link if it has an arXiv ID
• Every paper without a Kimi summary is annotated with a reason (⚠️ marker + notes)
• Notion page has exactly one root toggle (no duplicates from re-sending)
• Number of papers in categories + unclassified = total number of papers
• Spot-check one toggle on Notion: title → meta-info → Q1 source text (subheadings, lists, bold – all present)