0x6869726f6b69/claude-skills

Deep Review-like: Split → Parallel Verify → Refute → Synthesize

Replicates the publicly known design of Google's Deep Review (successor to the Paper Assistant Tool, PAT, trialed for authors at STOC, ICML, NeurIPS, and ICLR) using Claude sub‑agents. The goal is to surface real errors (proof holes, miscalculations, numerical inconsistencies, claims stronger than evidence, missing comparisons) that the author would not notice themselves, presented in a verifiable, priority‑ordered way. No scores are assigned.

Stages and why each exists:

Stage	Who	Why
1 Ingestion	Script	Prepare both text and page images. Extracted text corrupts formulas/tables, so images are needed
2 Segment split & budget allocation	You	Focus computation on dense proof sections; skim introductions and related work
3 Parallel deep verification	Sub‑agents (segment × perspective, lens)	Narrow the scope of deep reading at once; use independent eyes with different perspectives for coverage
4 Matching & deduplication	Script	Mechanically verify that verbatim quotes exist in the paper; group identical issues
5 Adversarial verification	Sub‑agent	Error finding produces more false positives with repeated trials. A skeptical second look restores precision
6 Synthesis	Sub‑agent	Determine severity, set priority, control length, produce Overall Feedback
7 Report	Script → You	Create report.md / report.html, summarize key points concisely

The design rationale (which findings from which papers each design decision comes from) is in references/research_basis.md.

Below, <skill_dir> is the directory containing this SKILL.md, <ws> is the working folder, and deep_review.py abbreviates python <skill_dir>/scripts/deep_review.py. If python is unavailable, use python3.

0. First: Confirm depth and focus

1. Identify the document. PDF is best (page images can be generated). Also accepts .md / .txt / .tex, but then cannot check figures or formulas via images. If only LaTeX source is available and it can be compiled, convert to PDF first.
2. Ask for depth and focus in one question using AskUserQuestion or similar (if unavailable, ask in chat). Skip questions already answered in the activation message. If the user says "just do it" or "delegate", start with standard depth and no focus.
   • Question 1 "Depth": Standard (recommended) / Quick / Deep (attach a table with approximate effort guidelines)
   • Question 2 "Focus areas" (multiple choice): None / Proofs & formulas / Experiments & statistics / Novelty & related work (allow custom entry for venue or free text)

Depth	Segments	Traces per segment	Lenses	Targets for adversarial verification	Approx. agents	Approx. time
quick	2–4	1	L0 (synthesis)	Major and those with unknown location	4–8	10–20 min
standard	4–8	High 2, others 1	L1, L2, L3	Major and those with unknown location	10–20	20–45 min
deep	5–12	High 3, Medium 2, Light 1	L1, L2, L3	+ Minor related to formulas/statistics/numerical, plus a critique agent	20–40	45–90 min

Time varies greatly with paper length.

3. If the user seems to be reviewing someone else's manuscript as a reviewer, add a brief note to check the target venue's AI usage policy (many publishers and conferences prohibit inputting a manuscript under review into AI tools). This is not needed if the author reviews their own manuscript.

1. Ingestion

python <skill_dir>/scripts/deep_review.py init <path_to_document> --depth standard --focus "<focus if any; omit otherwise>" --venue "<target venue if any; omit otherwise>" --lang en

• Working folder is <document_folder>/<document_name>_deep_review/ (if not writable, place under current directory; can be overridden with --ws)
• Creates paper.txt (pages separated by === Page N ===), pages/page_NNN.png (150 dpi), pages.json, config.json
• Use the per‑page outline (word count, formula symbol ratio, number of equation labels, number of theorems/proofs, headings, figure/table captions) in the next stage
• Read any WARN output. If the extracted text is poor (e.g., scanned PDF), add a note in the instructions for each reviewer to rely primarily on page images
• Report language uses --lang (ja / en). Match the user's language. Quotes remain in the original language

2. Segment split and budget allocation

Examine the outline and write <ws>/segments.json. For pages where the heading alone does not reveal the content, peek at the corresponding page in paper.txt (no need to read the full paper).

Rules:

• Split by logical units (theory, method, experiments, introduction, etc.). Units may overlap and need not be contiguous. For example, put "Theorem statement in Sec. 3" and "Its proof in Appendix B" in the same segment
• Cover all pages in some segment (plan will check)
• Assign track based on information density:
  • high: Main theorem and its proof, core derivations, the heart of the algorithm that claims correctness. If the paper's main contribution is experimental results, the section with the main result table/figure also qualifies
  • medium: Method description, experiments, additional experiments in appendix
  • light: Introduction, related work, discussion/conclusion
  • none: References, acknowledgments, etc. (reason required. References are checked by lens L3)
• high segments should be ≤ 8–10 pages (split if longer)
• theme: intro / related_work / discussion / other / method / algorithm / system / theory / appendix_proofs / experiments / analysis / appendix_experiments / references
• Optionally add a one‑line notes field with information for the reviewer (which claim this segment supports, location of dependent definitions, etc.)

{
  "segments": [
    {"id": "S1", "title": "Introduction and contributions", "pages": "1-2", "theme": "intro", "track": "light",
     "notes": "Contributions C1–C3 are listed on page 2"},
    {"id": "S2", "title": "Main theorem and its proof", "pages": "4-5,13-16", "theme": "theory", "track": "high",
     "notes": "Proof of Theorem 2 (p. 5) is in Appendix B (p. 13–16). Assumptions are Assumptions 1–3 on p. 3"},
    {"id": "S3", "title": "Proposed algorithm", "pages": "3-4", "theme": "algorithm", "track": "medium"},
    {"id": "S4", "title": "Experiments", "pages": "6-9", "theme": "experiments", "track": "medium"},
    {"id": "S5", "title": "Related work and conclusion", "pages": "9-10", "theme": "related_work", "track": "light"},
    {"id": "S6", "title": "References", "pages": "11-12", "theme": "references", "track": "none",
     "reason": "References are spot-checked by lens L3"}
  ]
}

Then create the plan:

python <skill_dir>/scripts/deep_review.py plan <ws>

plan.json lists the agents to launch (task ID, assigned segment, track, traces = perspectives, pages, output path, recommended effort). If errors occur, fix segments.json and rerun.

Report to user (one line): Split complete: 6 segments (High 1, Medium 2, Light 2), launching 9 verification agents in parallel.

3. Parallel deep verification

Launch all tasks in plan.json in parallel in a single message using the Agent tool (if more than 10, split into waves of ~8). Each task has an effort field; if the Agent tool supports effort, use that value (to allocate more reasoning to High track tasks).

Instruction for segment reviewer (kind: segment)

You are a Deep Review "Segment Reviewer." First read the following thoroughly and follow the procedure and output format.
  <skill_dir>/references/segment_reviewer.md
  <skill_dir>/references/checklists.md (only the sections relevant to your theme)
  <skill_dir>/references/schemas.md §1 and §4

- Full paper text: <ws>/paper.txt (=== Page N === is the page number)   Page images: <ws>/pages/page_NNN.png
- Original document: <ws>/<input_copy from config.json>
- Task: <T03>   Segment: <S2> "<title>"   Pages: <4-5,13-16>   Theme: <theory (proof)>
- Track: <high>   Trace: <B — counterexamples, boundaries, assumptions>
- Segment notes: <notes, or "none">
- User-specified focus: <focus or "none">   Target venue: <venue or "none">
- Output: <ws>/<candidates/T03_S2_B.json>
- Utility: python <skill_dir>/scripts/pdf_tools.py (crop, render --grid to enlarge figures or equations)

Read the full paper for context, but focus your findings on the assigned pages. Another reviewer is looking at the same segment from a different perspective.

For a Light‑track task bundling multiple segments, list all segments in the segment field.

Instruction for lens reviewer (kind: lens)

You are a Deep Review "Lens Reviewer (<L2 novelty/importance>)." First read the following thoroughly and follow the procedure and output format for your lens section.
  <skill_dir>/references/lens_reviewers.md
  <skill_dir>/references/checklists.md (§6–§8)
  <skill_dir>/references/schemas.md §1 and §4

- Full paper text: <ws>/paper.txt   Page images: <ws>/pages/page_NNN.png
- Task: <T08>   Lens: <L2>
- User-specified focus: <…>   Target venue: <…>
- Output: <ws>/<candidates/T08_L2.json> (include overall_notes)

Look across the entire paper through the lens of your assigned perspective. Deep per‑scope verification is handled by other reviewers.

After all reviewers finish, rerun plan to check that all tasks are marked [done]. Retry any missing tasks once. If some still fail, proceed and ask the synthesis agent to note the limitations.

Report to user (one line): Verification 9/9 complete

4. Matching and deduplication

python <skill_dir>/scripts/deep_review.py merge <ws>

• Checks whether each candidate's verbatim quote actually exists in the paper (exact / fuzzy / not_found) and its page; groups nearby candidates into clusters (K01, …) and writes them to merged.json
• support indicates how many tasks independently found the same issue—more independent finds increases likelihood of a real error
• Outputs the batch for adversarial verification (V01, …). If warnings about invalid or missing outputs appear, record the responsible reviewer's issue

Report to user (one line): 64 candidates → merged into 38, of which 9 will be adversarially verified

5. Adversarial verification

Launch all verify_tasks from merged.json in parallel in a single message.

Instruction for adversarial verifier

You are a Deep Review "Adversarial Verifier." First read <skill_dir>/references/verifier.md and
<skill_dir>/references/schemas.md §2 all the way through.

- Full paper text: <ws>/paper.txt   Page images: <ws>/pages/page_NNN.png
- Verification targets: clusters <K01, K04, K07> from <ws>/merged.json (relevant pages: <…>)
- Output: <ws>/verdicts/<V01>.json

For each candidate, first try to refute it from the standpoint that the paper is correct. Then only mark as confirmed if you can independently reproduce the issue yourself (with concrete evidence such as recalculation, counterexample, or two contradicting quotes).

After completion:

python <skill_dir>/scripts/deep_review.py collate <ws>

Creates collated.json, linking verification results to clusters. refuted / cannot_locate are recommended for removal.

Report to user (one line): Adversarial verification complete: confirmed 5, plausible 2, false positives 2

6. Synthesis

Instruction for the synthesis agent

You are a Deep Review "Synthesis Agent." First read <skill_dir>/references/synthesis.md and
<skill_dir>/references/schemas.md §3 and §4 all the way through.

- Input: <ws>/collated.json (candidates, verification results, lens observations, perspective distribution), <ws>/paper.txt,
        <ws>/pages/, <ws>/config.json (depth, focus, venue, language)
- Output: <ws>/findings.json
- After writing, run python <skill_dir>/scripts/deep_review.py render <ws> --check and fix until no errors
- Report language: <English>   User-specified focus: <…>   Target venue: <…>
- Extra notes (limitations to include): <missing tasks or other issues, or "none">

Deep only: Critique agent. After the synthesis agent finishes, ask a separate sub‑agent to do the following:

You are a Deep Review "Critique Agent." Check <ws>/findings.json (synthesis agent's draft) against <ws>/paper.txt and
<ws>/pages/. Examine every entry from the perspectives in <skill_dir>/references/synthesis.md §8
(accuracy of quotes and locations, concreteness, severity validity, existence of external references, tone, length).
Verify every Major yourself. Do not look for new issues.
Write the results in <ws>/critique.md as a bullet list with "ID • Problem • How to fix."

Once critique.md is ready, ask the synthesis agent (via SendMessage) to read critique.md, incorporate valid points into findings.json, and pass render --check. If SendMessage is unavailable, launch a new synthesis agent with the same instruction.

Finally:

python <skill_dir>/scripts/deep_review.py render <ws>

Generates report.md and report.html. The HTML allows filtering by Major / Minor / Suggestions and toggling between report order and page order.

7. Report to the user

Read findings.json and summarize concisely. Do not paste the full report.

• Counts (Major / Minor / Suggestions) and number of false positives removed by adversarial verification
• One line per Major concern (ID, location, key point)
• One or two sentences for the overall take (bottom_line)
• Files: report.html (open in browser), report.md, working folder
• One note of caution: these are AI‑generated findings; each should be verified by the author before making changes to the manuscript

Suggest next steps: deep dive into a specific finding, re‑review after revision, or a scored review‑revision loop (paper-review-loop).

Follow‑up

• Deep dive (e.g., "Tell me more about MJ2"): launch a single segment reviewer for that finding's scope with high track, trace B (counterexamples), incorporate result into findings.json, then render
• Re‑review of revised version: run from scratch with a new PDF (init in a new working folder). Add the path to the previous findings.json in the synthesis agent's instruction and ask it to fill in a previous field
• Resume after interruption: check which files exist in the working folder (segments.json → [done] in plan.json → merged.json → verdicts/ → collated.json → findings.json) and continue from the last completed stage. init --force deletes previous results, so use it only when you want a full restart

When called by another skill

Skills like review2revise‑loop may invoke this skill as the review stage of an iterative cycle. If the caller specifies input, working folder (--ws), depth, focus, venue, and previous findings.json:

• Skip the questions in §0. Use the specified values.
• Run §1–§6 normally to produce findings.json, report.md, and report.html. If a previous findings.json is provided, include a previous field in the synthesis agent's instruction.
• For §7, report only one line with the counts (Major / Minor / Suggestions) and return control to the caller. Do not suggest next steps.
• The review criteria, depth, and upper bounds for counts remain the same as usual. Do not relax them for the caller's convenience.

Environment without sub‑agents

The same stages apply, but you perform each task sequentially yourself. Independence is reduced, so follow these rules:

• Recommend quick depth (or standard if the user insists)
• Before each task, re‑read the assigned pages from paper.txt and page images; do not rely on memory
• For each task, look only through the perspective of the guide for that task (e.g., segment_reviewer.md)
• For adversarial verification, do not look at the reasoning you wrote when you created the candidates; reassess only from the paper's relevant passages

Principles and their rationale

• No unverifiable findings: Use verbatim quotes, verify formulas/tables via images, recalculate with code, check external references for existence. A report contaminated with false positives wastes the author's time and erodes trust even in real errors.
• Priority over volume: Major concerns must impact the paper's main claims. Enforce count limits. Overly detailed or excessively long reports are the most disliked aspect of AI reviews.
• No scores: Deep Review focuses on errors and improvement points; it does not assign subjective scores. LLM‑generated scores are optimistically biased and unstable. If scores are desired, use the paper-review-loop skill.
• Decisions remain with humans: This is a pre‑submission authoring tool, not a substitute for peer review. Final judgment lies with the author and reviewers.
• Protect manuscript confidentiality: Do not include manuscript text or unpublished title in web search queries. Do not send the manuscript to external services.
• Do not modify the original files: All work takes place within <ws>.