evidence-check

Evidence check

An independent audit. It assumes nothing in the draft is true until it is checked. Use it as the last step of research-plan, on the user's own draft or bibliography, or on any report before delivery.

Independence

• Best: a fresh subagent given only the files (draft, ledger/bibliography, notes, raw texts). Never a fork of the writing conversation.
• No subagents: a same-session check.
  • Re-read every file from disk before judging anything, and do not rely on what you remember writing.
  • Label the result same-session (not independent).
  • Tell the user that for anything high-stakes they should run this skill in a new chat on the same folder.

Deterministic checks first

Run the bundled checker first: python3 <this skill's folder>/scripts/research_checks.py all <dir> (or quotes <claims.json> / links <plan.md>). It is standard-library Python and runs in your own code sandbox; the user installs nothing. PDF text needs pypdf: if the import fails, pip install pypdf in the sandbox and retry. Its errors are findings: anchors that don't resolve, missing ⚠ flags, count mismatches, quotes that aren't in the source text, fake PDFs. Its warnings are judgment calls. If the folder has no claims.json (e.g. auditing the user's own draft), write one next to the draft for every claim you mark supported, so the quotes check can confirm each quote is verbatim.

Inputs

The draft; the reference list (ledger.md, a .bib, or a numbered list); notes and raw source texts if held. If only the draft is given, ask once for the bibliography.

Step 1 — Extract claims

Pull out every verifiable assertion:

• citation claims;
• numbers;
• named papers, datasets and tools;
• existence claims;
• negative claims ("no prior work…");
• uncited certainty words (proves, establishes, cures, prevents, conclusively, always, never).

Skip opinions, suggestions and the draft's own definitions.

Step 2 — Resolve

Each citation maps to exactly one reference. List orphans in both directions and author-year collisions.

Step 3 — Real? (per reference)

Look each one up now: OpenAlex / Crossref / Semantic Scholar / PubMed, plus Reference Checker check_references and check_retractions.

verdict	meaning
verified	title, first author and year match a record found in this session
partial	the work exists but a fact is off, or only part could be confirmed (e.g. the book but not the chapter)
mismatch	the DOI/ID resolves to a different work, or the citation names the wrong book or journal
not_found	searched properly, nothing matches
unchecked	could not look it up (blocked, budget, tool down); not evidence of fabrication
suspicious	red flags in the entry itself

Rules:

• Book chapters: compare the cited book title, editors, pages and publisher with the Crossref record (container-title, page). A right DOI with the wrong book is a mismatch; this exact error got past a trial run.
• Tool disagreement: an exact DOI match in Crossref or OpenAlex beats a title-search not_found. Record both results.
• What counts as a record: a DOI landing page, publisher, PubMed, PMC/Europe PMC, arXiv, OpenAlex, Crossref, Semantic Scholar, DBLP. A snippet or another paper's reference list does not. Memory is unchecked.
• Also check retraction status, and whether a preprint has since been published.

Step 4 — Supported? (per claim)

State what the source must say for the citation to be fair, then:

support	rule
supported	a verbatim passage of 40+ characters from raw source text (API XML, PDF text, a user file) says it, with a locator. Quote it in the report.
partial	the source says something narrower or weaker
contradicted	the source says otherwise
not-found-in-read-part	only part of the text is held, and the claim isn't in it
unverified-extracted	read only through WebFetch, which returns a model's extraction; can never be supported
unverified	no text held (abstract / metadata only, paywalled, blocked)

Checks that ride on Step 4:

• Numbers: copied exactly, not rounded. A number that isn't in the source is contradicted.
• Flags: each claim needs the flag matching its evidence: ⚠ abstract-only · ⚠ metadata-only · ⚠ extracted · ⚠ preprint. Add any that are missing.
• Independent count: "consistent across N studies" must say how many are independent.
• Negative claims: must name the searches behind them. Otherwise rewrite as "not found in ".
• Background vs evidence: a source cited for a finding must actually report it.

Step 5 — Outcome

Judge the final state after repairs, and log each repair ("fixed in round n: …").

• PASS: no not_found / mismatch / contradicted / orphans. Every unverified* item is flagged inline.
• PARTIAL: no errors, but some items cannot be resolved (e.g. a chapter whose contents can't be checked) or stayed unchecked. List them.
• FAIL: any not_found, mismatch, contradicted, retracted-but-cited source, or orphan remains after at most 2 repair rounds. Deliver with this banner at the top, followed by the unresolved items:

INVALID EVIDENCE — unresolved citation or support failures; do not treat as verified.

Pushback is not evidence. Concede only to a new source or a better argument, and log why.

Output

Append a collapsed Evidence check block to the draft:

Evidence check — <date> — PASS | PARTIAL | FAIL — <fresh context | same-session (not independent)> — route: <tools>
Deterministic: research-checks <pass/fail, n errors> | not available
References: n verified · n partial · n mismatch · n not_found · n unchecked · n suspicious · n retracted
Claims: n supported · n partial · n contradicted · n unverified-extracted · n unverified · flags added: n
Repairs: <round: what changed>
| # | Claim (short) | Cite | Real | Support | Quote / locator or reason | Fix |
Fix first: <invented/mismatched refs and contradicted claims → orphans → flags → style>

Privacy

Search with titles, DOIs and topic words only. Never paste the user's unpublished text into a search.