ChenXuanyu-1216/source-trace

SourceTrace

SourceTrace checks an AI-generated knowledge tree against the source images it came from, and explains every problem with traceable evidence. The user should never have to take a finding on trust: each one says why it is a problem, where the source material is (page and quote), and what it rests on (evidence IDs).

A plain model can usually spot the errors. What SourceTrace adds is that the result is checkable: every node and edge gets a verdict, every verdict cites evidence, categories and risk levels are fixed, and the output has the same shape every time.

Inputs

• Source images: PNG/JPG, one image per page (e.g. slides exported as images). They are the single source of truth. If the user has a PPT or PDF, ask them to export it to images first.
• Knowledge tree: JSON following schemas/knowledge-tree.schema.json (nodes with id, label, definition, kind; edges with id, source, target, relation). If the tree uses another format or other relation names, convert it first and tell the user what you mapped.

Paths below such as scripts/…, schemas/… and references/… are relative to this skill's folder.

Outputs

Write all files to one output folder, by default sourcetrace-output/ next to the knowledge tree:

File	Stage	Content
evidence-index.json	2	Every piece of evidence extracted from the images (you write it)
audit.json	4–6	A verdict for every node and edge, plus omissions (you write the judgments; the script completes it)
report.md	6	The human-readable report (generated by scripts/build_report.py; do not write it yourself)

Never modify the user's knowledge tree. SourceTrace reports and recommends; the user decides.

Workflow

Run the six stages in order. Do not skip ahead to judging the tree before the evidence index exists: auditing against an index, instead of against your impression of the slides, is what makes the findings traceable.

1. Source ingestion

python scripts/validate_input.py --tree <knowledge-tree.json> --images <image folder or files> --json

• Errors ("valid": false): stop, show the user the errors in plain words, and ask them to fix the input. Do not audit an invalid tree.
• Warnings (e.g. DISCONNECTED_TREE, ISOLATED_NODE, DUPLICATE_EDGE): continue. build_report.py puts them in the report's Structure section.
• summary.images gives each image its source_id and page. Use exactly these in stage 2.

Then look at every image and note its title, text, tables and diagrams.

2. Evidence extraction

Build evidence-index.json following references/evidence-extraction.md. Read that file before extracting; in short:

• One item per slide title, bullet, table row and diagram relationship.
• Quote text verbatim in its original language. Small wording changes are what the audit must catch.
• Extract everything, including homework or reading notes.
• Set certainty honestly; low evidence cannot reject a tree item on its own.

Then validate it, and fix every error before continuing:

python scripts/validate_evidence.py --index <output>/evidence-index.json

3. Knowledge mapping

Treat the tree as a set of claims to test, never as fact. Split it into:

• Nodes: is the label and definition correct?
• Edges: is the relationship between the two nodes correct? Read edges as source relation target.

Audit them separately. A node can be correct while an edge using it is wrong (and the other way round), and each gets its own verdict.

For each node and edge, list the evidence items that bear on it. Search the whole index, not just the page you expect: some claims are only confirmed or contradicted by combining two pages.

4. Bidirectional audit

• Tree → source: for every node and edge, decide whether the evidence supports it. Every node and edge gets a verdict, including the correct ones; confirming what is right is part of the result.
• Source → tree: go through the evidence index and find significant content that no node or edge covers.

Follow references/audit-rules.md for verdicts, what counts as significant, and the edge cases.

5. Conflict detection

Classify every problem into exactly one category and give it a risk level. Use the decision order in references/audit-rules.md: go through its questions in order and stop at the first "yes". Do not pick a category by how a finding feels; the order is what keeps two runs consistent.

Category	Meaning
conceptual-error	A node's definition contradicts the source.
incorrect-relationship	An edge contradicts the source, even if both nodes are correct.
unsupported-expansion	Content not found in the source, even if true in general.
overgeneralization	Supported content stretched beyond what the source says.
ambiguous	Too vague or uncertain to confirm or reject.
omission	Significant source content missing from the tree.

6. Audit report

Write your judgments to audit.json, then let the script check them and produce the report. You decide; the script counts, numbers, sorts and formats. That split is what makes two runs produce the same report.

audit.json as you write it (v0.1, schema schemas/audit.schema.json):

{
  "schema_version": "0.1",
  "items": [
    { "target": "N001", "verdict": "supported", "evidence": ["EV-002"] },
    { "target": "N002", "verdict": "finding", "evidence": ["EV-008"],
      "category": "conceptual-error", "risk": "high",
      "reason": "...", "recommendation": "..." }
  ],
  "omissions": [
    { "category": "omission", "tier": "core", "risk": "medium", "evidence": ["EV-016"],
      "reason": "...", "recommendation": "..." },
    { "category": "omission", "tier": "core", "risk": "high", "risk_reason": "central concept of the lecture",
      "evidence": ["EV-003"], "reason": "...", "recommendation": "..." }
  ]
}

• items: exactly one entry per node and edge of the tree.
• reason: why it is a problem, in one or two sentences, quoting the exact words that differ.
• recommendation: a concrete revision that stays within the tree schema (allowed kind and relation values only).
• tier (omissions only): core or supporting, as defined in references/audit-rules.md.
• risk_reason: only when the risk differs from the default.
• Leave out summary, structure_warnings and finding_id; the script computes them.

Then run:

python scripts/build_report.py --tree <knowledge-tree.json> --index <output>/evidence-index.json --audit <output>/audit.json

• Errors: nothing is written. Fix audit.json and run again until it succeeds. Do not work around the checks.
• Success: the script rewrites audit.json with summary, structure_warnings and finding_ids (F-001, F-002, … in report order), and writes report.md next to it. The report language follows the slides; pass --lang zh or --lang en if the user asks for another.

Finish by telling the user where the report is, the counts from the summary, and the top problems in a few lines. Do not paste the whole report.

When the input is unusual

• Unreadable image: extract what you can with low certainty, mention the page in the affected findings' reason, and tell the user; do not invent content.
• The tree covers material not in the images (e.g. only some slides were provided): report those items as unsupported-expansion, and tell the user that missing pages may be the cause.
• Very large decks or trees: work page by page in stage 2 and node by node in stage 4, writing results to the files as you go rather than holding everything in mind.