novelty-verifier

Novelty Verifier (novelty-verifier)

Overview

novelty-verifier is Stage 3 of 4 in the Academic Research Skill Suite:

1. paper-analyzer (Stage 1) $\rightarrow$ Standardized technical deconstruction of individual papers.
2. research-gap-finder (Stage 2) $\rightarrow$ Cross-paper synthesis and hypothesis generation.
3. novelty-verifier (Stage 3) $\rightarrow$ Academic collision scan and dataset/compute feasibility verification.
4. proposal-drafter (Stage 4) $\rightarrow$ Formal research proposal generation.

Acting as an objective "Academic Police & Reviewer 2", this skill prevents wasted effort on already-published ideas and verifies that public datasets, pretrained checkpoints, and baseline code exist before drafting a proposal.

Language Adaptation Rule

• Output the verification report in the language requested by the user (Vietnamese or English), keeping technical terms, paper titles, query strings, and URLs in their original English form.
• Always preserve the YAML frontmatter schema in novelty_verdict_template.md for downstream consumption by proposal-drafter.

---

Workflow

Step 1: Deconstruct the Idea into Orthogonal Search Queries

Given a specific research hypothesis (e.g., IDEA-01 from research-gap-finder), decompose it into 3 orthogonal keyword sets (consult collision_and_pivot_guide.md):

1. Core Phenomenon / Target: e.g., "register tokens", "high-norm tokens", "attention artifacts", "vision transformer outliers".
2. Proposed Mechanism / Technique: e.g., "test-time", "training-free", "quantization", "vision-language", "token pruning", "dynamic routing".
3. Task / Domain: e.g., "dense prediction", "segmentation", "multimodal", "DINOv2", "CLIP".

Step 2: Run Automated Academic & Feasibility Scan

1. Execute search_academic.py with 2–4 narrow query combinations and dataset queries:

   python3 <skill_dir>/scripts/search_academic.py \
     --queries "<query_1>" "<query_2>" "<query_3>" \
     --datasets "<dataset_1>" "<dataset_2>" \
     --models "<model_1>" \
     --out "<workspace>/research_workspace/03_verification/search_results.json"
   

2. Complement the script results by calling the search_web tool for recent conference/workshop papers (ICLR, NeurIPS, ICML, CVPR, ECCV, ICCV, ACL, OpenReview) from the last 24 months.

Step 3: Classify Collision Severity

Inspect the abstracts and methods of the top retrieved papers and classify each competitor into one of three tiers:

• 🔴 Direct Collision ($\ge 80%$ overlap): Same core mechanism applied to the same bottleneck and evaluation regime.
• 🟡 Partial / Adjacent Overlap ($30%–75%$ overlap): Similar motivation or related technique, but differs significantly in mechanism, training requirement, target modality, or theoretical formulation.
• 🟢 Distinct / Complementary ($< 30%$ overlap): Provides supporting evidence or baseline code without encroaching on the core contribution.

Step 4: Verify Empirical Feasibility (Datasets, Checkpoints & Compute)

Check the huggingface_datasets, huggingface_models, and github_repos sections in search_results.json:

• Public Datasets: Are the required evaluation datasets openly accessible (e.g., on HuggingFace Datasets, TorchVision, or standard academic mirrors)?
• Pretrained Checkpoints & Baseline Code: Are official checkpoints (e.g., facebook/dinov2-base, openai/clip-vit-base-patch16) and baseline repositories publicly available so the user does not have to pretrain from scratch?
• Compute Budget: Confirm whether the experiments can run on accessible hardware (e.g., 1x–2x consumer/academic GPUs).

Step 5: Issue Verdict (GO / PIVOT / NO-GO) & Pivot Angles

1. Issue an explicit verdict:
   • GO: Zero direct collisions found; strong differentiation against adjacent work; datasets and checkpoints are immediately available.
   • PIVOT: High overlap detected on the naive formulation, but a sharp positioning angle (provided in the report) restores novelty.
   • NO-GO: Fully scooped by recent work with no viable differentiation, or blocked by proprietary data/compute.
2. Write the report using novelty_verdict_template.md to <workspace>/research_workspace/03_verification/novelty_verdict.md.