ModelSieve · 模型筛
Vendor-neutral model-intelligence layer. Not a gateway — only aggregation, tiered recommendation, readable output.
When to use
- User wants selection advice: "Recommend a model", "Which one for coding?", "Best value", "Best regardless of cost".
- User wants side-by-side comparison: "Claude vs GPT?", "Is there something better at this price?"
- User wants price/quality intelligence: "Cheapest model now?", "Noteworthy recent releases?"
- User wants to integrate recommendations into workflow (optional): use
adapters/to write tier results back to Dify / LangGraph / n8n / otari.
Core positioning
- White space: Intelligence layer + write-back adapters. Gateways/routing (OmniRoute, LiteLLM, otari) is a red ocean — we don't reinvent.
- Explainable & revertible: Every recommendation includes source link and one-sentence rationale; user always knows "why this model" and can override.
- Always cite authority: Real-time prices from OpenRouter (reliable); quality scores prioritize Artificial Analysis / LMArena, otherwise explicitly mark as "heuristic/community signal, not authoritative benchmark".
Source matrix
| Dimension | Authoritative source | Access method | What it provides | Notes |
|---|---|---|---|---|
| Real-time price / availability | OpenRouter /api/v1/models | Free, no key needed; radar/snapshot.py fetches directly | 340+ models real $/1M, context limit, modalities | Most reliable price signal |
| Quality index | Artificial Analysis Intelligence Index | Public leaderboard; free key for API (AA_API_KEY) | Composite intelligence index, speed (tokens/s), price | radar/quality_aa.py integrates; heuristic fallback without key |
| Human preference | LMArena (ex Chatbot Arena) | Free API + daily JSON snapshot; arena-rank PyPI reproducible | Elo human preference scores (closest to "feels good to use") | Per-task leaderboards (text/code/multimodal) |
| Anti-contamination benchmark | LiveBench (livebench.ai) | Website; monthly refreshed questions | Math/code/reasoning, resistant to benchmark gaming | Complements AA/LMArena |
| Coding specialty | SWE-bench Verified / Aider Polyglot | Official leaderboards | Agentic coding / editor-style coding | Model selection for coding tasks |
| Community testing | r/LocalLLaMA (Reddit) | WebSearch / WebFetch | Quantization failures, real-world usability, Apple Silicon reports | Highest signal-to-noise forum |
| New release tracking | Hugging Face Trending / Daily Papers / LLM Stats | WebSearch | Models released within 24h, community reaction | For cutting-edge releases |
| Open-weight aggregation | Vellum / Onyx open-weight leaderboard | Website | Cross-comparison of open-weight models only | Self-hosting perspective |
Full access details in
references/sources.md(bilingual).
Workflow
1. Classify intent
best_quality/ "best regardless of cost" → highest quality ceiling (high both on AA Intelligence Index and LMArena Elo).best_value/ "best bang for buck" → optimize quality/price ratio (calculate value index using real-time prices).cheapest/ "cheapest fallback" → lowest price while still capable of the task.- If user specifies task domain (coding / reasoning / long-context / vision), overlay task-specific leaderboard filtering.
2. Gather (on demand, prioritize real-time)
- Price/availability: Run directly
python radar/snapshot.py(online fetch from OpenRouter; or--offlinefor cache). - Quality scores: Prefer Artificial Analysis (with key,
radar/quality_aa.pyuses real scores automatically); otherwise WebFetch LMArena / AA leaderboards. - Community/new releases: WebSearch "r/LocalLLaMA " or " benchmark 2026".
- Use structured APIs whenever available, not just web summaries; cross-validate multiple sources to avoid vendor self-promotion.
3. Rank
- Quality tier: Descending AA Intelligence Index primarily, with LMArena Elo as cross-validation.
- Value tier:
value index = quality score / average price, pick top. - Budget tier: Ascending real $/1M, exclude clearly toy models (context <8K or discontinued).
- Each selected model includes one-sentence rationale + source link.
4. Present
- Strictly follow
references/output-template.mdreadable template (table + one-line summary + disclaimer). - Default output: Markdown table. If user wants "visual" rather than "read", use Visualizer to render comparison cards.
- Always include: ① data timestamp/source; ② reminder "run 20 private evaluations on your real task before finalizing".
Boundaries and Caveats
- If quality score does not come from AA/LMArena, must explicitly mark "heuristic/community signal, not authoritative".
- Leaderboards have biases (LMArena over-indexes English-speaking tech users, sparse coverage of closed-source new models) — cross-validate, don't blindly trust a single source.
- We do not "fully auto-pilot" for users; recommendations are advisory. Final decision and write-back require explicit user confirmation.
Relationship with other parts of the repo
radar/snapshot.py: Daily entry, fetches prices + three tiers + cached snapshot.radar/diff.py: Day-over-day drift comparison (price changes, delistings, model swap suggestions).radar/quality_aa.py: Real Artificial Analysis quality score integration.adapters/{dify,langgraph,n8n,otari}: Write back three tiers to user workflows (default dry-run).- More design details in
docs/architecture.mdanddocs/roadmap.md.