assistant-lab は何をしますか？
Self-improvement workbench for /assistant. This is the lab where
/assistant diagnoses problems, trains new models, evaluates them, and
promotes passing models into the live registry.
Not to be confused with /monitor-skills which is the observability
daemon that watches ALL skills for health, drift, and sync issues.
/assistant-lab is specifically the self-improvement toolbox.
The Self-Improvement Loop
/monitor-skills detects problem ← observability
↓
/assistant-lab diagnoses root cause ← this skill
↓
Shadow mode: teacher (scillm) labels flow into shadow.jsonl
↓
ModelFactory.auto_improve(task)
├─ reads shadow agreement rate
├─ >= 90%: promote → update model_registry.json
├─ 80-90%: plateau → /prompt-lab redesign
├─ 70-80%: retrain → /create-gpt or /create-classifier
└─ < 70%: aggressive retrain + architecture change
↓
/gpt-lab benchmark or /classifier-lab evaluate
↓ passing? → promote to live registry
↓
/assistant uses the improved model at inference time
Tools Available
| Tool | Purpose | When Used |
|---|---|---|
/create-gpt | Train QLoRA/SFT/GRPO GPT (Tier 1.5) | Agreement < 80% or no model exists |
/create-classifier | Train DistilBERT/sklearn classifier (Tier 0.5) | Text classification tasks |
/create-regressor | Train sklearn/XGB regressor (Tier 0.75) | Continuous prediction tasks |
| RunPod Flash | Serverless GPU training via Python SDK (7B–70B) | Model too large for local A5000 — replaces /ops-runpod for training |
/gpt-lab | Benchmark GPT against teacher baseline | After training, before promotion |
/classifier-lab | Evaluate classifier accuracy/F1 | After classifier training |
/prompt-lab | Redesign prompts when plateau detected | Agreement stuck at 80-90% |
/scillm | Tier 2 teacher — creates labels | Always (teacher is ground truth) |
Usage
CLI
# Diagnose what a task needs
./run.sh diagnose --task stress-test-grader
# Full autonomous improvement loop
./run.sh auto-improve --task stress-test-grader
# Train specific model type
./run.sh train --task stress-test-grader --type gpt
./run.sh train --task sparta-ambiguity --type classifier
# Evaluate a model
./run.sh evaluate --task stress-test-grader --type gpt
# Promote a passing model (disables shadow mode)
./run.sh promote --task stress-test-grader --type gpt
# Harvest teacher labels from shadow.jsonl
./run.sh harvest --task stress-test-grader --since 24h
# Show shadow agreement stats for all tasks
./run.sh status
# Run full self-test (train → eval → promote cycle on test data)
./run.sh self-test
Python API
from assistant_lab import AssistantLab
lab = AssistantLab()
# Diagnose: what does this task need?
diagnosis = lab.diagnose("stress-test-grader")
# → {"has_gpt": False, "has_classifier": False, "shadow_agreement": 0.0, ...}
# Auto-improve: decide + train + eval + promote
result = lab.auto_improve("stress-test-grader")
# → {"actions": ["trained gpt", "evaluated (passing=True)", "promoted"]}
# Manual steps
lab.train("stress-test-grader", model_type="gpt")
lab.evaluate("stress-test-grader", model_type="gpt")
lab.promote("stress-test-grader", model_type="gpt")
Model Factory
The core engine is ModelFactory (from /common/model_factory.py).
/assistant-lab wraps it with CLI + diagnostics + reporting.
Shadow Agreement Thresholds
| Agreement Rate | Action | Rationale |
|---|---|---|
| >= 90% | Promote | Student reliably matches teacher |
| 80-90% | /prompt-lab redesign | Plateau — prompts may be the ceiling |
| 70-80% | Retrain with more labels | More data likely helps |
| < 70% | Aggressive retrain | Change architecture or base model |
| < 50 samples | Wait | Not enough data to decide |
Minimum Training Data (NON-NEGOTIABLE)
Do NOT call /create-gpt train or /create-classifier train with insufficient data.
| Model Type | Minimum Samples | Rule |
|---|---|---|
| GPT (QLoRA SFT) | 1,000+ total | < 1,000 → stays at Tier 2 teacher |
| Classifier (sklearn/SetFit) | 200 per class | n_samples / n_classes >= 200 |
| Regressor | 100+ total | Standard sklearn guidance |
Evidence: sparta_stress_grading was trained on 246 samples → 33.7% agreement.
That's 4 months of wasted shadow labels because the model was trained too early.
Collect first, train when ready.
Training Data Flow
Tier 2 teacher (scillm via persona)
↓ labels saved to shadow.jsonl
↓
/assistant-lab harvest --task X
↓ extracts input+output pairs
↓
/create-gpt train --task X --data labels.jsonl
↓ QLoRA on Qwen2.5-1.5B (default)
↓
/gpt-lab benchmark --task X
↓ compare student vs teacher
↓
/assistant-lab promote --task X --type gpt
↓ updates model_registry.json, shadow_mode=false
Remote Training (RunPod Flash)
For models too large for the local RTX A5000 (24GB VRAM, max ~1.7B with LoRA),
assistant-lab uses RunPod Flash — the serverless Python SDK with no Docker,
no SSH, and no rsync overhead. Flash replaces /ops-runpod for all training
paths. /ops-runpod is retained for persistent inference servers only.
When to Use Flash
| Model Size | Local A5000 | Flash Needed | GPU |
|---|---|---|---|
| 0.5–1.7B | LoRA fits | No | ./run.sh train (local) |
| 3–7B | Can't fit | Yes | B200 (preferred) or H200 |
| 8–13B | Impossible | Yes | B200 (preferred) or H200 |
Flash GPU Types
| GPU | VRAM | Notes |
|---|---|---|
B200 | 192 GB HBM3e | Fastest; 3–5× H200 on MoE / long-context |
H200 | 192 GB HBM3 | Good availability; solid baseline |
Billing: pay-per-second, 7-day execution maximum per job.
Flash cost estimates (approximate)
| Model Size | GPU | Est. Training Time | Est. Cost |
|---|---|---|---|
| 7B QLoRA SFT | B200 | ~1–2 hrs | ~$5–15 |
| 7B QLoRA SFT | H200 | ~2–3 hrs | ~$8–20 |
| 13B QLoRA SFT | B200 | ~2–4 hrs | ~$10–25 |
| 13B QLoRA SFT | H200 | ~3–5 hrs | ~$12–30 |
Commands
# Step 1: Always estimate cost first
./run.sh estimate --task taxonomy-assessor --target flash --gpu B200 --size 7B
# Step 2: Train on Flash B200 (requires --confirm for safety)
./run.sh train --task taxonomy-assessor --target flash --gpu B200 --size 7B --confirm
# Train on Flash H200
./run.sh train --task taxonomy-assessor --target flash --gpu H200 --size 13B --confirm
# With custom base model
./run.sh train --task taxonomy-assessor --target flash --gpu B200 --size 8B \
--base-model meta-llama/Llama-3.1-8B-Instruct --confirm
# With different quantization
./run.sh train --task qra-validator --target flash --gpu B200 --size 7B \
--quantize Q5_K_M --confirm
Flash Training Pipeline
estimate --target flash --gpu B200 --size 7B
↓
train --target flash --gpu B200 --confirm
├─ 1. Cost estimate + budget gate
├─ 2. Flash serverless job submitted (Python SDK — no Docker/SSH)
├─ 3. Dataset streamed to Flash worker
├─ 4. QLoRA / SFT training runs on B200/H200 (192GB VRAM)
├─ 5. Pull model weights to /mnt/storage12tb/models/
├─ 6. create-gpt export --quantize Q4_K_M (GGUF)
└─ 7. Evaluate + auto-promote if passing
Safety Gates
- Budget cap:
--max-cost(default $15.00) blocks training if estimate exceeds limit - Confirmation: Must pass
--confirm— without it, shows cost and exits - Auto-teardown: Flash job is cancelled on exit (even on error, via trap)
- Metrics logging: Every Flash training run logged to
lab_metrics.jsonl
Integration with Self-Improvement Loop
Remote-trained models integrate with the same cascade as local models:
Tier 0: Heuristic (free, instant)
Tier 0.5: Classifier (local, free)
Tier 1.5: Local GPT ≤1.7B (create-gpt, free after training)
Tier 1.5: Remote GPT 3-13B (train-remote, free after training) ← NEW
Tier 2: scillm/DeepSeek V3 (Chutes, $0.12/1K calls)
A 7-13B model at Tier 1.5 can significantly reduce Tier 2 (Chutes) escalations, potentially paying for its training cost in reduced overages.
Relationship to Other Skills
/monitor-skills ──→ "skill X is unhealthy"
↓
/assistant-lab ──→ diagnose → train → eval → promote
↓
/assistant ──────→ uses promoted model at inference time
/monitor-skills: Observes ALL skills. Detects problems. Reports health./assistant-lab: Fixes /assistant's models. Trains, evaluates, promotes./assistant: Runs inference. Uses whatever models the lab has produced.
Contract
- Input: Task name + optional model type
- Output: Diagnosis, training results, evaluation results, promotion status
- Dependencies: model_factory.py (from /common), create-* and *-lab skills
- Metrics: Appends to
~/.pi/assistant/lab_metrics.jsonl