Astronomical Agent Debugging & Post-Mortem
Trace symptoms back to the earliest violated contract, fix only the necessary layer, then separately prove correct run, mechanism effectiveness, and tangible gains. Record each conclusion independently.
This skill is distilled from the readable historical materials of the "astronomy-agent" project. Cases are diagnostic clues and do not imply that the current implementation necessarily has the same defects. Historical budgets, nodes, dates, card names, and pass lines must not be used as default constants for new tasks.
Establish Identity
Confirm whether the goal is bug fixing, diagnosis, equivalence acceleration, strategy improvement, or delivering a fix. Identify the actual path and current version; check CURRENT, freeze list, receipts, and final package; identity is based on source code, input, configuration, and result hashes.
Record the official protocol/scorer version, online-visible fields, initialize constraints, run mode, CPU calibration, and independent wall-clock limits. Only when there is a generic website error or a local card with the same name, the cloud root cause remains unconfirmed.
Distinguish: fully valid, budget-terminated, technical failure, user interruption, diagnosis completed but scientific gate not passed. Some season scores do not enter the full-season ranking; missing values remain null.
Read References by Symptom
- Illegal action, startup/ABI/manifest/ZIP, data numbering, backup cleanup: read protocol-and-data.md.
- Many allocations but low scores, zero scores, missed repairs, REQ/request omissions, replay divergence: read state-and-physics.md.
- Timeout, model with no gain, diagnosis acceleration but full season unchanged, holdout/acceptance failure: read budget-and-experiments.md.
- Check historical counterexamples and repair evidence: read casebook.md; check original file identity: read evidence.md and corpus-index.tsv.
Find the First Divergence
Compare along the actual data flow: Input/Constraints → Consumed Messages → State/Cache → Candidate Entry → Full Score/Allocation → Final Action → Official Execution → Feedback Settlement.
Find the first divergence from the minimal failing state or immutable trajectory with the same input; save expected, actual, unit, denominator, sample selection and source. Annotate source reproduction, log correlation, mechanism assumptions, and gain evidence separately.
First correct the protocol, identity, and replay contamination, then explain scientific differences. Faithful replay reproduces the original consumption sequence; altering historical behavior creates a new candidate and must not be mixed into the original trajectory. When initialization or key messages are missing, stop at the recognizable boundary; do not fabricate missing hidden states.
Design Minimally Falsifiable Contrasts
One hypothesis paired with one test that can disprove it. Synthetic fixtures for boundary validation, public state shadow for entry and selection mechanism validation, complete paired seasons for comprehensive performance validation, independent holdout for limited generalization; each layer only supports its own conclusion.
List the preserved invariants relevant to this task: physics/scoring, public information, RNG, message order, ledger, native ABI, candidate/tie order, and billing. If equivalence acceleration changes actions or candidate set, reclassify it as a strategy change.
For strategy candidates, first trace whether they can enter the actual candidate chain, be actuarially evaluated, satisfy completion protection, and change execution selection; close the hypothesis if the predetermined mechanism gate is not passed. Protocol fixes validate against real boundary counterexamples, without forcing changes to normal card actions.
Execute Limited Validation
Scoring and remote models use current authorizations and limited budgets; do not inherit remaining quotas from closed studies. Select meaningful boundaries, differentials, and normal controls based on the change.
When the project uses a registrar, confirm planned success, read back identity, source/input hash, and output isolation before starting; handle registration failure first. Exceptions still settle actual cost and request count in finally. Technical fixes preserve the failing parent version with a sub-version; annotate post-hoc registration as retrospective.
For final source code, compare with same source, same input, and billing OFF/ON. After source code changes, only reference evidence that still matches that source code. Report each required gate individually: complete final state, budget, REQ, request, science/total score, and required API roles. Failure on any required gate means non-acceptance.
After failure, pivot to a root cause hypothesis with new evidence; changing seed, picking source, adjusting pass line, or widening scan is not a fix for the original acceptance. Holdout sets used for selection or diagnosis become development data; subsequent generalization uses new holdout sets.
Output and Completion Standards
Output according to problem scale:
Symptom and first failing contract:
Level of evidence and source identity:
Root cause confirmed / pending:
Minimal change and preserved invariants:
Results: Protocol / State / Mechanism / Full Season / Holdout / Deployment
Failures, unexecuted items, and uncertainties:
Next minimal test to distinguish hypotheses:
The uploaded package is generated from the final frozen source code, built/run in a new directory via the official manifest parser; verify JSONL, resources, and source/ZIP hashes. Report mock, local native, real API, Linux, and website separately; do not substitute one for another.
Any new experience must include applicable conditions, minimal reproduction, and evidence; only retain rules that actually change decisions.