/bedtime — overnight autonomy
The user sleeps ~7–8 h. The failure to avoid: they wake and nothing happened. Second failure: compute burned on something pointless. Third: an in-session watcher that died with the session at 02:00 and left the GPUs idle. Design against all three, in that order.
1. Acknowledge (≤ 4 lines, no questions)
The user is leaving. Do not ask anything. State: what runs now, what you will do overnight (the picked item + fallback), where the morning report will be. If a decision is genuinely theirs, take the default that keeps compute busy and reversible, name the assumption in the ack, and again in NEED YOU in the morning.
2. Inventory — resources and candidates (≤ 10 min, cheap checks)
- Compute:
nvidia-smihere (shared with other users — take only free cards, nice long jobs, never touch theirs); remote nodes from the project's machine list, if one exists; cluster queues if any are in play. Note what is free vs busy vs finishing soon. - Claude budget: check remaining usage. Fleets and workflows overnight can drain the weekly limit; keep to the user's stop threshold. Prefer on-disk drivers (cost 0 tokens while they run) over agent-driven loops.
- Candidates, in priority order — pick the highest one that fits the free compute:
- Work the user already queued: today's conversation, the battleplan's next phase, the latest handoff file's "next session should pick up", WRAPUP_PLAN / TODO files.
- The experiment that most reduces uncertainty for the current goal (usually the paper): the ablation a reviewer would demand, the missing baseline, the seed repeat.
- Evals/metrics on finished checkpoints (project metrics) — cheap, always useful.
- Analysis of finished results → plots/report in the project's reports folder.
- Code hygiene — only if no compute-shaped work exists. Prefer GPU-hour-hungry work: idle GPU-hours are what is wasted overnight, not your time. Skip anything that needs the user's decision to be useful, anything already known pointless, and anything touching a design the user has fixed. Controlled comparisons: one factor at a time, nothing that breaks comparability with the baseline.
3. Launch — with the usual hygiene, plus durability
- Resolve the config and state what goes in (rows, filters, split, every deviation from baseline) in the log, not just your head. Smoke-test the exact command first.
- Detached driver on disk, not an agent loop:
nohup setsid <driver>.shwritingstate.txt/result.txt/ logs under<run>/(thefinal_now_driver.shpattern). The driver chains stage → stage and job → next job itself, with per-stage deadlines that fail loudly. If this session dies, the GPUs stay busy. - Verify effect, not action: process alive after startup, first log lines, first wandb points; wandb URL into the morning report.
- One item per free GPU max; leave a card free if a housemate's job is clearly finishing and will want it back — shared machine.
4. Watch — sparse, durable
- Arm a monitor (Monitor tool / ScheduleWakeup ~30–60 min) whose only job is: dead? → diagnose; relaunch if the cause is cheap and understood (OOM → smaller batch, path typo); otherwise fall to the next candidate. Finished? → the driver should already have started the next job; if it did not, start it. Idle GPU at any wake-up = act, not note.
- Expectation before observation at each wake-up: state the loss/epoch/step you expect, compare, one line in the overnight log.
- Context getting tight → write a handoff file
YYYY-MM-DD_overnight.mdNOW, including how to check every driver, then continue. - Pushes: none overnight except one, if the whole plan collapsed and cannot be recovered — they read it on waking. Everything else waits for the report.
5. Morning deliverable
- A report file
YYYY-MM-DD_overnight.md: what was launched and why (the ladder rung), what finished, numbers vs baseline, what died and what was salvaged, what is still running with absolute ETAs, links. - First message when the user appears: /qu format, unprompted, pointing at that file. Anything you decided for them → top of NEED YOU with the default you took.
Never overnight
No irreversible or outward-facing actions: no push to main, no force-push, no deleting runs/checkpoints/data, no paid-service spend beyond what they approved, no reopening locked decisions, no changes to paper claims. Nothing that needs them to be useful. Nothing on another user's process, GPU, or config.