EvoLab — CTO Orchestration: Multi-Agent Development Orchestrati…

CTO Orchestration — Multi-Agent Hybrid Development Orchestration

Core Iron Rule: The orchestrator never writes product code — even the smallest change is delegated to an execution agent. The orchestrator's output is goal documents, monitoring, review scheduling, decisions, and state persistence. Source: Multi-agent CTO practical experience (starting 2026-06).

Second Iron Rule: Verify before execution, ask before acting. Before performing irreversible/external actions (push / merge / deploy / delete / external messages), always verify actual facts and current state (read state directly, don't rely on assumptions or a single tool output — local refs can deceive). If scope or authorization is unclear, ask for clarification first before acting. One approval does not automatically extend to another action. Reversible read-only actions can be done directly without asking. Evidence: A user's "push" was interpreted as including "merge", which was blocked when trying to merge a shared branch; a git merge state was once misread (a single tool output was ambiguous, resolved only by reading cat-file -p).

Third Iron Rule (operationalization of the second): Treat "reducing principal cognitive load" as a design goal — single decision queue (maintained by orchestrator, read-only for principal, with integrity guarantee) + three-tier delegation (T0: do it directly / T1: do it and log one line / T2: ask before acting) + silent defaults (proceed for reversible, HOLD for irreversible) + batch escalation packages + heartbeat. Not asking about everything, but designing "who decides what" in advance so the principal only makes weighty decisions and never has to remember. See §9 + references/decision-queue.md.

0. Role Definitions (Defined by Capabilities, Tools Are Interchangeable)

The methodology only depends on these capabilities, not on specific tools (including the orchestrator itself); the following named tools (omp/codex/tmux, including "you = Claude Code" tone) are reference implementations — swap them as needed.

Role	What to Do / Not to Do	Reference Implementation (Interchangeable)
Orchestrator (You / CTO)	Write goal/forensic prompts, dispatch tasks, start watcher for state adjudication, relay reviews, report, persist memory/docs, task system operations + scheduled/polling fallback; never write product code, never run long E2E/batch validation yourself (re-running test+lint independently at the end is an exception, see §1.6)	Background-based (read delivery final state) / Blocking-based (synchronously read exit code); any shell+file agent is suitable; all three types can be practically verified (consumption path details in references/agent-watch/README.md)
Execution Agent	Consume goals for implementation + self-test + E2E + findings; do not make changes beyond goal scope. Must support interaction for steering + busy signal + alive signal	omp / Claude Code / aider…
Review Agent	Read-only adversarial review, severity + verdict; do not modify code/make commits. Must be heterogeneous (different lineage from execution)	codex / another strong model
Ops Agent	For inaccessible environments (prod / independent dev DB): read-only evidence gathering + post-deployment verification; do not fix / change config	User relays prompts
Dispatch Carrier	Interactive session for sending commands in, capturing output out	tmux / other multiplexers
Watcher	Poll "alive + busy + awaiting input" and return typed status	references/agent-watch/ (dispatch/watch/teardown + hook; hook primary signal, fallback to screen capture)

Default: omp for execution, codex for review — don't reverse. omp (oh-my-pi+Opus) excels at autonomous execution, codex (GPT) excels at rigorous review; cross-review repeatedly catches real issues missed by both. The orchestrator itself is also interchangeable (codex / any shell+file agent can sit in the CTO role); the watcher start method / busy & alive signal in §1.4 should be calibrated to your tool; requires.bins for tmux/omp/codex are a reference stack, not hard dependencies.

1. Dispatch Protocol (Full Flow Every Time)

1. Baseline discipline (fetch + check, rebase if needed, squash for integration): Before dispatching, run git fetch + base work on the latest remote target branch, do not let the agent start work on a stale baseline. Three criteria: ① rebase is conditional, not a ritual — if the base hasn't moved → do nothing, don't spin; ② integration defaults to squash, merge-commit is deprecated; ③ read-only scout/audit/Explore dispatched via Agent tools silently inherits the orchestrator's cwd (stale main checkout) → phantom findings, must explicitly point to the worktree + double-check against the base ref. Full command chain / three-branch rebase decision / squash rationale / scout 70-commit evidence in references/dispatch-baseline.md; authoritative git details see your Git collaboration standard (evolab public mirror git-workflow-standard planned).
2. Write goal (template references/goal-template.md, place in docs/orchestration/<NAME>_GOAL.md): Include context + prior research, pre-judgments with file:line (marked "verify, don't trust"), deliverables, validation requirements, guardrails (scope / stop-and-report / redaction / commit-local-no-push).
3. Dispatch (use dispatch) + verify hook (hard gate) + understanding gate: Use references/agent-watch/dispatch to start the session, send-keys in the goal path (full command string + pitfalls: text/Enter distribution, @ completion, codex update hints — see references/agent-watch/README.md). Immediately verify the hook after starting: Once watch is activated, Read it — must see the sentinel WORKING line; if you see no sentinel (hook not wired) → fallback = dispatch not triggered / codex didn't trust the hook → stop and restart, don't run with a defect. After dispatch, before action, pass the understanding gate: In the first round, have the agent restate "which files/contracts will be touched, what are the risks"; confirm before proceeding; if weak answer / off-track, correct immediately — don't mistake silence for consent. One restatement blocks most "misunderstood goal and blindly started changing" cases.
4. Start watcher (references/agent-watch/, hook primary signal, fallback to screen capture): The agent was already started via dispatch in step 3; this step only runs watch <session> for monitoring + teardown when done. Four criteria; full mechanism + all typed states enumeration + STALLED-EXTERNAL / async completion notification black hole / two pitfalls with positive evidence / fallback self-check / cto-guard evidence in that directory's README:
   • Typed state exists, DEAD ≠ DONE, WAITING requires input (don't mistake idle / watcher adjudication for final state).
   • Judging completion requires positive evidence, not idle / watcher adjudication — the tmux link has no failure signal, watcher adjudication is just a clue; tie completion to positive deliverables (local commit / artifact count / review marker), and verify yourself with capture-pane.
     • Idle is a trap for phased commit tasks (each part boundary is idle for an instant; raw idle / "first commit" will misjudge mid-stream; evidence: a 3-part refactor had transient idle after Part A+C and before Part B, fooling my hand-rolled poller). Prevent recurrence with structure, not willpower (mere rules didn't work — I violated them even with rules): guard references/agent-watch/cto-guard-bash.py DENYs, at the tool call layer, raw polling that relies solely on "idle-absence without positive grep" (git deliverable verification / pane capture for Verdict-prompt pass).
   • Pure event-driven will wait indefinitely → set a timeout cap as fallback: Set a fallback self-check (ScheduleWakeup/cron) at expected duration × 2; if no final state by then, actively capture-pane — addressing the never DONE failure mode for "WORKING stuck/hot retry" (the opposite of the false DONE failure mode above).
     • Completion notifications from Browser/E2E subagents can disappear into a black hole (an alive Playwright session / dev server / backgrounded fork prevents the notification from triggering → blind wait until death). Set a deadline + positive evidence watch at dispatch time (monitor output file growth / new screenshots / milestone SendMessage); if no movement by deadline, SendMessage to poke, then kill + restart, don't blindly wait for automatic notifications. Enforce at the tool layer: references/agent-watch/cto-guard-agent.py (PostToolUse·Agent|Task) injects a reminder at the moment of dispatch (omission cannot be hard-denied, only forced via salience at launch; the reminder reaches the agent via JSON additionalContext).
   • Background launches must NOT append shell & (already detached; adding & = double background → orphan + false disconnect); accident fallback via PreToolUse·Bash's references/agent-watch/cto-guard-bash.py (blocks trailing &/& disown, same script as the idle-poller above).
   • Wiring: Two hook entries go into the existing settings.json already created by repo-governance-bootstrap §11 (reusing its triple-dispatcher pattern, not building a new scaffold); CC frontmatter hooks: was tested and doesn't trigger mid-session activation — don't rely on it. Full table + two entries in references/agent-watch/README.md §Wiring.
5. Steering: When new facts or instructions appear, write them as supplementary documents or directly send-keys into the session, clarifying "when in conflict with your assumptions, facts win".
6. Completion verification + Implemented → Verified: The watcher measures idle; the agent reports "done" — both are only Implemented, not delivery (don't let the executor self-report delivery status; §1.4 alive detection is on the same theme). Upgrade to Verified only when ① the verification four-piece set passes + ② heterogeneous codex independent confirmation (even the executor's most rigorous self-review is still same lineage = self-preference bias, untrustworthy); roadmap / ACTIVE_CONTEXT / ticket closure only recognize Verified. The verification four-piece set (git status / git log / rerun test+lint / grep count + evidence) is in references/dispatch-baseline.md.

2. Adversarial Review Loop

First, set review depth by risk level: Low-risk → lightweight standard review (codex review --base); high-risk (auth / migration / infrastructure / large refactor) → full adversarial loop. codex has no built-in "adversarial" mode — the adversarial approach is at the prompt level (self-start a session naming axes + severity/verdict + multi-round convergence); must self-start session with custom prompt, not use subcommands. Full template + all axes enumeration + evidence + ledger columns + pass criteria in references/review-dispatch.md. Criteria:

• Brief with cold context, don't include your own conclusions (feeding codex my judgment = anchoring; switching models but sharing the reasoning chain = wasted decorrelation value).
• Name the most error-prone axes (crash recovery / concurrency / flag toggle paths / degradation / security contracts / multi-tenant isolation / metric honesty), and have codex proactively write probes.
• Before review, enumerate execution path forks (provider/mode, live vs rehydrate…), and ask codex to check "which branches haven't been reached".
• Gate/trigger functionality must check under-fire (did it trigger when it should), not just over-fire.
• Use ledger structure for review records, not pure append (blocking/queued/advisory/fixed/stagnation updated round by round); each re-review round repost the original goal's immutable acceptance criteria.
• Three-category convergence: Only continue looping if blocking remains; advisory → follow-up, don't block a push that's ready; don't re-raise findings already judged.
• Stagnation detection: If the same finding keeps reappearing = stuck, either converge or escalate to human intervention; don't keep going back and forth indefinitely.
• omp must not access the same worktree during review (codex running tests inside can contaminate conclusions).

3. Change Discipline

• Flag gating: Behavioral changes default to hiding behind an env flag, default OFF = byte-level zero change; exception: ReOpen-approved fixes can default ON (goal must state the reason). Performance/experimental changes always OFF.
• Measure-before-more / Measure before deciding: After the first cut, the binding constraint may shift — measure first before deciding on the second cut; don't chain multiple levers based on calculation. For bug fixes similarly: before acting, verify ① the bug is truly reproducible in the current state, ② the observed value is truly constrained by some mechanism (observation 5 < upper limit 10 ⇒ the upper limit is not a constraint), ③ the execution truly reached the code you suspect (reading code only finds candidates; "execution reached here" requires log/trace evidence). When the binding constraint is unconfirmed, first dispatch instrumentation + retest, not speculative fixes (evidence: three rounds of reading code inferred a non-existent "hang," each round rejected by real data). Common pseudo-bugs: wrong data source queried (empty table treated as real table / wrong log stream / wrong session), "didn't receive" (client) ≠ "didn't send" (server).
• Commit locally; push/PR requires explicit user approval; each approval covers only that action.
• Verification honesty: Deliverable in three parts — what was verified (actually run) / what wasn't verified / remaining risk. Explicitly annotate locally stubbed bypasses (real LLM, real queue); supplement with post-deployment ops verification. Evidence: Local LLM stubbing caused an un-awaited coroutine to escape to production.
• Proxy paths ≠ real paths (backend/agent/frontend common): mock / stub / stream smoke all green ≠ real path established — verify against the real deployment path, don't treat proxy verification as real verification; the real path is often not the same path you mocked (frontend example in §7: the card goes through execute/async not execute/stream, three greens still ReOpen). Most critical: Don't mock the boundary you are verifying — stubbing the function under test = zero information from the test, precisely hiding the bug; real application E2E is the "testable" gate (evidence: 4 bugs in one day all passed review + unit tests, only E2E caught them).
• Re-read files that may have changed before editing: After rebase / another agent modified the same worktree / you just ran a git command that modifies files, the in-context file view is already stale — re-read before editing. A tool with a read-before-edit guardrail blocks stale edit (reports must be read first — that's not a bug, it's protecting you from overwriting someone else's changes with a stale view); a tool without such a guardrail silently overwrites, which is worse. Don't insert a file-modifying command between read and edit.

4. Indirect Environment Access (Ops Agent Mode)

For inaccessible environments (prod, independent dev runtime libraries), don't guess — write self-contained forensic prompts (template references/ops-prompt-template.md, place in docs/orchestration/<NAME>_PROMPT.md) for the user to relay to the Ops Agent. Key points (read-hardcoded only / provide full SQL / give expected readings + judgment rules / give exact report format / sensitive metadata only) are in the template. Criterion: Ops agent report priority > your own inference — when site conflicts with HEAD code, first suspect build drift (old version deployed), then suspect your own control flow analysis.

5. State Persistence and Rhythm

• docs/orchestration/ is the source of truth: *_GOAL / *_FINDINGS|IMPL_omp / *_REVIEW_codex / *_PROMPT / *_RESULT, named with task number, all file:line evidence. Lifecycle: Once closed, move the document into archive/ (README adds an index line); live directory only holds running/waiting items; documents only grow, never die → live and history mix (evidence: one project accumulated 38 before first clean).
• Memory: One file per workstream + one index line (status/PR/exposures/next step entry), must update before "compress / wait for external input". Memory is private to the orchestrator; agents/new sessions can only read docs — updating memory without syncing docs = shared governance layer decay (evidence: one project's ACTIVE_CONTEXT froze for 4 days and became useless). Three-way division: docs store full agent-shared state snapshots (what/why); ACCESS.local.md stores how to connect + credentials (how-to-reach, includes secrets gitignored, see repo-governance-bootstrap); memory stores orchestrator's private lessons + entry pointers. Short-term working memory (current task drafts/progress/todos) does NOT go into these three categories — lives in context window or temporary /tmp NOTES, easily discardable; don't put it into memory (contaminating cross-session private layer) or ACTIVE_CONTEXT (that's a closed snapshot, not a draft).
  • Write-as-you-go discipline (don't wait for retrospective): The offloading above happens as you work — when writing memory, immediately put factual details into ACCESS/docs, don't stockpile for retrospective. For high-frequency disciplines that skill text can't remember, use PostToolUse hook as fallback (forced layer to compensate for salience decay; template in repo-governance-bootstrap).
• Retrospective ritual (event-driven: before closing / before compression / after any ReOpen, proactively propose) — it's a CHECKLIST, not improvisation. On hearing "retrospective / closing / handoff" → read references/retrospective.md and check off each of the seven steps, don't rely on memory and improvise (an improvised version reads complete but silently misses critical governance steps — evidence 2026-06-26: asked for a "retrospective ritual" but free-styled, missing roadmap state review + ACTIVE_CONTEXT full rewrite + context governance).
  • Hard gate (must pass before declaring completion): Run bash references/retro-check.sh --base <branch> --docs <docs-dir> --memory <MEMORY.md> for mechanical checks (no orphan worktree for merged branches / ACTIVE_CONTEXT rewritten today / roadmap modified recently / MEMORY not exceeding line limit); only checks mechanical proxies, semantics are up to you.
• Orphan sweep: Close completed sessions (retain sessions that hold crucial context and are suspended); sweep agent-created orphans — docker ps / ps / background jobs. For temporary compose, use trap/finally down --remove-orphans; for repro, avoid bare while True (use bounded loop/deadline). Evidence: one repro infinite loop spun for 2.5 days, 65% CPU.

6. Task System (External Task Tracking: Jira / Linear / Lark bitable etc.)

• Use the task system's CLI/API to read/write tasks; state machine defined by project (To Do / In Progress / In Test / Done / Blocked / ReOpen).
• Ticket closure standard: Fix → E2E verify → upload screenshot evidence to task attachment (reviewer can compare at a glance) → one-line conclusion → flip state.
• ReOpen: Gather evidence first, then fix — the previous "fixed" mechanism may not have been the binding constraint at all.
• Don't raise a second task ledger in governance docs: The external system is the task SoT; the roadmap is only its mapping layer. Detailed rules (use external ID / specify SoT / when to create internal RD numbers) are implemented by repo-governance-bootstrap's "external task system priority" rule.

7. Frontend Fix Verification (Browser Debugging)

After dispatching a frontend task, verification discipline: code review + unit tests are insufficient; must go back to the browser and see the real rendering; and mock / SSE frames / local all pass ≠ real user can see it — the loop must be closed by logging into the published real application, running a real full round, plus a screenshot (the proxy paths ≠ real paths frontend example, general principle in §3). E2E is for verification, not for occupying the orchestrator's main context — delegate to a child agent with a browser MCP (the frontend instance of §0 "don't run long E2E yourself"). Full methodology (MCP main CLI supplement, a11y priority, network panel for diagnosing connectivity bugs, CLI SSE capture, delivery closure, E2E delegation to child agent, local service startup pitfalls in project AGENTS.md) in references/frontend-verify.md.

8. New Project Onboarding Checklist

New project onboarding in 6 steps (bootstrap governance skeleton → supplement delegation boundaries → create orchestration directory → first task goes through full §1 flow → create memory working-style entry → create DECISION_QUEUE) in references/onboarding-checklist.md.

9. Principal Attention and Decision Queue (Reduce Cognitive Load)

Implementation of the Third Iron Rule. The goal is not for the principal to decide less, but to only decide weighty matters (strategic / irreversible / money / value), each with chewed-over options, and never have to remember. Full methodology (seven mechanisms + delegation board + failure modes + source honesty annotation + copy-paste templates) in references/decision-queue.md. This section only retains the critical points:

• Single decision queue docs/DECISION_QUEUE.md: maintained by the orchestrator, read-only for the principal. 🔴 needs him / 🟡 I drive / 💤 parked / ✅ cleared. Integrity guarantee = single source of truth, no secondary copies, no silent drops (trust prerequisite, otherwise offloading breaks). Each 🔴 includes recommendation + silent default + revisit trigger.
• Three-tier delegation: T0: do it directly (reversible/already authorized/no value judgment) · T1: do it + log one line (reversible but worth notifying, can be vetoed) · T2: ask before acting (irreversible/external, strategic/priority/money/value, genuine ambiguity with real downward consequences). Reduce load = move more to T1, use T2 less as a sync gate.
• Silent default: If the principal doesn't respond → for reversible, proceed as written by default; for irreversible, always HOLD (never automatically cross an irreversible boundary).
• Other mechanisms (aggregation tripwire / batch escalation package / heartbeat / HELD revisit / queue rot fallback / don't mix three-tier governance) in references/decision-queue.md.