agent-desktop

What does agent-desktop do?

agent-desktop is a CLI that lets an AI agent observe and operate desktop apps through the OS accessibility tree. It is a tool, not an agent. Every command prints one JSON envelope (ok, data or error). The observe, act and verify loop runs in you.

Shell syntax

Examples use a POSIX shell. In PowerShell, quote every ref ('@s8f3k2p9:e1'), because a bare @token is the splatting operator. Use $env:NAME = 'value' instead of export, and | ConvertFrom-Json instead of $(...) and jq. The Windows skill has the PowerShell forms.

Preflight

Run this before the first command of a task. It is the only place the tool gets installed from.

1. Run agent-desktop version. Exit 0 means the CLI is present: go to step 4.

2. If it is missing, install it with npm. npm is the only supported path:

   npm install -g agent-desktop

   Do not try Homebrew, Scoop, a downloaded archive, or an older version. This skill is written against whatever version npm resolves. If npm is not on PATH, run node -v and where npm (Windows) or which npm (macOS, Linux) and report what you found. Do not guess another package manager.

3. Run agent-desktop version again. If it still fails, stop and tell the user: the last 20 lines of the install output, whether agent-desktop --help runs at all, and the OS you are on. Do not fall back to screenshots or mouse automation on your own.

4. Run agent-desktop status. It reports the platform, permissions and supported_surfaces. On macOS a missing Accessibility grant shows up here; tell the user to run agent-desktop permissions --guide instead of trying to grant it yourself.

5. Run agent-desktop skills get platform. It loads the skill for the OS the binary runs on (macOS, Windows or Linux), and it wins wherever it disagrees with this file.

6. For flags and syntax, run agent-desktop <command> --help. This skill never repeats flag lists, so --help always matches the installed version.

If agent-desktop skills get desktop prints rules that differ from this file, the binary is right: this file is a copy, the binary is what you are actually driving.

The loop

Use find when you know the role or exact name. Otherwise map the app first and drill into one region.

agent-desktop snapshot --skeleton --app "Notes" -i --compact   # shallow map, children_count on cut branches
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # drill into one region
agent-desktop click @s8f3k2p9:e12                              # act on a qualified ref
agent-desktop snapshot --root @s8f3k2p9:e3 -i --compact        # verify: re-read the same region

• A small app needs no skeleton. Take a full snapshot -i.
• -i keeps interactive nodes only, so a row's static-text label is dropped. To tell which row a repeated button belongs to, drill that row's container with --root and leave out -i. On Windows and Linux that reads the row but does not make its button actionable: a button repeated in other rows is a twin in the window and its ref is refused (see Refs).
• A --root drill adds its refs to the same snapshot id. Refs outside the drilled subtree, and the drilled root itself, stay valid; refs that were already inside it are replaced, so an old ref there returns STALE_REF.
• Never skeleton a surface (--surface menu, sheet, alert). A surface is already focused.
• An action that opens a menu, sheet or alert lists it in data.surfaces. Target that overlay next.
• After an async step, use wait (see references/commands-system.md). A fixed sleep is a race.

Refs

• Output refs are qualified: @<snapshot_id>:e<N>, for example @s8f3k2p9:e1. Pass them to later commands as they are.
• Refs are valid for one snapshot. After any UI change, re-drill the affected region or take a new snapshot.
• Only addressable elements get refs. Static text and inert containers do not.
• A command re-identifies the element before it acts. On macOS the ref carries the element's own address, so identical twins each keep their ref. On Windows and Linux a ref resolves only when it was the only element in the window its snapshot captured with its role, native id, name and description, counted even below a skeleton or depth cut, and exactly one live element still matches. Identical twins there have no ref path: only --headed mouse-click --xy X,Y can choose one, it takes the foreground, and a Linux Wayland session refuses it.
• STALE_REF: the element is gone or no longer matches. details.kind is saved_ref_absent, or retained_object_stale on macOS. Take a fresh snapshot and use the new ref.
• AMBIGUOUS_TARGET: an identical element appeared (saved_ref_ambiguous), or the ref was not proven unique at capture (identity_not_unique). Repeating the ref or re-snapshotting identical twins gives the same refusal; act on a control that is unique in the window. identity_not_unique also follows a snapshot that could not read the whole window in time: retake it with a larger --timeout-ms.
• TIMEOUT with details.kind: resolution_identity_unknown: a live read did not answer, so identity is unproven. Nothing was sent; retry.
• SNAPSHOT_NOT_FOUND: the snapshot id expired. Snapshot again.

Delivery and retry

Action results carry data.disposition and errors carry error.disposition, each with delivery and retry. Observation commands such as snapshot have none on success; their errors, and every INVALID_ARGS refusal, carry not_delivered and retry: safe.

• Retry or substitute an action only when disposition.retry is safe. Only not_delivered is safe.
• Any other delivery value means the action may have landed. A repeat can click, type or submit twice, and toggle can undo itself.
• On a result that is not safe, read the live state (get, is, or a fresh snapshot) and decide from that. details.post_state helps when present.
• Branch on error.code and error.disposition. Message and suggestion text can change.
• details.retryable: false means waiting cannot change the outcome. Try a different approach.

Headless and headed

Ref actions are headless by default. They use semantic accessibility calls and do not take focus, move the cursor, synthesize keys or use the clipboard. If no semantic path exists, the action fails closed. Use the global --headed flag only when the user explicitly permits physical input, such as double-click, hover, drag, or focus. Never switch to headed mode to recover from a headless failure. Raw coordinates (--xy) never imply focus. Which actions need --headed differs by OS: read the platform skill.

References

Load one only when you need it: agent-desktop skills get desktop <reference>.

Reference	Open it when
commands-observation	You need snapshot, find, get, is, screenshot, list-surfaces semantics, partial trees, or the --root rules
commands-interaction	You choose between click, type, set-value, select, toggle, scroll, press and mouse commands, or you debug an action error
commands-system	You launch or close apps, manage windows, use the clipboard, wait, batch, sessions, multi-agent cursors, tracing, or load skills
workflows	You want worked patterns (forms, menus, dialogs, waiting for a new window, scroll-find, drag) and the anti-patterns list