Desktop Takeover

Overview

Unified orchestration for "having an agent take over the desktop to complete GUI operations": first route to determine the best path, then ensure computer use permissions are ready, and finally dispatch the task and verify. Computer use is a fallback path, not the preferred one—only used when the task must be completed via GUI.

Workflow Decision Tree

When a desktop operation request is received, determine the path in the following order, stop on the first match:

1. Connector/MCP directly accessible?
   Target app has an already-connected connector (GitHub, WPS knowledge base, WeCom, etc.)
   → Use the connector, don't touch GUI
   Example: Check a GitHub PR → gh connector; Read a WPS knowledge base doc → wps-knowledgebase

2. Pure browser operation?
   Task takes place on a webpage (fill forms, web clicks, screenshot comparison)
   → Use browser-use skill, don't touch system GUI
   Example: Log in to a website to change settings → browser-use

3. Office/WPS file read/write?
   Target is doc/docx/xls/xlsx/ppt/pptx file content
   → Use the office skill family (workbuddy-docx / sheetagent / local-office-edit)
   Example: Modify a table in Word → local-office-edit

4. Command line/script accessible?
   Task is fundamentally file operations, data processing, system configuration, and has a CLI or script path
   → Use Bash/scripts directly
   Example: Batch rename → script; Check system info → command

5. None of the above → computer use desktop takeover
   Task must operate on a GUI application's interface elements (buttons, menus, non-web forms)
   Example: Change a setting in some domestic software that only has a GUI entry

Decision principle: With each level down, reliability and speed decrease, uncertainty increases. GUI operations involve multiple uncertainties such as element positioning, window focus, and interface language, making it the last resort.

Note: The visibility dependency of level 5 (window enumeration + screenshots) is a structural weakness when Stage Manager is enabled (see Pre-flight item 2 and the "Invisible Window Trap"). AX tools (accessibility tree perception) read logical geometry and are not fooled by Stage Manager's thumbnail displacement; they are an alternative/supplementary path—re-checked on 27.0.1 has partially confirmed this (see Trap section Finding 5), with the caveat that newly launched windows that have never been on the Stage are also invisible to the AX tree. This skill's built-in AX fallback tool is: tools/axctl (usage in the "AX Fallback Layer" section).

When routing, if unsure whether a target app has a connector, first check list_installed_plugins, don't assert from memory.

Pre-flight: Dual Permission and Environment Check

Before going to computer use, two pre-flight checks must be completed, neither can be skipped:

1. Permission Check

Call gui_permission_status:

all_granted: true → Proceed to the second pre-flight check
Missing permissions → Guide the user to grant permissions following the flow in references/authorization-macos.md (macOS)
Windows → The current version only supports macOS; clearly inform the user

2. Stage Manager Pre-flight Check (Mandatory; users often have Stage Manager on)

Execute defaults read com.apple.WindowManager GloballyEnabled:

Returns 1 (enabled) → Before dispatching, must open -a <AppName> to activate the target app and bring it back to the current Stage (this will briefly steal focus; inform the user first), then confirm the target window is visible with gui_list_windows, and only then create_task. If the window is still not in the list after activation, go directly to the "Invisible Window Trap" troubleshooting sequence below—do not retry create_task, do not conclude the app is faulty
Returns 0 (disabled) → Proceed directly with the normal dispatch flow

This pre-flight check upgrades the Stage Manager trap from "post-diagnosis" to "pre-emptive avoidance". Stage Manager is not uncommon among macOS 15+ users; for those who have it enabled, this path is the default for every dispatch, not a special case. The activate-manifest mechanism was re-verified on macOS 27.0.1 on the same day and holds true, but with two new details: "first activation doesn't stick" and "manifestation then retraction", see the "Invisible Window Trap" section's 27 re-check subsection.

Task Dispatch

Once permissions are ready, follow the standard workflow of the built-in computer use skill:

gui_list_windows to locate the target application's window; if not found, open_app to launch it
Verify visibility after launch: After open_app, must run gui_list_windows again to confirm the window is actually in the list. If it doesn't appear, don't jump to "the app is broken"—first see the "Invisible Window Trap" below
When Stage Manager is enabled (pre-flight confirmed): First open -a to activate the target app, then dispatch—see Pre-flight item 2; no need to re-evaluate here
create_task(window_id, task) to dispatch the task

Invisible Window Trap (Stage Manager Scenario)

Scope of Conclusion: Established by controlled experiments on macOS 15; re-checked and re-verified on macOS 27.0.1 (Swift dual-channel probe + native AX API binary test; mechanism confirmed with new details, see below). AX layer conclusions have been confirmed by 27.0.1 post-authorization testing (AXError=0 path), no longer dependent on inference.

Trigger Condition: Stage Manager is active, unrelated to background launch itself. When enabled, windows of applications not currently on the Stage do exist but cannot be properly captured by window enumeration and screenshots—the process is alive, windows are created (e.g., Cmd+N works), but gui_list_windows doesn't see normal windows. Once the app is activated (window brought back to the current Stage), the window immediately becomes visible.

macOS 27.0.1 Re-check New Findings:

Thumbnail bar mechanism visualized: Windows from staged-out apps are reduced to side thumbnail windows (tested: Calculator main window shrunk to 39×100pt, flush left, plus 4 full-width 34pt Stage placeholder windows). If gui_list_windows filters by "normal-sized windows," these thumbnail windows may be missed or misidentified
Activate-manifest still holds, but with a "first activation doesn't stick" phenomenon: Tested: first open -a left the main window in thumbnail state (when another app occupied the foreground); second open -a expanded it to full size (405, 517, 230×408). Action: After activation, immediately verify with gui_list_windows; if not manifested, open -a again (max 2 times); only if still not manifested should you proceed to the troubleshooting sequence
Manifested window may retract: If foreground is taken by another app, within ~5 seconds the window can shrink back to thumbnail state (reproduced in testing). Corollary: Screenshot verification has a time window—if foreground is taken during create_task execution, the task itself is unaffected, but if gui_screenshot is taken after retraction, the target window will be out of frame. Action: If a foreground switch just occurred before taking a screenshot, open -a first, then capture
CGWindowList channel unaffected: optionOnScreenOnly and full list can enumerate the window (though titles require screen recording permission)—but note that the onScreenOnly list contains thumbnail windows, not the main window. Tools relying on window coordinates for click targeting will click the wrong position (clicking the thumbnail instead of the real button)
AX tree reads logical geometry, immune to thumbnail displacement, but has blind spots (27.0.1 post-authorization testing): The AX return window frame is the full-size logical coordinate (e.g., (405,517,230×408)), while at the same time the CGWindowList rendering state is a 39×100 thumbnail—Stage Manager's geometric displacement does not fool AX. But blind spot: For a newly launched window that has never been on the Stage, the AX tree reports windows=0; after it has been on the Stage once and then moved to background, AX continues to see it. Corollary: AX perception is not unconditionally immune; apps launched for the first time still need to be activated onto the Stage first. Also: Sending synthetic mouse events to AX logical coordinates on screen will click the wrong app (logical position ≠ current rendering position)—AX tools must use accessibility action APIs like AXPress (e.g., axctl), not coordinate-based clicks

Troubleshooting Order (when the target app is not in the window list):

First check if the process is alive (pgrep): alive → likely a visibility issue, not an app fault
Confirm if the user has Stage Manager enabled (macOS 15+, defaults read com.apple.WindowManager GloballyEnabled, 1=enabled): enabled → go to step 3; not enabled → possibly window doesn't exist anymore or other reason
Activate the app to make it visible: open -a <AppName> (will briefly steal focus; inform the user first); On 27, the first activation may not stick; verify immediately after activation; if not manifested, activate again (max 2 times)
Run gui_list_windows again; usually the window_id will be available

Additional Signal: When Stage Manager is enabled, open_app usually does not get a window_id (returns a hint to do a second enumeration); when disabled, returns window_id normally.

Important Corollaries:

Reporting no visible windows ≠ window doesn't exist, and ≠ previous operation failed—the previously "failed" task may have actually succeeded (window created, just not visible)
Dialog-type windows (e.g., "Open" file dialog) become invalid once closed by the task; subsequent screenshots will error—this is normal; re-enumerate to get a new ID
Some apps (e.g., TextEdit) have no window when launched without a document; this is normal behavior; need to open with a file

Three-Part Task Description (Critical for Success)

The task parameter must include three parts:

Goal: Describe step by step what needs to be completed
Boundaries: Clearly state what must not be touched, e.g., "Do not click any entry in the history list"
Failure handling: If an action has no response, accurately record what happened and proceed to the next step; do not retry or change the method

The failure handling rule is not optional—without it, a single failure will trigger repeated retries, exhausting the entire step quota.

In the description, do not include pixel coordinates or numeric positions; use visible text and control names on the interface to refer to targets.

Example

User request: "Help me open the XX app and change the export format to CSV"

Goal: 1. Find and click the "Settings" or "Preferences" entry in the menu bar or toolbar
2. In the settings panel, find the "Export Format" or "Export Format" option
3. Change its value to CSV
4. Save settings and close the panel
Boundaries: Do not modify any other settings; do not click the "Reset" button
Failure handling: If the above controls are not found, record the actual control names seen and end the task; do not guess-click

AX Fallback Layer (axctl)

When the built-in computer use is unavailable (not enabled, gui_* tools missing, task too small to warrant dispatching) or semantic-level element operations are needed, use this skill's built-in AX fallback tool tools/axctl. Purely Accessibility API implementation, no mouse touching, no focus stealing, not limited by AppleEvents sandbox (WorkBuddy sandbox prohibits sending control events to System Events; the osascript path is deprecated; do not attempt it further).

Location and Installation: Binary tools/axctl, source tools/axctl.swift (Swift single file), both distributed with the skill. Before first use, run sh tools/install.sh: if swiftc is detected, it will compile on-site (recommended; source is auditable, tailored to the local architecture); otherwise, it uses the pre-compiled binary in the package (Apple Silicon only). After modifying the source code, recompile by running swiftc -O -o axctl axctl.swift in the tools directory.

Prerequisites: The host must have the "Accessibility" TCC permission (check in System Settings → Privacy & Security → Accessibility for the host app, then restart the host for it to take effect; confirmed ready when AX API call returns AXError=0).

Subcommands

axctl list <bundle-id|pid>        List windows: title + logical geometry frame
axctl tree <target> [depth]       Read UI tree: role/title/description/value/frame (default depth=2)
axctl actions <target> <title>    List available actions for a matching element
axctl press <target> <title>      Perform AXPress (semantic click, element-level, no coordinates needed)
axctl activate <target>           Activate app (NSRunningApplication.activate)

<target> supports bundle id (recommended, e.g., com.apple.calculator), numeric PID, or localized app name (lowercase matching—on Chinese systems, it's "计算器" not "Calculator"; name matching is error-prone; prefer bundle id).

Usage Rules

Element Lookup: First tree or actions with a known button title to confirm AXPress exists, then press. press works only on elements with the AXPress action; menu items use AXPick; check per type
Action Verification: After press returns PRESSED, use tree to read the target element or the new value of an associated display area as evidence (see Calculator test: after pressing "3", the StaticText value becomes 3)—same principle as computer use's screenshot verification; text self-report doesn't count
Stage Manager Scenario: AX tree reads logical geometry; thumbnail-state windows are readable and operable (tested); but newly launched apps never on the Stage have an empty AX tree—first axctl activate or open -a to bring it to the Stage before operating
Scope: axctl is suitable for element-level read/write (buttons/menus/text fields); for cross-app menu bar clicks, system-level keyboard/mouse simulation, or complex multi-step GUI flows, still use the built-in computer use dispatch
Known Limitations: No UI change observer (axmcp's observe_changes capability; will be re-evaluated after upstream adapts to Swift 6.4); single-element operation per call, no batch transactions

Test Record (macOS 27.0.1, Stage Manager Enabled)

axctl press com.apple.calculator "3" → PRESSED path=w/0/0/0/1/0/15 → tree reads back AXStaticText value as 3—full chain of tree reading, positioning, semantic click, and effect verification works.

Result Verification

A GUI agent's text self-report is insufficient as evidence of completion—residual old UI state may be misjudged as success. You must:

After the task returns done=true, call gui_screenshot(window_id) to get the final screenshot
Use the screenshot content to determine if the goal was achieved
When reporting to the user, note that the conclusion is based on the screenshot

Between rounds, do not take screenshots by default (the continue_task return value already contains execution history); only take an additional screenshot when the agent's reasoning/action_desc remains unchanged for several consecutive rounds to determine if it's stuck.

Troubleshooting Quick Reference

Symptom	Action
`permission_denied`	See references/authorization-macos.md
Stuck on the same step for multiple rounds	Call `continue_task` with `hint` parameter providing the exact visible text of the control
Interface unchanged but reports success	Rely on `gui_screenshot`; confirm window_id is correct
`conflict`	That window already has a task; first `delete_task`
`window_not_found`	Window closed or ID expired; re-run `gui_list_windows`
`no visible windows` (process alive)	Stage Manager activated making the window uncapturable: `open -a` to activate → immediately re-enumerate; if not manifested, activate again (on 27, first activation may not stick; max 2 times); see "Invisible Window Trap" (experimentally confirmed on both 15 and 27.0.1)
Screenshot shows an abnormally small window / screen edge	Target window is in Stage Manager thumbnail state (27 re-check finding): first `open -a` to activate and restore full size, then capture; see "Invisible Window Trap" 27 re-check subsection items 1 and 3
State is `call_user`	Human intervention needed; relay `reasoning` to the user; do not call `continue_task`

Platform Support

macOS: Full support. Authorization process in references/authorization-macos.md
Windows: Not yet tested; current version does not promise support. When encountering a Windows environment, honestly inform the user of this limitation; do not attempt to apply the macOS permission model

Resources

references/authorization-macos.md — Full macOS authorization process, including tested pitfalls (permission not effective after checkbox, Quit & Reopen dialog, etc.)
tools/axctl — AX fallback tool (source tools/axctl.swift in the same directory; run sh tools/install.sh before first use), usage in the "AX Fallback Layer" section

GreenWich-art/agent-skills

What is agent-skills?

Ask in your favorite AI

Documentation