iwishwp/AgentOS

AgentOS Desktop Operations Manual (Generic for Any Agent)

This manual is self-contained: as long as your framework can mount stdio MCP servers (Goose / Claude Code / Z Code / Cursor, etc.), by mounting according to §1 you gain all capabilities below. All entries have been tested on real hardware (Fedora + niri Wayland, end-to-end evidence as of 2026-10 in the repository at docs/capabilities.md).

0. Mental Model: Four Layers of Capability, Priority Descending

1. L0 System Layer — Windows/workspaces/launch/files/notifications/clipboard (niri/desktop/system/files extensions): Most stable; use first.
2. L1 Accessibility Tree — Click by name and read/write text inside GUI applications (a11y extension, AT-SPI2): No focus stealing, no user disruption. Works end-to-end on GTK3/GTK4/Qt (the Qt global switch and Chromium's toolkit-accessibility are system defaults as of 2026-10-18); Chromium tree reading and text input are possible (page clicks go through L2); Java not installed (path in capabilities.md appendix B); pure X11 apps (xterm class) have no tree, can only use L3.
3. L2 Browser CDP — All web page operations (browser extension, agent-dedicated headless Chromium).
4. L3 Visual/Input Fallback — grim screenshots (reliable), keyboard injection (scripts/kb-inject.sh: wtype user-level as primary, text/modifiers reliable; control keys (Enter/Tab) fall back to ydotool requiring sudo), pointer injection (unreliable: drift under acceleration, only as a last resort).

When encountering "can't open / can't use" applications: first check if L1 prerequisites are met (a11y switches), then use L3 screenshots to see the actual state; do not blindly retry.

1. Mounting (One-Time)

goose (~/.config/goose/config.yaml, format tested with v1.54; name: field is required — if missing, the entire extension is silently discarded):

extensions:
  niri:
    type: stdio
    name: niri
    enabled: true
    cmd: /path/to/AgentOS/scripts/env/with-niri-env.sh
    args: []
    env_keys: []
    envs: {}
    timeout: 300          # in seconds
  # desktop/system/files/session/a11y/browser/cred are isomorphic,
  # cmd uses with-desktop-env.sh / with-system-env.sh / ... (eight launchers under scripts/env/)

Other frameworks (Claude Code / Z Code, etc., mcpServers JSON style):

{
  "mcpServers": {
    "niri":    { "command": "/path/to/AgentOS/scripts/env/with-niri-env.sh" },
    "desktop": { "command": "/path/to/AgentOS/scripts/env/with-desktop-env.sh" },
    "system":  { "command": "/path/to/AgentOS/scripts/env/with-system-env.sh" },
    "files":   { "command": "/path/to/AgentOS/scripts/env/with-files-env.sh" },
    "session": { "command": "/path/to/AgentOS/scripts/env/with-session-env.sh" },
    "a11y":    { "command": "/path/to/AgentOS/scripts/env/with-atspi-env.sh" },
    "browser": { "command": "/path/to/AgentOS/scripts/env/with-browser-env.sh" },
    "cred":    { "command": "/path/to/AgentOS/scripts/env/with-cred-env.sh" }
  }
}

Key points:

• Use the launcher scripts; do not point directly to node dist/index.js — the launcher handles discovering the niri IPC socket (path contains the compositor PID, different each session, e.g., /run/user/1000/niri.wayland-1.2762.sock) and populates Wayland/dbus environment variables (XDG_RUNTIME_DIR/WAYLAND_DISPLAY/DBUS_SESSION_BUS_ADDRESS), so that tools work when spawned from SSH/service environments.
• To test a launcher manually: timeout 2 scripts/env/with-niri-env.sh </dev/null; exit 0 means it's working.
• Prerequisites: Node ≥ 20, repository has been built (npm install && npm run build), running under the user that owns the niri graphical session.

2. Tool Reference (Grouped by Extension; Tool Name = <extension>__<tool>)

niri — Windows/Workspaces/Launch (L0)

Tool	Purpose	Key Parameters
list_windows	List all windows (id/app_id/title/workspace/focus)	—
list_workspaces	List workspaces (idx (index) and id are two different things; for move operations use index or name)	—
list_outputs	List displays	—
focus_window	Move focus to foreground (switches user view; use with caution)	id
focus_workspace	Switch workspace (switches user view)	reference: "3" or name
move_window_to_workspace	Move window (without stealing focus; use for organization)	id? (default = focused window), reference
close_window	Close window	id?
open_app	Launch an application on the user's desktop	command: array, e.g., ["ghostty"]

a11y — GUI Application Operations (L1, AT-SPI2)

Tool	Purpose	Key Parameters
a11y_list_apps	List applications registered on the accessibility bus	—
a11y_dump_tree	Dump the accessibility tree of an application (find button names / structure)	app
a11y_click	Click by accessible name (no focus stealing, no user disruption)	app, buttonName
a11y_read_texts	Read all non-empty text from an application (screens, results, labels)	app
a11y_set_text	Write text to a text field by node name	app, match (substring of name; default * for first), text

Tested discipline: Address by name, not by tree index (tree structure is rebuilt across AT connections); GTK4 trees only become complete after an AT client registers; for multiple identical instances, the bridge resolves to the first one registered — first read text to identify the correct instance by state, close extra instances if needed. Chromium exceptions: the tree builds lazily (the bridge triggers it automatically; the first dump may take 1-2 seconds — this is normal); for writing text fields, use the "grabFocus + kb-inject type" combination (Chromium entries do not accept setTextContents); button clicks on pages do not work (doAction fires but onclick does not run); all page operations should go through the browser extension.

browser — Web Page Operations (L2, Agent-Dedicated Headless Chromium)

Tool	Purpose	Key Parameters
browser_open	Navigate (first call automatically launches the browser)	url
browser_eval	Execute JavaScript on the page; use for form filling, clicking, reading content	JS code
browser_screenshot	Take a screenshot (browser is headless; vision depends entirely on this)	—

Standard form-filling action: after el.value = "...", you must call el.dispatchEvent(new Event("input", {bubbles: true})), otherwise modern frameworks won't receive the input. If a CAPTCHA is encountered: stop and inform the user; never automatically bypass.

files — File Sandbox (L0, root = ~/Agent/, all paths are relative)

Tool	Purpose	Key Parameters
read_file / write_file	Read/write text files (automatically create parent directories)	path, content
list_dir / create_dir	List directory / create directory	path
move / delete	Move / Delete (goes to .trash for recovery; do not use shell rm)	source/dest, path
find_by_name	Find file by name	pattern
download	Download from http(s) into the sandbox (≤50MB / 60s; do not use shell curl)	url, path

desktop — Presentation/Notifications/Clipboard/Confirmation (L0)

Tool	Purpose	Key Parameters
open	Open an artifact: URL or file; for office documents you must specify app: "libreoffice" (whitelist; system MIME associations are unreliable)	target, app?
notify	Desktop notification (three levels; see §3)	summary (required), body, urgency, action?
confirm	Ask the user for a decision (T1 shows four banners: Allow / Allow Session / Deny / Never; 30s no response = Deny)	summary (do not put sensitive content), tier? (t1/t2, affects wording)
clipboard_read / clipboard_write	Read/write clipboard	—

session — The Aquarium (Agent-Dedicated Nested niri Session; All GUI Work Done Here)

Tool	Purpose
agent_session_start / agent_session_stop / agent_session_status	Nested session lifecycle (a watchable window appears on the desktop)
agent_open_app	Launch an application inside the aquarium (e.g., gnome-calculator, chromium)
agent_list_windows	List windows inside the aquarium
agent_screenshot	Take a directed screenshot of the aquarium's interior (using grim)

system — System Queries (L0 read-only, fully permitted)

system_status / list_services / service_status / journal / network_info / list_packages / package_info / package_install (T2 strong confirmation: shows polkit password dialog; requires user presence; if headless, it will always time out after 300s — do not call during unattended operation).

cred — Credentials (Agent Uses References, Not Plaintext)

cred_list (available credentials and their references/env variable names) · cred_autofill (Channel A: automatic page filling; plaintext discarded immediately) · cred_exec (Channel B: whitelisted execution with output masked) · cred_inject (Channel C: short-TTL environment variable injection; entire set revoked at expiry). Any operation requiring a password/token must go through cred only; never ask the user for plaintext.

3. Behavioral Conventions (Disciplines Verified Through Testing)

1. All artifacts go into ~/Agent/, organized under downloads/ inbox/ reports/ notes/ canvas/ screenshots/; placement automatically adds them to the artifacts index (http://127.0.0.1:7788/artifacts); when a user asks "where's my last report" → open that page or report the relative path.
2. Presentation priority: Dashboards/reports/interactive content → write as a self-contained single-file HTML into canvas/<name>/index.html, then open to open http://127.0.0.1:7788/<name>/; office documents → open with app: "libreoffice"; web pages → browser extension.
3. Three notification levels: Completion → notify with urgency=low + action ({kind:"open_path", path:"reports/x.fodt"} or {kind:"open_url", url:...}; clicking the notification navigates directly to the artifact); failure / needs attention ASAP → urgency=critical + one-sentence conclusion + cause suggestion; when user decision is needed → do not use notify; use confirm (notify has no selection semantics). Always notify async results; never complete silently.
4. Confirmation and boundaries: Do not touch windows the user is currently using unless explicitly requested (first use list_windows to identify the target); when encountering policy rejections (isError with "policy rejected" message), inform the user honestly; do not try to bypass; perform GUI operations inside the aquarium to avoid stealing the user's focus.
5. Credentials only via cred-mcp references (the discipline stated after the cred table in §2).

4. Environment Pitfalls Quick Reference (All Encountered in Testing)

Pitfall	Symptom	Solution
niri IPC socket path changes every session	niri msg cannot connect	Use the launcher (automatically discovers NIRI_SOCKET); do not hardcode
Chromium keychain locks network services	Browser hangs on launch	Launch with --password-store=basic (the browser extension already includes this; if you start chromium manually, remember to add it)
soffice single-instance lock	When LibreOffice is already open, opening another document may result in no window / hang	First check soffice process state; use app:"libreoffice" with open for launcher semantics; after an abnormal exit, a "Document Recovery" window may pop up (tested 2026-10-18); when listing windows for confirmation, do not mistake it for the target document window
LibreOffice MIME associations unreliable	Fedora identifies fodt as text/xml and opens it in an editor	open must include app:"libreoffice"
Qt applications do not appear on the a11y bus	a11y_list_apps cannot see them	System-wide enablement has been applied (QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1 in /etc/environment, persists after reboot); if still not visible, the app started before the environment was set, or it's not an official Qt binding; restart the application
Chromium/Electron a11y tree is empty	Dump produces no content	① Launch with --force-renderer-accessibility (the browser extension already includes it by default); ② toolkit-accessibility must be true before the application starts (already globally set; changing afterward has no effect; must restart the application); ③ tree builds lazily; dump once to let the bridge trigger it (see §2 a11y discipline)
Keyboard injection selection	Need to type / use keyboard shortcuts	scripts/kb-inject.sh: for text and modifier keys use wtype (user-level); control keys like Enter/Tab do not work with wtype on GTK; use ydotool (sudo) for those; Chinese input may drop the first character; after typing, verify by reading back from the a11y tree
Aquarium initially shows the niri hotkey overlay	Screenshots are obscured by the "Important Hotkeys" overlay	Inject Esc to dismiss; the agent sandbox cannot connect to the ydotoold socket (root 600); use niri__open_app to execute the injection in the real desktop environment
ydotool pointer drift	Absolute positioning is inaccurate (libinput acceleration ≈0.5)	Pointer is not reliable; keyboard injection is reliable; note the socket path difference between /tmp/.ydotool_socket and $XDG_RUNTIME_DIR
GTK4 tree initially incomplete	Dump shows only a dozen nodes	Call a11y_list_apps/dump once to let the AT client register; the dump will then be complete (from 16 to 105 nodes, tested)
Headless T1/T2 confirmation	Automatically rejects after 30s/300s if no one responds	Expected behavior: write operations require user presence; for testing, you can respond programmatically via org.agentos.Confirm1.Respond
Package installation blocked by packagekitd lock	dnf occasionally fails	Stop systemctl stop packagekit before installation (the T2 helper already handles this)

5. Escalation Path (Try in Order When a Target Application Cannot Be Opened)

L0 (system layer; always try first) → L1 (a11y: GTK3/4/Qt already reachable by default; check Chromium flags and toolkit-accessibility order; for third-party Electron, add flags per capabilities.md appendix B) → L2 (web content must go here) → L3 fallback (grim screenshot + kb-inject keyboard; pointer is unreliable). Types still not openable: pure X11 apps (xterm class, no tree; L3 is the end), Java (requires JAB; path in capabilities.md appendix B), compositor own UI (only input injection) — see the repository docs/capabilities.md for details.