基于视觉的桌面自动化，使用 Midscene 驱动。通过自然语言控制本地桌面（macOS、Windows、Linux）或通过 RDP 远程 Windows 桌面。全程仅依赖截图操作——无需 DOM 或无障碍标签。可交互屏幕上所有可见元素，不受技术栈限制。
该技能利用 Midscene.js 实现视觉驱动的桌面自动化，支持通过自然语言命令控制本地桌面（macOS、Windows、Linux）或通过 RDP 控制远程 Windows 桌面。其核心机制是仅依赖截图进行交互，无需 DOM 或可访问性标签，因此能够操作屏幕上任何可见元素，无论底层技术栈如何。这对于浏览器无法承载的桌面原生应用（如 Electron、Qt、原生 macOS/Windows/Linux 应用）以及远程 Windows 主机的自动化尤其有用。支持的触发短语包括：打开应用、按下按键、桌面、电脑、点击屏幕、输入文本、截取桌面截图、启动应用程序、切换窗口、桌面自动化、控制电脑、鼠标点击、键盘快捷键、屏幕捕获、在屏幕上查找、读取屏幕、验证窗口、关闭应用、测试 Electron 应用、RDP、远程桌面、Windows 服务器、通过 RDP 连接。注意：在本地模式下，它会接管用户真实的鼠标和键盘，因此对于 Web 应用程序，应优先使用“浏览器自动化”技能。使用时请谨慎，确保仅在合适场景下调用。
npx skills add https://github.com/web-infra-dev/midscene-skills/tree/main/skills/desktop-computer-automationInstalled? Explore more 生产力与协作 skills: steipete/gemini, steipete/gh-issues, steipete/skill-creator · View all 6 →
基于视觉的桌面自动化，使用 Midscene 驱动。通过自然语言控制你的本地桌面（macOS、Windows、Linux）或通过 RDP 控制远程 Windows 桌面。全程仅依赖截图操作——无需 DOM 或无障碍标签。可交互屏幕上所有可见元素，不受技术栈限制。⚠️ 在本地模式下，它会接管用户真实的鼠标和键盘。对于 Web 应用，请优先使用“浏览器自动化”。仅将此技能用于无法在浏览器中运行的桌面原生应用（Electron、Qt、原生 macOS/Windows/Linux），或用于通过 RDP 驱动远程 Windows 主机。支持的触发短语：打开应用、按下按键、桌面、电脑、点击屏幕、输入文本、截取桌面截图、启动应用程序、切换窗口、桌面自动化、控制电脑、鼠标点击、键盘快捷键、屏幕捕获、在屏幕上查找、读取屏幕、验证窗口、关闭应用、测试 Electron 应用、rdp、远程桌面、windows 服务器、通过 rdp 连接 由 Midscene.js (https://midscenejs.com) 驱动。
This repo contains 5 individual skills — each has its own dedicated page.
Vision-driven Android device automation using Midscene. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control Android devices with natural language commands via ADB. Perform taps, swipes, text input, app launches, screenshots, and more. Trigger keywords: android, phone, mobile app, tap, swipe, install app, open app on phone, android device, mobile automation, adb, launch app, mobile screen, test android app, verify mobile app, QA on phone, check the app on android, test on device, see if the app works on phone, end-to-end test on android, visual verification on mobile Powered by Midscene.js (https://midscenejs.com)
Vision-driven browser automation using Midscene. Operates from screenshots — no DOM or accessibility labels needed. Runs in headless Puppeteer — does NOT take over the user's mouse or keyboard. Also supports CDP mode and Bridge mode to connect to an existing Chrome. Use this skill when the user wants to: - Browse, navigate, or open web pages - Scrape, extract, or collect data from websites - Fill out forms, click buttons, or interact with web elements - Verify, validate, test, or QA frontend UI behavior - Take screenshots of web pages - Automate multi-step web workflows - Test what was just built, see if it works in browser - Connect to Chrome via CDP, DevTools Protocol, or remote debugging - Connect to user's Chrome browser, control my browser, operate my Chrome Powered by Midscene.js (https://midscenejs.com)
Vision-driven HarmonyOS NEXT device automation using Midscene. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control HarmonyOS devices with natural language commands via HDC. Perform taps, swipes, text input, app launches, screenshots, and more. Trigger keywords: harmony, harmonyos, 鸿蒙, hdc, huawei device, harmony app, harmony automation, harmony phone, harmony tablet, test harmony app, verify on harmonyos, QA on 鸿蒙, check the app on harmony, test on huawei device, see if the app works on harmony, end-to-end test on harmonyos, visual verification on 鸿蒙 Powered by Midscene.js (https://midscenejs.com)
Vision-driven iOS device automation using Midscene CLI. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control iOS devices with natural language commands via WebDriverAgent. Triggers: ios, iphone, ipad, ios app, tap on iphone, swipe, mobile app ios, ios device, ios testing, iphone automation, ipad automation, ios screen, ios navigate, test ios app, verify on iphone, QA on ipad, check the app on ios, test on ios device, see if the app works on iphone, end-to-end test on ios, visual verification on ios Powered by Midscene.js (https://midscenejs.com)
Enhances Vitest with Midscene for AI-powered UI testing across Web (Playwright), Android (ADB), and iOS (WDA). Scaffolds new projects, converts existing projects, and creates/updates/debugs/runs E2E tests using natural-language UI interactions. Triggers: write test, add test, create test, update test, fix test, debug test, run test, e2e test, midscene test, new project, convert project, init project, 写测试, 加测试, 创建测试, 更新测试, 修复测试, 调试测试, 运行测试, 新建工程, 转化工程.
Gemini CLI one-shot prompts, summaries, generation, skills, hooks, MCP, or Gemma routing.
Fetch GitHub issues, select candidates, spawn background fix agents, open PRs, and optionally process PR review comments.
Create, edit, audit, tidy, validate, or restructure AgentSkills and SKILL.md files.
Search, install, update, sync, or publish agent skills with the ClawHub CLI and registry.
Delegate coding work to Codex, Claude Code, or OpenCode as background workers; not simple edits or read-only code lookup.
List, configure, authenticate, call, and inspect MCP servers/tools with mcporter over HTTP or stdio.