使用 Midscene 進行視覺驅動的桌面自動化。透過自然語言指令控制本地桌面（macOS、Windows、Linux）或透過 RDP 控制遠端 Windows 桌面。完全依靠螢幕截圖運作，無需 DOM 或無障礙標籤。
這項技能提供基於視覺的桌面自動化能力，驅動引擎為 Midscene.js。它允許使用者透過自然語言指令控制本地桌面（支援 macOS、Windows、Linux）或透過 RDP 連線的遠端 Windows 主機。與傳統自動化工具不同，它完全依賴螢幕截圖進行操作定位與互動，因此無需依賴 DOM 結構、無障礙標籤或特定技術棧。這使得它能與螢幕上所有可見元素互動，無論該應用程式是使用 Electron、Qt、原生系統框架還是其他技術構建。技能會觸發真實的滑鼠和鍵盤操作，因此在本地模式下它會接管使用者的實際輸入裝置。主要使用場景包括測試桌面原生應用程式、自動化 RDP 工作階段，以及操作無法在瀏覽器內執行的應用程式。對於網頁應用，建議優先使用「瀏覽器自動化」技能。此技能支援多種觸發詞，如「開啟應用程式」、「按下按鍵」、「點擊螢幕」、「輸入文字」、「截取桌面」、「啟動應用程式」、「切換視窗」、「桌面自動化」、「控制電腦」、「滑鼠點擊」、「鍵盤快捷鍵」、「螢幕擷取」、「螢幕搜尋」、「讀取螢幕」、「驗證視窗」、「關閉應用程式」、「測試 Electron 應用程式」、「rdp」、「遠端桌面」、「windows 伺服器」、「透過 rdp 連線」。
npx skills add https://github.com/web-infra-dev/midscene-skills/tree/main/skills/desktop-computer-automationInstalled? Explore more 生產力與協作 skills: steipete/gemini, steipete/gh-issues, steipete/skill-creator · View all 6 →
使用 Midscene 進行視覺驅動的桌面自動化。透過自然語言指令控制本地桌面（macOS、Windows、Linux）或透過 RDP 控制遠端 Windows 桌面。完全依靠螢幕截圖運作，無需 DOM 或無障礙標籤。能與螢幕上所有可見元素互動，不受技術棧限制。⚠️ 在本地模式下，這會接管使用者的真實滑鼠和鍵盤。對於網頁應用，建議優先使用「瀏覽器自動化」。僅在處理無法在瀏覽器中執行的桌面原生應用程式（Electron、Qt、原生 macOS/Windows/Linux）或驅動透過 RDP 連線的遠端 Windows 主機時才使用此技能。觸發詞：開啟應用程式、按下按鍵、桌面、電腦、點擊螢幕、輸入文字、截取桌面、啟動應用程式、切換視窗、桌面自動化、控制電腦、滑鼠點擊、鍵盤快捷鍵、螢幕擷取、螢幕搜尋、讀取螢幕、驗證視窗、關閉應用程式、測試 Electron 應用程式、rdp、遠端桌面、windows 伺服器、透過 rdp 連線。支援 Midscene.js (https://midscenejs.com)
This repo contains 5 individual skills — each has its own dedicated page.
Vision-driven Android device automation using Midscene. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control Android devices with natural language commands via ADB. Perform taps, swipes, text input, app launches, screenshots, and more. Trigger keywords: android, phone, mobile app, tap, swipe, install app, open app on phone, android device, mobile automation, adb, launch app, mobile screen, test android app, verify mobile app, QA on phone, check the app on android, test on device, see if the app works on phone, end-to-end test on android, visual verification on mobile Powered by Midscene.js (https://midscenejs.com)
Vision-driven browser automation using Midscene. Operates from screenshots — no DOM or accessibility labels needed. Runs in headless Puppeteer — does NOT take over the user's mouse or keyboard. Also supports CDP mode and Bridge mode to connect to an existing Chrome. Use this skill when the user wants to: - Browse, navigate, or open web pages - Scrape, extract, or collect data from websites - Fill out forms, click buttons, or interact with web elements - Verify, validate, test, or QA frontend UI behavior - Take screenshots of web pages - Automate multi-step web workflows - Test what was just built, see if it works in browser - Connect to Chrome via CDP, DevTools Protocol, or remote debugging - Connect to user's Chrome browser, control my browser, operate my Chrome Powered by Midscene.js (https://midscenejs.com)
Vision-driven HarmonyOS NEXT device automation using Midscene. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control HarmonyOS devices with natural language commands via HDC. Perform taps, swipes, text input, app launches, screenshots, and more. Trigger keywords: harmony, harmonyos, 鸿蒙, hdc, huawei device, harmony app, harmony automation, harmony phone, harmony tablet, test harmony app, verify on harmonyos, QA on 鸿蒙, check the app on harmony, test on huawei device, see if the app works on harmony, end-to-end test on harmonyos, visual verification on 鸿蒙 Powered by Midscene.js (https://midscenejs.com)
Vision-driven iOS device automation using Midscene CLI. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control iOS devices with natural language commands via WebDriverAgent. Triggers: ios, iphone, ipad, ios app, tap on iphone, swipe, mobile app ios, ios device, ios testing, iphone automation, ipad automation, ios screen, ios navigate, test ios app, verify on iphone, QA on ipad, check the app on ios, test on ios device, see if the app works on iphone, end-to-end test on ios, visual verification on ios Powered by Midscene.js (https://midscenejs.com)
Enhances Vitest with Midscene for AI-powered UI testing across Web (Playwright), Android (ADB), and iOS (WDA). Scaffolds new projects, converts existing projects, and creates/updates/debugs/runs E2E tests using natural-language UI interactions. Triggers: write test, add test, create test, update test, fix test, debug test, run test, e2e test, midscene test, new project, convert project, init project, 写测试, 加测试, 创建测试, 更新测试, 修复测试, 调试测试, 运行测试, 新建工程, 转化工程.
Gemini CLI one-shot prompts, summaries, generation, skills, hooks, MCP, or Gemma routing.
Fetch GitHub issues, select candidates, spawn background fix agents, open PRs, and optionally process PR review comments.
Create, edit, audit, tidy, validate, or restructure AgentSkills and SKILL.md files.
Search, install, update, sync, or publish agent skills with the ClawHub CLI and registry.
Delegate coding work to Codex, Claude Code, or OpenCode as background workers; not simple edits or read-only code lookup.
List, configure, authenticate, call, and inspect MCP servers/tools with mcporter over HTTP or stdio.