Community研究與資料分析github.com

web-infra-dev/browser-automation

使用 Midscene 進行基於視覺的瀏覽器自動化。透過螢幕截圖操作——無需 DOM 或無障礙標籤。在無頭 Puppeteer 中運行——不會接管使用者的滑鼠或鍵盤。也支援 CDP 模式和 Bridge 模式以連接到現有 Chrome。

browser-automation 是什麼?

此技能提供基於 Midscene.js 的視覺驅動瀏覽器自動化。它透過分析螢幕截圖來執行操作,完全無需依賴 DOM 樹或無障礙標籤,使其在不同網站上更具復原力。核心引擎在無頭 Puppeteer 瀏覽器中運行,不會干擾使用者對滑鼠或鍵盤的控制。除了標準的無頭模式外,它還支援 Chrome DevTools Protocol (CDP) 模式和 Bridge 模式,用於連接使用者現有的 Chrome 瀏覽器實例,實現如遠端除錯等進階案例。此技能非常適合網頁抓取、資料收集、表單填寫、UI 驗證與測試、多步驟工作流程自動化,以及驗證剛剛建置的內容在瀏覽器中是否正常運作。

相容平台~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/web-infra-dev/midscene-skills/tree/main/skills/browser-automation

Installed? Explore more 研究與資料分析 skills: affaan-m/uspto-database, affaan-m/scholar-evaluation, affaan-m/literature-review · View all 6 →

在你喜歡的 AI 中提問

開啟一個已預先載入此 Agent Skill 的新對話。

說明文件

browser-automation 是做什麼的?

使用 Midscene 進行基於視覺的瀏覽器自動化。透過螢幕截圖操作——無需 DOM 或無障礙標籤。在無頭 Puppeteer 中運行——不會接管使用者的滑鼠或鍵盤。也支援 CDP 模式和 Bridge 模式以連接到現有 Chrome。在以下情況下使用此技能:

  • 瀏覽、導航或開啟網頁
  • 從網站抓取、提取或收集資料
  • 填寫表單、點擊按鈕或與網頁元素互動
  • 驗證、確認、測試或 QA 前端 UI 行為
  • 對網頁截取螢幕截圖
  • 自動化多步驟網頁工作流程
  • 測試剛建置的內容,在瀏覽器中檢視是否正常運作
  • 透過 CDP、DevTools Protocol 或遠端除錯連接到 Chrome
  • 連接到使用者的 Chrome 瀏覽器、控制我的瀏覽器、操作我的 Chrome

由 Midscene.js (https://midscenejs.com) 提供支援

Individual skills in this repo

This repo contains 5 individual skills — each has its own dedicated page.

web-infra-dev/android-device-automation

Vision-driven Android device automation using Midscene. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control Android devices with natural language commands via ADB. Perform taps, swipes, text input, app launches, screenshots, and more. Trigger keywords: android, phone, mobile app, tap, swipe, install app, open app on phone, android device, mobile automation, adb, launch app, mobile screen, test android app, verify mobile app, QA on phone, check the app on android, test on device, see if the app works on phone, end-to-end test on android, visual verification on mobile Powered by Midscene.js (https://midscenejs.com)

web-infra-dev/desktop-computer-automation

Vision-driven desktop automation using Midscene. Control your local desktop (macOS, Windows, Linux) or a remote Windows desktop over RDP with natural language commands. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. ⚠️ In local mode this takes over the user's real mouse and keyboard. For web apps, prefer "Browser Automation" instead. Only use this for desktop-native apps (Electron, Qt, native macOS/Windows/Linux) that cannot run in a browser, or for driving a remote Windows host via RDP. Triggers: open app, press key, desktop, computer, click on screen, type text, screenshot desktop, launch application, switch window, desktop automation, control computer, mouse click, keyboard shortcut, screen capture, find on screen, read screen, verify window, close app, test Electron app, rdp, remote desktop, windows server, connect via rdp Powered by Midscene.js (https://midscenejs.com)

web-infra-dev/harmonyos-device-automation

Vision-driven HarmonyOS NEXT device automation using Midscene. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control HarmonyOS devices with natural language commands via HDC. Perform taps, swipes, text input, app launches, screenshots, and more. Trigger keywords: harmony, harmonyos, 鸿蒙, hdc, huawei device, harmony app, harmony automation, harmony phone, harmony tablet, test harmony app, verify on harmonyos, QA on 鸿蒙, check the app on harmony, test on huawei device, see if the app works on harmony, end-to-end test on harmonyos, visual verification on 鸿蒙 Powered by Midscene.js (https://midscenejs.com)

web-infra-dev/ios-device-automation

Vision-driven iOS device automation using Midscene CLI. Operates entirely from screenshots — no DOM or accessibility labels required. Can interact with all visible elements on screen regardless of technology stack. Control iOS devices with natural language commands via WebDriverAgent. Triggers: ios, iphone, ipad, ios app, tap on iphone, swipe, mobile app ios, ios device, ios testing, iphone automation, ipad automation, ios screen, ios navigate, test ios app, verify on iphone, QA on ipad, check the app on ios, test on ios device, see if the app works on iphone, end-to-end test on ios, visual verification on ios Powered by Midscene.js (https://midscenejs.com)

web-infra-dev/vitest-midscene-e2e

Enhances Vitest with Midscene for AI-powered UI testing across Web (Playwright), Android (ADB), and iOS (WDA). Scaffolds new projects, converts existing projects, and creates/updates/debugs/runs E2E tests using natural-language UI interactions. Triggers: write test, add test, create test, update test, fix test, debug test, run test, e2e test, midscene test, new project, convert project, init project, 写测试, 加测试, 创建测试, 更新测试, 修复测试, 调试测试, 运行测试, 新建工程, 转化工程.

相關技能