UI Explorer MCP

Example Use Cases

• 1

  Inspect and traverse the UI element hierarchy of the active screen or specific screen regions to locate controls by type, text, or visibility.

• 2

  Capture annotated screenshots showing UI element boundaries and hierarchy for verification, reporting, or UI analysis.

• 3

  Automate interactions by clicking at precise coordinates, sending keyboard input, or recording and replaying UI workflows as macros.

Description

This MCP server scans and analyzes UI hierarchies on screen, captures screenshots with highlighted UI elements, and can drive mouse clicks and keyboard input. It includes a standalone macro recorder/player for recording UI workflows (with screenshots, context, timing) that can run independently of the MCP server. The tool is intended for Windows and integrates with MCP clients (examples shown for Roo and Claude Desktop) or can be run directly via uvx. It requires Python 3.10+ and common automation libraries like PyAutoGUI, PyWinAuto, and Pillow.