Desktop Control
Advanced desktop automation with pixel-perfect mouse control, keyboard input, screen capture, window management, and clipboard operations. Built on PyAutoGUI.
by
matagul
Desktop Control Overview
Desktop Control provides comprehensive desktop automation capabilities, allowing OpenClaw to interact with any application on your computer.
It offers pixel-perfect mouse control, fast keyboard input, screen capture with image recognition, window management, and clipboard operations with built-in safety features.
How It Works
The skill uses PyAutoGUI as the core engine and exposes a DesktopController class for automation.
Mouse moves, clicks, drags, scrolling, and keyboard events are routed through DesktopController.
Screenshots, pixel checks, and optional OpenCV image recognition help locate UI targets.
Failsafe corners, pause controls, and optional approval mode help prevent unsafe actions.
Key Features
Mouse Control
Absolute and relative positioning, smooth movement, left/right/middle/double click, drag-and-drop, and scrolling.
Keyboard Input
Fast text typing with configurable WPM, hotkeys, special keys, and hold/release control.
Screen Operations
Full-screen and region screenshots, pixel color detection, image recognition, and multi-monitor support.
Window Management
List windows, activate by title, read window info, and minimize/maximize control.
Clipboard Operations
Copy text to the system clipboard and read text back from it.
Safety Features
Failsafe corner abort, emergency pause, bounds checking, and action logging.
Requirements
- No VM required.
- Python packages: PyAutoGUI, Pillow, and PyGetWindow are required. OpenCV (opencv-python) is optional for image recognition.
- Works on macOS, Linux, and Windows; some features may require elevated privileges.
- A graphical desktop environment is required (not compatible with headless servers).
Use Cases
Form Filling
Automatically fill out web forms or desktop application fields with typed input.
Application Automation
Control any desktop application through mouse and keyboard simulation.
Screen Monitoring
Capture screenshots, detect specific UI elements, and respond to visual changes.
File Management
Drag and drop files between folders and multi-select with Ctrl+click.
Testing & QA
Automate manual GUI testing workflows across desktop applications.
How to Install
npx clawhub@latest install desktop-controlRun in your terminal, or click the Install button at the top of this page for one-click setup.
FAQ
Which Python packages are required?
PyAutoGUI, Pillow, and PyGetWindow are required. OpenCV (opencv-python) is optional for image recognition.
Does Desktop Control work on macOS, Linux, and Windows?
Yes. It supports macOS, Linux, and Windows, though some actions may need elevated privileges.
Can I run Desktop Control on a headless server?
No. A graphical desktop environment is required.
Is there a safety failsafe?
Yes. It includes a failsafe corner, pause controls, optional approval mode, and logging.
How do I enable image recognition?
Install OpenCV (opencv-python). It enables image matching in screen operations.