> ## Documentation Index
> Fetch the complete documentation index at: https://panopticon-cli.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Computer use

> Opt-in, off-by-default desktop control (accessibility tree, clicks, keys, screenshots) for agents on Linux, macOS, or Windows through WSL2

# Computer use

Overdeck agents can read and operate a local desktop app (the accessibility tree, clicks,
typing, keys, scrolling, dragging, and screenshots) through `pan computer`. It is
**off by default** and works on **Linux, macOS (see [macOS](#macos)), or Windows through WSL2**.

## What it does

`pan computer` runs a vendored copy of Orca's Linux computer-use helper (`native/computer-use-linux/runtime.py`,
MIT, see [its README](https://github.com/eltmon/overdeck/blob/main/native/computer-use-linux/README.md)).
The helper reads the AT-SPI accessibility tree, takes window screenshots, and sends input
through AT-SPI actions, XTest and `xdotool`. Agents use it to read a window's state, find
elements by their accessibility tree index, and act on them the way a person would, when no
programmatic path (shell, files, git, HTTP, an existing CLI) can reach the same app.

When Overdeck runs inside WSL2, `pan computer` drives the Windows desktop instead, with Orca's
Windows helper; see [Windows (WSL2)](#windows-wsl2).

## Turn it on

### From Settings

Open **Settings → Computer Use** in the dashboard (or run **Open Computer Use settings** from the
command palette) and press **Set up and turn on**. One click does the whole setup on the machine
running Overdeck:

1. **Installs missing system packages.** Overdeck runs your package manager through `pkexec`, so your
   desktop asks for your password once. Overdeck never sees the password and never runs
   `apt-get update`. If apt reports "Unable to locate package", run `sudo apt-get update`, then press
   **Try again**. The package names are fixed on the server:

   | Needed for | `apt-get` | `dnf` | `pacman` |
   | - | - | - | - |
   | Accessibility bindings | `python3-gi gir1.2-atspi-2.0` | `python3-gobject at-spi2-core` | `python-gobject at-spi2-core` |
   | Mouse and keyboard input (X11) | `xdotool` | `xdotool` | `xdotool` |
   | Clipboard | `xclip` | `xclip` | `xclip` |
   | Screenshots (Wayland) | `grim` | `grim` | `grim` |

   When the desktop has no password-prompt agent, Overdeck opens a terminal window that runs the
   `sudo` install command instead; enter your password there, then press **Try again**. With
   `zypper`, `apk` or no detected package manager, setup shows the command to run yourself and stops
   before turning anything on.
2. **Turns on GNOME accessibility** (`toolkit-accessibility`), without a password. Apps opened
   before this need a restart to show their contents.
3. **Turns `computer_use.enabled` on** in `~/.overdeck/config.yaml`.
4. **Tests the helper** with a handshake.

When nothing is missing, the button reads **Turn on Computer Use** and needs no password. Once it is
on, the **Allow agents to use the desktop** switch turns it off again without uninstalling anything.
**Take back control** releases the desktop from the agent that holds it. The collapsed **What
Overdeck checked** list shows every readiness check, whether or not the feature is on.

### In config.yaml

Add a `computer_use:` block to `~/.overdeck/config.yaml`:

```yaml theme={null}
computer_use:
  enabled: true
```

This is a **global-layer-only** setting: a project's `.pan.yaml` can never set it, so a work
agent committing to its own feature branch can't switch on desktop control for the operator. With
it off, every `pan computer` command except `capabilities` and `release` refuses with `disabled`.

### Dashboard routes

The Settings page uses five admin-only routes: `GET /api/computer-use/status`,
`POST /api/computer-use/setup`, `PUT /api/computer-use/settings`,
`POST /api/computer-use/release` and, on macOS only, `POST /api/computer-use/open-permission`
(opens one System Settings permission pane). No scoped access token can call them, so an agent can't turn
computer use on for itself.

## macOS

Computer use works on macOS 14 (Sonoma) or newer when you run Overdeck from `npx @overdeck/core`
or a global npm install. It drives native apps through **Overdeck Computer Use**, a small helper
app adapted from Orca's macOS helper (MIT, see
[its README](https://github.com/eltmon/overdeck/blob/main/native/computer-use-macos/README.md)).
The helper ships inside the npm package, so you don't need Xcode. The Overdeck desktop app does
not bundle the helper yet.

### Why macOS asks for two permissions

macOS never lets an app grant itself these permissions, so you switch them on yourself, once:

* **Accessibility** lets the helper read app windows' accessibility trees and click, type and
  scroll in them.
* **Screen Recording** lets the helper take window screenshots. Without it the tree still works,
  but screenshots fail.

macOS ties both permissions to the helper app itself, not to Terminal or Overdeck, which is why
the helper is installed as its own app.

### Setup

Open **Settings → Computer Use** and press **Set up and turn on**:

1. **Installs the helper** to `~/.overdeck/computer-use/macos/Overdeck Computer Use.app`. No
   password is needed. The app must stay at that path, because macOS ties the permissions to it.
2. **Opens the two System Settings pages** (Privacy & Security → Accessibility and Screen
   Recording). Switch on **Overdeck Computer Use** in each, then press **Continue** in Overdeck.
3. **Turns `computer_use.enabled` on** and **tests the helper** with a handshake.

Once it is on, the card lists **Accessibility** and **Screen Recording** with their status. A
permission that is not allowed has an **Open settings** button that opens its System Settings
page again. Overdeck re-checks when you come back to the dashboard window.

The helper runs as one background process per user, shared by every `pan computer` call. It
listens on `/tmp/overdeck-cu-<uid>/agent.sock`, only accepts callers running as your user with
the token in `~/.overdeck/computer-use/macos/agent.token`, and exits after 10 minutes without a
request. On macOS, actions run in the background by default: the target app is not brought
forward, your frontmost window keeps focus and the pointer does not move. An agent that passes
`--restore-window` brings the app forward and uses your screen, keyboard focus and pointer for
that action; the skill tells agents to do this only when a background action can't work, and to
tell you first.

### Updates and signing

Overdeck builds the helper in CI. Without an Apple Developer ID certificate it is **ad-hoc
signed**, and macOS ties the permissions to that exact build. So after an Overdeck update that
changes the helper, macOS asks you to allow the updated helper again: the card shows **Needs
attention**, and **Fix setup** reinstalls the helper and reopens the two System Settings pages,
where you switch Overdeck Computer Use off and on again. A helper signed with a Developer ID
certificate keeps its permissions across updates.

For maintainers: the release workflow signs with a Developer ID certificate when
`MACOS_CERTIFICATE` exists, and signs ad-hoc otherwise. It notarizes the helper only when it is
Developer ID signed and the App Store Connect API key secrets exist; an ad-hoc build skips
notarization, because Apple notarizes only Developer ID-signed apps:

| Secret | Holds |
| - | - |
| `MACOS_CERTIFICATE` | The Developer ID Application certificate as a base64 `.p12` |
| `MACOS_CERTIFICATE_PWD` | The `.p12` password |
| `APPLE_API_KEY` | The App Store Connect API key (`.p8`), base64 |
| `APPLE_API_KEY_ID` | The API key's ID |
| `APPLE_API_ISSUER_ID` | The API key's issuer ID |

A source checkout without the bundled zip builds the helper on first setup with
`npm run build:computer-macos`, which needs the Xcode Command Line Tools
(`xcode-select --install`).

## Requirements

On macOS, see [macOS](#macos). On Linux:

* A Linux desktop with a logged-in session (X11 or Wayland). A headless host or a linked machine
  with no desktop session gets `no_desktop_session`.
* System packages: `python3-gi`, `gir1.2-atspi-2.0`, `xdotool` (X11 hotkeys/clicks), optionally
  `xclip` or `xsel` (clipboard paste; `wl-copy` on Wayland), and `grim` on Wayland (screenshots).
* GNOME accessibility on: `gsettings set org.gnome.desktop.interface toolkit-accessibility true`.
* Under WSL2, none of the above applies: see [Windows (WSL2)](#windows-wsl2).

## Windows (WSL2)

Overdeck on Windows runs inside WSL2. There, `pan computer` and **Settings → Computer Use** drive
the **Windows desktop**, never WSLg windows. They use a vendored copy of Orca's Windows UI
Automation helper (`native/computer-use-windows/runtime.ps1`, MIT, see
[its README](https://github.com/eltmon/overdeck/blob/main/native/computer-use-windows/README.md)).
The commands, flags and output are the same as on Linux.

### Requirements

* WSL2 with Windows interop on (the default). WSL1 is not supported.
* Windows 10 or 11.
* Windows PowerShell 5.1, which ships with Windows. Nothing is installed on Windows, so the
  Settings button reads **Turn on Computer Use** and asks for no password.

### What runs where

* Before each command, Overdeck copies the helper to `%LOCALAPPDATA%\Overdeck\computer-use\` on
  the Windows side, when the copy is missing or differs. Operation files go to its `ops\` folder
  and are deleted after each call. The WSL path of that folder is cached in
  `~/.overdeck/computer-use/windows-helper-dir.txt`.
* Each command starts one `powershell.exe` with `-ExecutionPolicy Bypass`. The bypass applies to
  that one process only and needs no administrator rights. Each command takes 1 to 3 s, because the
  helper compiles its Windows API bindings on every start.
* A command that runs longer than 30 s is killed with `helper_timeout`. Killing it from WSL may
  leave the Windows `powershell.exe` running; the helper's own steps are short, so this is rare.

### Known limits

* `list-windows` returns only an app's main window (index 0). Wait for a dialog with
  `pan computer wait --text` instead.
* The helper is DPI-aware only while it captures screenshots, so clicks can land in the wrong
  place on setups that mix display scales.
* An app running as administrator refuses input with a `target_elevated:` message.
* A WSL terminal started as administrator runs the helper elevated too.
* A Group Policy that enforces signed scripts overrides `-ExecutionPolicy`; the helper then fails
  with a message asking the PC's administrator to allow local scripts.
* `hotkey` and `press-key` refuse Alt+F4, Win+L and Ctrl+Shift+Escape unless
  `--allow-system-shortcut` is passed.

## Commands

Every subcommand takes `--json`. Every action and `get-app-state` also take `--window-index <n>`,
`--restore-window` and `--no-screenshot`.

| Command | What it does |
| - | - |
| `capabilities` | Reports platform, opt-in and desktop-session readiness. Never refuses. |
| `list-apps` | Lists open apps with at least one window. |
| `list-windows --app <app>` | Lists an app's windows. |
| `get-app-state --app <app>` | Reads a window's accessibility tree and screenshot. |
| `wait --app <app>` | Polls until a `--text`, `--enabled` or `--window-title` condition holds. |
| `click --app <app>` | Clicks a cached element or a window-local coordinate. |
| `perform-secondary-action --app <app>` | Runs one of an element's named secondary accessibility actions. |
| `set-value --app <app>` | Sets an element's value directly through accessibility. |
| `type-text --app <app>` | Types text at an element or the current focus. |
| `press-key --app <app>` | Presses a single key. |
| `hotkey --app <app>` | Presses a key chord. |
| `paste-text --app <app>` | Pastes text at the current focus. |
| `scroll --app <app>` | Scrolls at an element or a coordinate. |
| `drag --app <app>` | Drags from one element or coordinate to another. |
| `release` | Releases the desktop lease. |

See the `pan-computer-use` skill for the full field reference and error-handling guidance.

## One agent at a time

A desktop lease, held in a file under [Where files go](#where-files-go), ensures only one holder
drives the desktop's input at a time. An input command claims the lease for 120 seconds, renewed
on every call; a holder without it gets `desktop_busy` naming the current holder and when the
lease expires. Read-only commands (`capabilities`, `list-apps`, `list-windows`, `wait`, and
`get-app-state` without `--restore-window`) never need the lease, so other agents can still look.
Run `pan computer release` when a GUI task is finished, or `pan computer release --force` to
clear a stuck lease held by someone else.

## Where files go

Everything lives under `~/.overdeck/computer-use/`:

| Path | Holds |
| - | - |
| `ops/` | Short-lived operation files passed to the helper (mode 0600, deleted after each call) |
| `desktop.lock` | The cross-process file lock that serializes every helper call |
| `lease.json` | The desktop lease: holder, pid, claimed-at and expiry |
| `snapshots/<holder>/<app>.json` | The cached element records behind `--element-index` |
| `screenshots/` | Captured PNGs (the 50 newest are kept) |

## Safety

* `hotkey` and `press-key` refuse a chord that would quit, lock, log out or switch virtual
  terminals (`alt+F4`, `ctrl+alt+F2`, `super+l`, `ctrl+q`, …) with `system_shortcut_blocked`,
  unless `--allow-system-shortcut` is passed. On macOS the refused chords are `cmd+q`,
  `cmd+shift+q`, `cmd+ctrl+q`, `cmd+alt+escape` and `cmd+shift+backspace`.
* `--text-stdin` and `--value-stdin` read sensitive text from stdin so it never lands in argv or
  shell history.
* The shipped skill tells agents never to push, submit, send, buy, delete, or change account
  settings through computer use unless the user explicitly asked for that action.

## Limits

* Wayland can't do hotkeys (`xdotool` needs X11); screenshots there need `grim`.
* No remote hosts: a linked machine with no logged-in desktop session always refuses.
* On some multi-monitor setups, AT-SPI can report a window's bounds at `(0,0)` instead of its
  real screen position. When that happens, its screenshot can show the wrong area of the
  desktop, and `click`/`scroll`/`drag` coordinates computed from that screenshot can miss the
  window entirely. Prefer `--element-index` actions over `--x`/`--y` coordinates when a window's
  reported bounds look suspicious, and verify a coordinate action's effect against a fresh
  `get-app-state` rather than trusting the screenshot alone.
* The helper is a verbatim vendored port of Orca's Linux computer-use bridge; see
  [`native/computer-use-linux/README.md`](https://github.com/eltmon/overdeck/blob/main/native/computer-use-linux/README.md)
  for how it's re-vendored.

## Troubleshooting

On macOS, Settings → Computer Use shows every check (macOS version, helper, Accessibility,
Screen Recording, handshake); `pan doctor` points there instead of probing. A
`permission_denied` error means a permission is off: open Settings → Computer Use and finish
setup.

On Linux, run `pan doctor`. With the feature off it prints no `Computer use` row at all. With it on, it
checks the AT-SPI Python bindings, the desktop session, `xdotool` on X11, a clipboard tool,
`grim` on Wayland, and GNOME accessibility, ending in a single `Ready (<session>, display <d>)`
row once the helper's handshake succeeds.

Under WSL2, `pan doctor` and the **What Overdeck checked** list show four checks:

| Check | When it fails | What to do |
| - | - | - |
| Windows interop | WSL can't run Windows programs | Enable interop in /etc/wsl.conf (\[interop] enabled=true), then run wsl --shutdown from Windows. |
| Windows PowerShell | `powershell.exe` was not found or failed to start | Check `appendWindowsPath` in `/etc/wsl.conf`; WSL normally adds the Windows `PATH`. |
| Helper folder | `%LOCALAPPDATA%\Overdeck\computer-use\` can't be written from WSL | Fix the folder's permissions on Windows; the check shows the error. |
| Helper test | The handshake failed | Read the message. An execution-policy message means a Group Policy blocks local scripts; ask the PC's administrator. |

If the CLI works but the dashboard's helper test fails with "Exec format error" or a vsock error,
the dashboard service may be missing the WSL interop socket. Overdeck sets `WSL_INTEROP` from
`/run/WSL/` when it's missing; report the error with your WSL version if it persists.

Still stuck? Ask on the [Overdeck Discord](https://discord.gg/ZpHE2ZTv5P) or file a [GitHub Issue](https://github.com/eltmon/overdeck/issues).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.