computer-use-mcp

The open, agent-agnostic version of macOS computer use. A single signed Swift binary that exposes your Mac as a standard MCP server. Point Claude Code, Cursor, Codex CLI, Gemini CLI, or your own agent at it and the agent can see and operate the apps on your Mac — in the background, without hijacking your cursor or stealing focus.

Status: pre-1.0, used in production by the authors. macOS only. Runs while the Mac is unlocked (the system is kept from idle-sleeping during active sessions; if the screen locks, mutating tools pause with a recoverable error until you unlock).

Table of contents

• Why it's different
• Install
• Quickstart per client
• How it works
• Architecture
• Tools
• What's new this cycle
• Requirements
• Safety
• Configuration
• Comparison
• Distribution notes
• Known limitations
• Development
• Contributing
• License

Why it's different

• Universal — works on every app. Accessibility-first for precision, with an automatic pixel-coordinate fallback (z-order hit-testing) for apps with poor or absent accessibility trees. The agent's own vision provides the grounding — no bundled ML model. Web apps included: Chromium/Electron web-content accessibility is enabled on demand, and structural wrapper nodes are collapsed so deeply nested page content actually reaches the agent.
• Background-safe. A layered input ladder (AX action → per-window event → per-PID event, with explicit global fallback for clicks) delivers actions to the target app without moving the real cursor or changing focus by default. You keep working while the agent works.
• Agent-agnostic. Standard MCP over stdio. Any compliant client connects with one line of config — no lock-in.
• Observable. A smooth self-drawn agent cursor (separate from your real pointer) glides to each target so you can watch what the agent does.
• Verified outcomes. Every mutating action re-reads the target and reports whether the effect actually happened — not just whether the call returned without throwing. See What's new this cycle.
• Multi-session safe. Sessions are thin shims over one shared engine daemon (spawned on demand, retired on version changes, self-reaping when idle), so any number of concurrent agents go through a single process that owns capture, accessibility, input, and the cursor — and per-app leases keep two agents from interleaving actions inside the same app.
• Reliable. Every action returns fresh app state (screenshot + accessibility tree). Element IDs retain exact live AX handles and fail stale if the app recreates or detaches them; destructive actions pass a confirmation policy.
• One native binary. Zero runtime dependencies, frictionless install.

Install

macOS 14 or newer. On first run the binary asks for Accessibility and Screen Recording permission (Input Monitoring is not required).

From GitHub Releases (recommended once published)

Heads-up: the first tagged release may not be published yet. If the Releases page is empty, use build from source below. These steps are correct as soon as the first release is cut.

Download the notarized .app (or standalone binary) from the latest release, move it to /Applications (or anywhere on your PATH), and confirm it runs:

computer-use-mcp version
computer-use-mcp doctor          # check Accessibility / Screen Recording grants

A notarized, stably-signed build matters here: macOS ties Accessibility and Screen Recording grants to a signing identity, so a notarized bundle keeps its permissions across upgrades (see Distribution notes).

Build from source

git clone https://github.com/minghinmatthewlam/computer-use-mcp.git
cd computer-use-mcp
swift build -c release
.build/release/computer-use-mcp version

The binary is .build/release/computer-use-mcp. Put it on your PATH, or point your MCP client's command at the absolute path. To build the stable-identity .app wrapper locally:

python3 scripts/build_app_bundle.py            # local .app wrapper
python3 scripts/deploy_app_bundle.py           # release build + install to ~/Applications

Homebrew (coming)

A Homebrew formula is drafted under packaging/homebrew/computer-use-mcp.rb as groundwork. It is not tapped or published yet; once the first release ships with a stable tarball, the intended flow is:

# Not live yet — planned:
brew install minghinmatthewlam/tap/computer-use-mcp

Quickstart per client

The server speaks MCP over stdio; every client launches it the same way — computer-use-mcp serve. Pick your client below.

claude mcp add computer-use -- computer-use-mcp serve

{
  "mcpServers": {
    "computer-use": { "command": "computer-use-mcp", "args": ["serve"] }
  }
}

[mcp_servers.computer-use]
command = "computer-use-mcp"
args = ["serve"]

{
  "mcpServers": {
    "computer-use": { "command": "computer-use-mcp", "args": ["serve"] }
  }
}

{
  "mcpServers": {
    "computer-use": { "command": "computer-use-mcp", "args": ["serve"] }
  }
}

How it works

Each MCP client spawns serve, a thin stdio shim; tool calls are forwarded to a shared engine daemon (one per user, spawned on demand over a unix socket) that owns accessibility, screen capture, input delivery, and the agent cursor. One engine process means concurrent agent sessions cannot collide on shared system services, and short per-app leases keep two sessions from interleaving actions inside the same app. Tool calls fail fast with structured DAEMON_UNAVAILABLE metadata if the daemon cannot be reached; they never fall back in-process. The daemon also owns privacy-filtered runtime metrics: events are buffered and flushed in bounded batches to rotating JSONL plus an aggregate summary.

Click interactions first resolve to an accessibility element and a screen point, then descend a delivery ladder, stopping at the first tier that works:

1. Accessibility action (AXPress, etc.) — precise, background, no event posted.
2. Per-window event — a windowNumber-routed event delivered to the target process, so the action lands without activating the app or moving the cursor.
3. Per-pid event — delivered to the process when no window id resolves.
4. Global cursor — opt-in last resort only (allow_global_cursor: true); it moves the real pointer, then restores it.

State is re-perceived after every action and returned to the caller, so the agent always acts on current ground truth. Element ids carry a snapshot generation, so reusing a stale id fails loudly instead of mis-clicking.

For the precise production contract across observation, dispatch, coordinate spaces, foreground/background guarantees, TCC requirements, stale snapshots, and failure recovery, see Modality Contract. For strict background focus/cursor behavior, see Background Control Contract.

Architecture

flowchart LR
  subgraph Clients["MCP clients"]
    C1["Claude Code"]
    C2["Cursor"]
    C3["Codex CLI"]
    C4["Gemini CLI / your agent"]
  end

  C1 & C2 & C3 & C4 -->|stdio MCP| SHIM["serve (thin stdio shim)"]
  SHIM -->|unix socket| DAEMON["shared engine daemon<br/>(one per user, per-app leases)"]

  subgraph ENGINE["Engine"]
    AX["Accessibility<br/>(perceive + AX actions)"]
    SCK["ScreenCaptureKit<br/>(background-safe capture)"]
    LADDER["Input ladder<br/>AX → per-window → per-pid → global"]
    OVERLAY["Agent cursor + status chip overlay"]
  end

  DAEMON --> AX & SCK & LADDER & OVERLAY
  AX & SCK & LADDER & OVERLAY -->|drive / observe| APPS["Target macOS apps<br/>(foreground or occluded)"]

A longer walkthrough of the process model, dispatch ladder, outcomes, and source map lives in docs/architecture-overview.html.

Tools

Perceive get_app_state (with scope_element_id/max_elements for huge windows, skeleton: true for a shallow overview of a large tree, and ocr: true for apps that draw their own UI) · list_apps · list_windows

Act click · type_text · press_key · scroll · drag · set_value · select_text · perform_secondary_action

System open_app · open_url · manage_window · read_clipboard · write_clipboard · health_report

Element actions use ids from get_app_state; coordinate-capable tools also accept raw screenshot coordinates. The intentionally small surface keeps perception, delivery, and verification in one inspectable path.

Action results return a reduced-resolution screenshot to keep the agent loop fast, and skip resending the element tree when the action changed nothing (existing ids stay valid). When the UI did change, results carry a compact diff of what changed, appeared, or disappeared — elements that survive a change keep their ids, so everything the agent holds stays valid. Pass include_screenshot: false for tree-only results, include_state: false for a bare confirmation (fastest), and call get_app_state whenever full-resolution pixels are needed.

What's new this cycle

Three capabilities landed on main this cycle; the README documents them so it stays truthful about current behavior.

Verified action outcomes

Mutating tools no longer report "success = the call didn't throw." Each action now does a read → act → re-read of the target and classifies whether the intended effect actually occurred, surfaced in a computer-use-mcp/outcome block in the result's _meta. The classification is one of four values:

isError is unchanged — a thrown exception still sets it, and everything else stays false — so agents that ignore _meta behave exactly as before. Agents that read the outcome block get an honest verdict plus a plain-language sentence in the text body for non-success results. The full design, including the per-tool false-success trap matrix, is in docs/outcome-contract.md.

Delivery fallback telemetry

The input ladder silently falls from tier to tier; results previously reported only the tier that finally landed the event. A computer-use-mcp/delivery block in _meta now carries delivery_tier, a fallback_reasons array explaining why each higher tier was skipped (e.g. axActionUnsupported, windowNumberUnresolved, eventBridgeFailed, globalCursorRequested), and ui_changed. This is pure telemetry — it does not change which tier is attempted — and it lets an agent reason about whether escalating the delivery tier is worth trying.

Skeleton overview + dense-collection viewport windowing on get_app_state

Two ways to keep large windows from blowing up the tree:

• skeleton: true returns a shallow overview: the outline recurses a few levels, then a deeper container is emitted with a children_count annotation instead of its subtree, and stays a drill target. Pass that container's id as scope_element_id for a full scoped re-query. Skeleton is the overview, scope_element_id is the drill-in, and max_elements still bounds either.
• Dense-collection viewport windowing (automatic): virtualized collections (lists, tables, outlines, grids) with many children are windowed to the on-screen slice — preferring the app's own visible-rows attributes, else a viewport-frame intersection — instead of a blind first-N prefix that ignores scroll position. Off-window items are summarised as a count on the container's line (never silently dropped), and materialised rows retain live-handle identity while the app keeps the same AX objects. Use scope_element_id to inspect a specific retained container with the full element budget.

Requirements

• macOS 14+
• Permissions granted on first run: Accessibility and Screen Recording (Input Monitoring is not required). Run computer-use-mcp health_report to inspect current identity/permission state, or computer-use-mcp doctor --prompt when you intentionally want macOS prompts.

Safety

The server gates risky actions itself (it does not trust the calling agent). Destructive/irreversible button clicks (Delete, Erase, Reset, …), typing into secure password fields, and actions against apps on a confirmation list return a recoverable Confirmation required: … error until the caller retries with "confirm": true.

Browser pages get their own gate: before acting in a known browser the server reads the current URL from the accessibility tree and applies the URL policy — url_deny patterns block the action outright (confirm does not override), url_confirm patterns (plus built-in payment-page defaults) require confirm per action. The server also yields to the human: when real hardware input was seen in the last second and the target app is the one the user is working in (or the action uses the global cursor), the call returns a recoverable error instead of interleaving with the user (see interference_idle_seconds).

See SECURITY.md for the TCC permission model, threat model, and private disclosure process.

Configuration

Every option is settable as an environment variable (COMPUTER_USE_MCP_<KEY>) or a key in ~/.config/computer-use-mcp.json (env wins):

Comparison

How computer-use-mcp relates to the closest projects. This is a factual snapshot; cells marked "Not public" mean the project is closed-source and the behavior isn't documented, not that it's absent.

The two independent open-source efforts above both converged on verified outcomes ("don't trust an AX success — re-read and confirm the effect"), which is the model computer-use-mcp adopts in docs/outcome-contract.md. computer-use-mcp's distinguishing bets are the standard MCP surface, the shared-daemon multi-session model, and the visible agent cursor.

Distribution notes

The binary needs Accessibility and Screen Recording permission, and macOS ties those grants to the host process that spawns the server (your terminal or agent app). A rebuilt binary keeps its grants; a different host needs its own. Use computer-use-mcp health_report --json to record the current executable, bundle id (if any), parent process, permission state, and daemon socket/secret paths without revealing daemon secrets. Add --probe-capture when you intentionally want a bounded ScreenCaptureKit/replayd responsiveness probe. For redistribution, codesign with a Developer ID and notarize (codesign --sign "Developer ID Application: ..." && xcrun notarytool submit ...) so TCC grants attach to a stable identity. See Permissions and app identity for the first productionization checklist.

Known limitations

• Background delivery uses macOS per-process event posting, which is app-dependent: a few apps that require real keyboard focus (e.g. some pro audio apps, secure input fields) may ignore background events. For clicks, use allow_global_cursor: true as an explicit fallback.
• Menu key-equivalents (e.g. cmd+a) are reliable when the app is the key window; some apps ignore them when targeted purely in the background.
• macOS only (the engine is built on Accessibility, ScreenCaptureKit, and CoreGraphics). The protocol/tool layer is OS-agnostic.

Development

swift build
swift test
.build/debug/computer-use-mcp serve                  # run the stdio MCP server
.build/debug/computer-use-mcp call get_app_state '{"app":"Calculator"}'   # drive one tool
.build/debug/computer-use-mcp health_report --json   # non-mutating diagnostics
.build/debug/computer-use-mcp health_report --probe-capture   # bounded capture-service probe
.build/debug/computer-use-mcp doctor                 # check permissions
python3 scripts/preflight.py                         # CI-safe release preflight
python3 scripts/build_app_bundle.py                  # local .app wrapper build
python3 scripts/deploy_app_bundle.py                 # release build + bundle + install to ~/Applications + daemon handover
python3 scripts/deploy_app_bundle.py --check         # exit 1 if the installed bundle is older than source/build
python3 scripts/preflight.py --use-app-bundle        # non-live checks through the .app executable
python3 scripts/e2e_demo.py                          # safe structured smoke artifact; no GUI mutation

Default CI covers the non-mutating path: package build, pure unit tests, and CLI version/help/health_report --json smoke checks. Live app-control checks are local-only because they require a logged-in macOS desktop plus Accessibility/Screen Recording permission and can operate real apps. See Testing and Preflight for the testing tiers, local command loop, and release preflight expectations. The architecture-level behavior target for these tiers is captured in Modality Contract.

Benchmark and smoke tiers

Use deterministic checks for hosted CI and live GUI checks only on a local Mac or a future self-hosted macOS runner with explicit permissions.

Live GUI mutation is opt-in. Use --live for local manual runs. Do not add the live tier to hosted CI — it depends on an unlocked macOS desktop, TCC permissions, Finder/TextEdit behavior, and user-visible app state. Before running the live tier, build the binary and make sure the spawning terminal has Accessibility and Screen Recording permission:

swift build
.build/debug/computer-use-mcp doctor --prompt
python3 scripts/e2e_demo.py --live

Contributing

Contributions welcome. See CONTRIBUTING.md for build/test conventions and PR expectations, SECURITY.md for the security model and private disclosure process, and CHANGELOG.md for release history.

License

MIT — see LICENSE.