# Vercel Labs open-sources agent-desktop for cross-platform computer use

Digest AI · Agents & Tools · published 2026-10-31T10:10:28Z

Canonical: https://digestai.news/story/vercel-labs-open-sources-agent-desktop-for-cross-platform-computer-use

## Summary

Vercel Labs released **agent-desktop**, an open-source CLI tool for AI agents to interact with desktop apps on macOS, Windows, and Linux. The tool reads apps via OS accessibility APIs—no screenshots or vision models—and returns JSON snapshots of UI elements with unique refs. Version **1.0.2** launched on October 30, 2026, under the **Apache-2.0** license, targeting developers who want to control native apps programmatically. It supports **60 commands** (56 operational) and claims **78% to 96% fewer tokens** for dense apps like Slack or VS Code by using skeleton traversal. For example, Apple Maps’ token usage dropped from **3,811 to 273** per snapshot, and a 52-step task fell from **198,000 to 31,000 tokens**—figures self-reported by the team. The tool runs headless by default, avoiding pointer movement or focus theft, though a `--headed` flag allows manual input. It integrates with any shell-capable agent via Rust CLI or a C library, enabling Python, Swift, Go, Ruby, and Node.js hosts. Installation requires npm (`npm install -g agent-desktop`) and OS-specific permissions (e.g., macOS’s Accessibility API). Vercel Labs positions it as a **model-agnostic** alternative to pixel-based tools like Anthropic’s (which cost **1,000–1,800 tokens per screenshot**) or OpenAI’s click-action APIs. Limitations include apps without accessibility trees and no independent benchmarks yet. The project is documented on GitHub and skills.sh, with a C library for in-process calls.

## Key points

- Open-source CLI **agent-desktop** lets AI agents control desktop apps via OS accessibility APIs, not screenshots
- Self-reported **78–96% token savings** for apps like Slack or VS Code using skeleton traversal
- Runs on macOS, Windows, Linux (x64/ARM64), headless by default, with Rust CLI or C library support

## Why it matters

This tool lowers the barrier for building cross-platform agents that interact with native apps without vision models, reducing token costs and enabling headless automation for developers. It could accelerate agent-based workflows in enterprise or developer tools, though its accessibility limitations may exclude some apps.

## Sources

1. [Vercel Labs Open-Sources agent-desktop for Agent Computer Use](https://marktechpost.com/2026/10/31/vercel-labs-agent-desktop-computer-use) (MarkTechPost, 2026-10-31)

## Cite

Digest AI, "Vercel Labs open-sources agent-desktop for cross-platform computer use", 31 October 2026, https://digestai.news/story/vercel-labs-open-sources-agent-desktop-for-cross-platform-computer-use

---

Written by Digest AI's editorial model from the linked sources; the sources are the record. Headlines, digests and key points are written by Digest AI and may be quoted with a link to the story page. Linked articles belong to their publishers. Terms: https://digestai.news/terms#reuse
JSON: https://digestai.news/story/vercel-labs-open-sources-agent-desktop-for-cross-platform-computer-use.json