Computer use is a capability that lets a Large language model operate a standard graphical computer interface directly, viewing screen content and issuing mouse and keyboard actions, rather than interacting with software only through dedicated APIs written specifically for it. Where ordinary Tool use (function calling) requires a developer to expose each capability as a discrete function call, computer use lets a model act on any software that has a visual interface, including websites, desktop applications, and operating system controls, in the same way a human user would, by looking at what is on screen and clicking, typing, or scrolling in response.
Anthropic introduced the technique publicly in October 2024 as a feature of its Claude (AI model family) models, describing it as letting the model take screenshots, move a cursor, click buttons, and type text within a controlled virtual environment. The capability depends on the model being able to interpret images of a screen and translate that understanding into coordinate-based actions, which made it a natural extension of Multimodal AI and Vision-language model capabilities rather than a purely text-based technique. Other labs, including Google DeepMind and OpenAI, released comparable browser- and computer-operating agent features through 2024 and 2025 as the approach became an established category within the broader agentic wave.
How it works
A computer-use loop typically follows a perceive-decide-act cycle: the model is given a screenshot of the current screen state along with the user's overall goal, it decides on a single next action, such as clicking a specific coordinate, typing a string, or pressing a key, that action is executed in the environment, and a new screenshot is captured and returned to the model's Context window for the next decision. This cycle repeats until the model determines the task is complete or a limit is reached. Because it operates through the same visual and input surface a human uses, computer use does not require an application to expose an API at all, which lets it work with legacy software, internal enterprise tools, and websites with no public integration.
Applications
Computer use has been applied to browser automation for research and data entry, testing software by simulating a real user's clicks and inputs, and automating repetitive office tasks such as filling out forms across systems that were never designed to be scripted. It is often discussed as an extension of the broader AI agent category into environments beyond a defined toolset, and it can be combined with more conventional tool calling within the same agent, sometimes wired through a shared interface such as the Model Context Protocol, calling a dedicated API when one exists and falling back to direct interface control when it does not.
Limitations and risks
Early computer-use systems were noticeably slower and less reliable than API-based tool use, since each action requires a full round trip through image interpretation before the next step can be taken, and models could misclick, misread small text, or become confused by unfamiliar interface layouts. Anthropic and other labs described the capability as experimental at launch and recommended running it in sandboxed environments given the risk of unintended actions. The safety implications are also more severe than for narrow tool use, since a model operating a full computer interface has, in principle, the same range of possible actions available to it that a human user would, including actions with financial, security, or irreversible consequences, which is one reason computer-use agents are considered a heightened target for Prompt injection attacks and are typically deployed with guardrails, human oversight, and restricted execution environments. As of 2025, computer use remained a fast-moving but still maturing category, with accuracy and reliability improving substantially across successive model generations but still lagging behind purpose-built API integrations for tasks where a dedicated API exists.