Anthropic’s AI Learns to Navigate Computers Like Humans

AI startup Anthropic has released an updated version of its Claude 3.5 Sonnet model, which can interact with computers like a human—moving the cursor, clicking buttons, and typing text.

Introducing an upgraded Claude 3.5 Sonnet, and a new model, Claude 3.5 Haiku. We’re also introducing a new capability in beta: computer use.

Developers can now direct Claude to use computers the way people do—by looking at a screen, moving a cursor, clicking, and typing text. pic.twitter.com/ZlywNPVIJP

— Anthropic (@AnthropicAI) October 22, 2024

“Instead of creating special tools to help Claude perform specific tasks, we are teaching it general computer skills, allowing it to use a wide range of standard solutions and programs developed for humans,” noted Anthropic.

Developers can leverage this capability to automate repetitive processes, create and test software, and perform other tasks. An API has been created to enable Claude to perceive and interact with computer interfaces.

The feature is experimental, so errors may occur. It is being tested by Asana, Canva, Cognition, DoorDash, Replit, and The Browser Company.

“Some actions that people perform effortlessly—scrolling, dragging, zooming—pose certain challenges for Claude,” warned representatives of the AI startup.

Also introduced is the new AI model Claude 3.5 Haiku, set to be released at the end of October. It matches the performance of the company’s previous flagship neural network, Claude 3 Opus.

The company reported improvements in Claude 3.5 Sonnet’s programming capabilities, citing customer feedback. GitLab noted the AI’s stronger reasoning, making it an “ideal choice” for supporting multi-stage software development processes. Cognition and The Browser Company also gave positive reviews.

Programming was also highlighted as a strength for the Claude 3.5 Haiku model. It scores 40.6% on SWE-bench Verified, surpassing many competitors.

Earlier in October, OpenAI’s GPT-4o model achieved the highest score in programming with Solidity, outperforming o1-preview, o1-mini, and competitors.