Computer-Use Agents Explained: AI That Clicks, Types, and Browses for You

On the short desktop tasks of the OSWorld benchmark, leading computer use agents now score above the people tested in 2024, yet on long, realistic office workflows the best of them still fail most tasks they are given. These agents operate a computer through its screen the way a person does, and since October 2024 Anthropic, Google, OpenAI and Microsoft have all announced one.

That uneven record points to a clear first use: short jobs whose result you can check, such as filling in a web form from a clean list, copying figures between programs or collecting the same facts from several sites. Paying, sending, deleting and signing in should stay with a person, because any page the agent reads can try to give it orders.

Agents in general are covered in how AI agents plan and act through tools, and the wider skills in the guide to AI literacy for staff in any role. What follows is limited to agents that work through a screen.

What is a computer-use agent?

A computer-use agent is an AI agentAI agent: AI software that pursues a goal by taking steps on its own: it keeps track of what has happened, calls tools such as email or a browser, and repeats until the goal is met or someone stops it. Because it acts rather than only answers, its mistakes can have direct effects.Full entry in the glossary whose tools are a screen, a pointer and a keyboard. Anthropic, which said its October 2024 release made it the first frontier AI model to offer computer use in public beta, described it as directing the model to use computers the way people do, by looking at a screen, moving a cursor, clicking buttons and typing text.¹

Most software that connects one system to another does it through an interface built for programs, called an API. A screen agent needs none. If a person can use the app, the agent can try, which is why Microsoft pitched computer use in its Copilot Studio as a way to handle tasks when there is no API to connect to the system directly.²

Suppose a purchasing team is stuck with a supplier portal from 2009 that has no API to plug into, and someone copies order numbers from a spreadsheet into its forms every Friday. A computer-use agent could do that copying through the same screens. The same reach that makes it useful also makes it risky: an agent that can click anything a person can click has the same access that person has, so treat its permissions as a person’s permissions.

How does AI computer use work?

The agent runs a short loop. It takes a screenshot, a language model compares that picture with the goal and picks one action, the action happens, and a new screenshot shows whether it worked. Anthropic explained that its model looks at screenshots of what is visible, then counts how many pixels it must move the cursor to click in the right place.³

1. Look: take a screenshot of the screen as it is now
2. Decide: the model compares the picture with the goal and picks one next action
3. Act: move the pointer, click, scroll or type, on whatever the screen shows
4. Check: take a new screenshot to see whether the action worked, then go round again

The padlock: before a step that pays, sends, deletes or signs in, the loop should stop and wait for a person.

Counting pixels sounds trivial, yet Anthropic called getting it right critical. Working from a picture, the model has to find the field, estimate where it is and hit it. Anthropic also listed the limits of working from still images: the agent can miss a notification that appears and vanishes between two screenshots, and some ordinary moves, such as dragging and zooming, were beyond it at launch.³

On a screen, a pop-up asking about cookies, a moved button or a slow page each costs a person a second. For the agent, each one is a new picture to interpret, and a wrong reading sends the next click to the wrong place. When you watch a demonstration, watch for these moments, because they are where runs go off course. Long, exact click sequences on cluttered pages are the weakest ground; tidy forms and clear pages are the strongest.

Which computer-use agents have been released since late 2024?

Four large developers launched screen agents within six months, in different forms. The table lists them in release order with no ranking; names, prices and availability have changed since and keep changing, so check each developer’s current page before relying on it.

Where the agent runs matters more than whose model it uses. An agent in its own virtual computer starts with nothing but what you give it. An agent inside your everyday browser works with whatever you are already signed in to.

What can AI browser agents do well?

Browser agents do best on repetitive, low-stakes work whose result you can compare with the original: copying data into forms, collecting the same facts from several websites into one table, and checking that a list of pages loads. Microsoft’s own examples were data entry, gathering market data from online sources and moving invoice details into accounting systems.²

A recruiter who compares the advertised salary bands on ten job boards has a good first task. The agent visits each site, copies the band into a table, and the recruiter spot-checks three rows. If a row is wrong, nothing has been sent or spent, and the source page is one click away.

Expect friction on the open web. The team behind the OSWorld benchmark, refreshing its tests in 2025, found sites blocking automated agents, showing CAPTCHA challenges or changing their layout, all of which broke tasks that had worked before.⁶ Read-only tasks on sites your team uses often are the gentlest start.

Each task on the right moves money, messages, data or access out of your hands, and none can be undone with a click. Developers have built some of these stops in already. Google’s Mariner asks for final confirmation before purchases, and Anthropic’s Chrome agent asks before publishing, purchasing or sharing personal data.⁴⁷ Leave those confirmations switched on, even in modes that offer to skip them.

How computer use agents score against people on OSWorld

OSWorld, a benchmark released in April 2024 by researchers led by the University of Hong Kong, set 369 real tasks on real desktop and web apps and checked each result with a script. People completed about 72 percent of the tasks, and the best AI agent about 12 percent.⁸

The tasks were ordinary: change a browser’s default search engine, fill in a spreadsheet column, carry work across two programs. The human testers were computer science students who had basic software skills but had not seen the tasks or the software before, so 72 percent is a rough stand-in for a capable newcomer. The authors found the agents struggled to turn screenshots into precise click positions, repeated the same action, and were thrown by unexpected windows.⁸

The gap closed fast on these short tasks. By mid-2026, the team behind the next version reported that leading agents scored around 80 percent on OSWorld-Verified, the team’s 2025 repaired version of the original tasks.⁶⁹ That is above the 2024 human figure, though the people were scored on the earlier task set. Read that as progress on short, self-contained jobs. A vendor quoting an OSWorld score is quoting this kind of task, so ask how long the tasks behind any number were.

Myth

A high OSWorld score means an agent can run a whole office workflow.

Fact

The original OSWorld tasks are short and self-contained. On longer workflows built by the same team, leading agents completed only a small share.

Long work tasks are where agents still fall short

Agents lose ground as tasks get longer and messier. OSWorld 2.0, a 2026 benchmark from the same research group, uses workflows that mostly take a skilled person more than an hour, and leading agents completed only a small share of them.⁹

The study

Moderate evidence

OSWorld 2.0: long, realistic workflows (Yuan et al., 2026)

The 108 tasks span research, finance, administration, health care and creative work, and take a skilled person a median of about 1.6 hours. Under strict pass-or-fail scoring, the best agent tested completed about one in five, while earning about half of the available credit for partial steps. On tasks longer than about 2 hours 40 minutes, none of the agents plotted completed any.⁹

The authors say the failures were rarely about clicking. Agents lost track of constraints, missed information that arrived mid-task, guessed instead of asking the user, and skipped checking their own work.⁹ In one published example, an agent building a purchase-order sheet noticed a late approval message in a team chat, then submitted a sheet that left out approved requests, and its final check confirmed only the list it had already made. That is a preprintpreprint: A research paper posted publicly before it has been peer reviewed or published in a journal; in economics and finance the same stage is usually called a working paper. Its findings may change, or fail to hold up, by the time it is published.Full entry in the glossary built on recreated websites, and the team includes researchers from AI companies, so treat the exact rates as a snapshot from mid-2026.

The partial-credit figure matters most to a team planning real use. An agent that does half of a long job and stops quietly still leaves the other half to someone. Split long work into short stages a person checks, and judge a trial by whether each stage was right.

Are computer-use agents safe to use at work?

Not by default. The main risk is prompt injection: text on a web page, email or document that the agent reads as an instruction. Because a screen agent reads everything in front of it, any page it visits can try to redirect it, and inside your browser it acts with your logged-in access.

Screen agents widen the reach of that attack. In August 2025, researchers at Brave, which makes a rival browser, showed that a comment hidden on a Reddit page could make Perplexity’s Comet browser, asked only to summarize the page, fetch the user’s email address and a one-time login code and send them out. Brave noted that the browser’s usual walls between websites do nothing here, because the AI operates with the user’s full privileges across signed-in sessions. Perplexity issued fixes, which Brave later said were incomplete.¹⁰

Developers’ own tests suggest defenses reduce the risk without removing it. Anthropic reported that, in its own red-team tests of its Chrome agent, attacks succeeded in about 24 percent of attempts without new safeguards and about 11 percent with them.⁷ A 2025 benchmark preprint from researchers at Meta, WASP, found that simple hand-written injections partly diverted agents in up to 86 percent of cases, though agents rarely completed the attacker’s full goal. The authors called this security by incompetence and tied it partly to the agents’ limited abilities, so the margin could shrink as agents get more capable.¹¹

prompt injection attacks will remain a residual risk, and cannot be fully mitigated with a product or appliance

The UK’s cyber security agency says the best hope is to reduce the likelihood or impact of attacks through careful design, build and operation; other national cyber security agencies publish their own guidance, so check yours.¹² In practice, the safest agent is one that cannot reach much: no saved passwords, no payment details, no open sessions it does not need.

Questions to ask IT before you use one

Ask before you install anything. A screen agent sees what you see and acts as you, so an unapproved one is a case of shadow AI, the use of AI tools nobody signed off, with more reach than a chatbot. Some security teams block AI browsers outright. In a December 2025 advisory titled Cybersecurity Must Block AI Browsers for Now, the analyst firm Gartner argued they are too risky for most organizations.¹³

Gartner’s concerns, as The Register reported them, were data on open tabs going to the AI provider, agents misled by hidden instructions or their own reasoning errors, and stolen credentials if an agent wanders onto a phishing site. The analysts also imagined staff handing mandatory security training to an agent.¹³ Each concern turns into a question you can ask.

Before you use a computer-use agent at work

• Is this agent on our approved tools list?
• Does it run in its own virtual computer or in my everyday browser with my logins?
• Which sites and apps can it reach, and can IT limit them?
• Which actions make it stop and ask me first?
• Where do its screenshots go, and are they kept or used for training?
• Is there a log of every step, and how do I stop a run?

If the answers are unclear, run the agent only on public websites, signed out, with nothing sensitive open in other tabs.

The bottom line

Computer-use agents can now finish short, clear screen tasks, and they still struggle with long work and with hostile text on the pages they read. Use one where you can check every result and nothing irreversible happens without you: read-only research and form filling first. Before your first run, take the six questions above to IT and keep the confirmation prompts switched on for payments, sending, deleting and signing in.