How do AI agents control the screen, mouse and keyboard?

Through a helper program that sits between the model and the operating system. The helper captures the screen and sends the image to the model; the model answers with an action like "click at 640, 410" or "type this text"; the helper turns that into real pointer and key events using the system's input functions, then captures the screen again. Everything the agent does is a sequence of those small exchanges.

Coordinates and scaling

A model reasons about the image it was given, which is often smaller than your display. The helper scales its coordinates back to real pixels before clicking. When that mapping is off, clicks land a little to the side of the target, which is a common cause of an agent "missing" a button. Anthropic's toolset includes a zoom action so the model can look closer at a crowded region before it commits.

What each operating system allows

On macOS, an app must be granted Accessibility to send clicks and keys and Screen Recording to capture the display. On Windows, Microsoft's input function lets a program inject input only into applications at an equal or lower integrity level, so an ordinary agent cannot click inside a window running as administrator. On Linux desktops that use the XDG desktop portals, screen capture and remote input go through portal sessions, and the portal typically shows the user a dialog to choose what to share.

Short answers

How does superbot move the mouse on my computer?

Through the permissions your system grants it. superbot's docs say that on a Mac the first run of computer use asks for Accessibility to click and type and Screen Recording to take screenshots, while on Windows there is no system permission to grant and the first screen action shows an Allow card in the chat instead.

What are the best open source frameworks for computer use agents?

Agent S from Simular operates real desktop and web apps on macOS, Windows and Linux. Microsoft's UFO is an MIT-licensed research project centred on the Windows desktop. Cua provides open-source drivers, sandboxes across operating systems and a benchmark. Browser Use is an open-source Python library for agents that work in a browser.

Why use screenshots instead of reading the app directly?

Because a screenshot works for every app. Some agents also read structured data, such as a web page's elements, when they can, but the picture is the one interface every program shares.

Can an agent type into a window running as administrator on Windows?

Not unless the agent itself runs at that level, because Windows blocks injected input to higher integrity windows.