Apps on your Mac
Your agent can open an app on your Mac, read what it shows, click and type in it, wait for what it expects to show up, and be told what each step changed.
Besides the web, an agent can work in the apps on your Mac as you would: it opens an app, reads what its windows show, clicks and types in it. It tries to do it behind your windows, so your pointer, your keyboard and the app you are in stay yours. For this your Mac must have allowed NoBreiniers to control the computer (Accessibility) and to see the screen (Screen Recording).
Wait for what it expects, without asking the model again
After a click that opens a window, loads a list or saves a file, the agent does not need to wait a bit and look again. It can say what must show and what must be gone, and NoBreiniers looks at the app again and again, every fraction of a second, until that is so.
- What must show and what must be gone are texts. A text matches something on the screen that holds all of its words.
- It waits 5 seconds unless the agent asks for another time, and never more than 1 minute in one go.
- While it waits, no model call is made: the model is asked once, when the wait ends.
- When the wait ends without success, the answer says which text never showed, or which one is still there, so the agent knows what to do next.
- If you press Stop, the wait stops and is never reported as met.
It is told what each step changed
After a click, after typing and after a key, the agent gets one more line saying what appeared or went from the app's elements, or that no change was seen. So it does not have to look at the whole window again to know if its step did anything.
- Nothing is said about typing into a password field, or into a field your Mac cannot identify.
- Nothing is said about NoBreiniers' own window or about your own browsers.
A whole chain of steps in one go
Instead of one step at a time, the agent can send up to 30 steps together: open the app, type, choose File › Export as PDF…, wait for the save sheet, type the name, press Return.
- Every step is checked whole before anything runs.
- A step can say what must show before it acts, or before the next one goes. It waits as described above.
- The chain stops, and says in plain words what was done and what was not, when the page is not as expected, when another chat used your Mac in between, or when a step would land in another app.
- A menu item is pressed by its name with its whole path (menu, then item). The approval card says «… wants to choose … in a menu on your Mac» and names that path, so you see exactly what it will choose and in which app.
- A chain can do nothing that a single step cannot: each step goes through the same approvals and rules. What a chain types stays hidden until it ends, however it ends, and a value put into a password field is judged as that field.
