Feature Request: computer/browser use

[Korayem]

Would be epic to use Ollama Vision and push a screenshot of the current app. Use cases are immense:

• Logic Pro: how can I EQ guitar track
• Excel: what formula should i write in cell A1 to SUM rows 3-15
• etc...

Then this opens the door for onit to also take action and click inside the app to accomplish tasks like browser/computer use solutions

[timlenardo]

@Korayem - yes, this is the dream! This is a big part of Onit's strategy. We are building the ideal computer-use interface, before good computer-use models are generally available, with full knowledge that they're coming soon. My takeaway from Cursor, Loveable, Devin, is that the 'sidebar' interface is the winning design pattern for interacting with agents. You want a sidebar where you can give instructions, follow up, and see a log of actions. In addition to the sidebar, you want most of the interface to display the application where the work is actually being done. In Cursor, it's the IDE. In Devin, its the IDE, Terminal, Browser, etc.

So, what does this look like for a computer-use agent? You'll need a sidebar that lives on your desktop, independent of any specific application, that can interact with any application on your computer. That's Onit. We don't offer this yet because the models aren't good enough. However, we're positioning ourselves to create a killer product when the models inevitably arrive.

[Korayem]

@timlenardo I just tried this MCP which takes screenshots on my mac https://github.com/steipete/Peekaboo and it works with local models! I could ask "what song is playing on spotify" or "OCR current open PDF" and get an answer

MCP offers 3 functions:

1. list: Lists various system items on macOS, providing situational awareness.
2. image: Captures macOS screen content and optionally analyzes it. Targets can be entire screen, specific app window, or all windows of an app (via app_target). Supports foreground/background capture. Output via file path or inline Base64 data (format: "data"). If a question is provided, image is analyzed by an Al model (auto-selected from PEEKABOO_A|_PROVIDERS). Window shadows/frames excluded.
3. analyze: Analyzes a pre-existing image file from the local filesystem using a configured Al model.

If onit can add MCP support, it could feed visual info into onit that willl be very helpful when using onit with Excel or Powerpoint and using local vlm model like gemma or ollava would be killer

computer use can come later because as you said, that's the vision for onit

[timlenardo]

Hey @Korayem - very cool, thanks for sharing!

To make sure I'm understanding correctly: you have the Peekaboo MCP locally and are looking for a way to connect Onit to it? Is that correct?

@jrswanson has been working on adding local tool calling. He has an ongoing PR for adding local tools (#288) - the initial demo creates calendar events, but it'll be extendable to support local MCP servers as well.