Vision and computer use
Let 8gent look at images and your screen with a local vision model, and operate your Mac's mouse and keyboard.
8gent can see (images, PDFs, your screen) and, on a Mac, act on the desktop (click, type, scroll). The workspace shows these as the body indicators in the activity panel: hands, eyes and handeyes.
Vision: a local model looks for you
Ask it to look at something:
> Describe the screenshot at ./bug.png and say which element is misaligned
> Read the table on page 3 of ./invoice.pdf
> What does the error dialog on my screen say?The tools behind this: read_image, describe_image, read_pdf, read_pdf_page, and the screen tools eyes_see, eyes_find, eyes_describe, eyes_wait_for.
Set up the vision model
Descriptions come from a local vision model. qwen2.5vl:7b (about 6 GB) is the one we use:
ollama pull qwen2.5vl:7bThen, inside 8gent, in your project:
/vision model qwen2.5vl:7bName it explicitly
Auto-discovery and the built-in hints look for qwen2.5-vl, which is not Ollama's tag, so an installed qwen2.5vl:7b is not picked up on its own. /vision model qwen2.5vl:7b fixes it for the project. Tracked in 8gent-code #3217.
| Command | What it does |
|---|---|
/vision | Current vision settings and the vision models found |
/vision model <name> | Use this model for descriptions |
/vision ocr <name> | Use this model for text extraction (auto to discover) |
/vision on, /vision off | Turn image understanding on or off |
/vision local | Local models only (the default) |
/vision pull | Suggested models to pull |
Vision settings are saved per project, under vision in .8gent/config.json.
Computer use: hands on your Mac
On macOS, 8gent can drive the desktop:
> Open System Settings and turn on Do Not Disturb
> Take a screenshot and click the Save button| Tools | What they do |
|---|---|
desktop_screenshot, desktop_click, desktop_type, desktop_press, desktop_scroll, desktop_drag, desktop_hover | Screen capture, mouse and keyboard |
desktop_windows, desktop_clipboard | Window list, clipboard |
run_computer_task | A multi-step task on the desktop, looking and acting in a loop |
handeyes_locate_and_click, handeyes_click_and_verify, handeyes_type_and_confirm | Find a thing on screen, act on it, and check the result |
Set it up
- Grant your terminal Screen Recording and Accessibility in System Settings, Privacy and Security.
8gent permissionschecks both and8gent permissions --openopens the right panes. - For drag and hover, install
cliclick:brew install cliclick. Without it, screenshot, typing, key presses, scrolling and the clipboard still work. - For the screen-reading tools (
eyes_*), 8gent uses a small bundled accessibility bridge at~/.8gent/bin/8gent-ax-bridge. It is built when you install from source on a Mac.
Every desktop action goes through the same safety checks as any other tool. /hands, /eyes and /handeyes toggle the indicators in the activity panel.