Ambient Desktop Docs
Quickstart Download ambient.xyz
Capabilities

Local Visual Analysis

Your favorite text model might not see images. Ambient bridges that gap locally: it routes screenshots, images, and video frames to a MiniCPM-V vision model on your own machine and hands the agent structured observations back.

Developer Preview
Local-First Visual Analysis Visual analysis can route to a local MiniCPM-V runtime even when the selected text model lacks native vision. Local-First Visual Analysis Visual analysis can route to a local MiniCPM-V runtime even when the selected text model lacks native vision. captures pixelsanalyzes locallyreturns schemacontrols risk Vision bridge local runtimeschemarepair 1 Input ImageVideo frameScreen state 2 Local runtime MiniCPM-Vllama.cppSchema output 3 Agent Text modelTool resultFollow-up 4 Control RepairTimeoutEndpoint review
Visual analysis can route to a local MiniCPM-V runtime even when the selected text model lacks native vision.
Product screenshots

Settings

Settings search keeps provider, permission, and runtime configuration discoverable.

Ambient Desktop settings search.

Vision without leaving your machine

A coding agent needs to look at things — a screenshot of a failing UI, a diagram, a rendered page — but many strong text models have no native vision, and sending images to a cloud vision service is not always acceptable. Ambient solves this with a local vision bridge: when a task needs to understand pixels, it routes the image to a MiniCPM-V model running locally and returns structured observations the text model can act on.

How a look-at-this step works

  1. Capture

    An image, video frame, or screenshot enters as the visual input — for example, proof that a calculator actually rendered.

  2. Analyze locally

    MiniCPM-V runs on a local llama.cpp runtime and produces a structured description, rather than the image going to a cloud endpoint.

  3. Return to the agent

    The structured result flows back to the text model as a tool result, so it can reason about what it saw and take the next step.

  4. Repair if needed

    If the runtime is missing assets, times out, or returns invalid output, diagnostics flag it and guide a repair instead of failing silently.

Platform support

The local vision runtime ships with platform-appropriate acceleration. Heavier-accelerated platforms get it on by default; lighter ones are experimental.

macOS (Apple Silicon)

Metal-accelerated; the local vision runtime is enabled by default.

Linux (x64)

Vulkan/CUDA acceleration; enabled by default on capable machines.

Windows (x64)

Experimental and CPU-only today, so analysis is slower. Treat it as a preview path.

Safety and repair

  • Local runtime checks surface missing binaries, absent model assets, timeouts, invalid structured output, and memory pressure — with guidance to fix each.
  • Hosted or remote visual endpoints are a separate, higher-risk path that requires explicit security review before they are enabled.
  • Local visual analysis is a routing building block: it lets local-first operation include vision without forcing every screenshot to the cloud.

Keep going

Ambient NetworkLocal Model RoutingNot every task should leave your machine. Ambient can route work across local runtimes, cloud models, and specialized providers — keeping private, cheap, or latency-sensitive work local while reserving heavy reasoning for the cloud.Use AmbientProvider RoutingAmbient can prioritize and fall back through providers for search, fetch, browser, media, and model-adjacent work.Trust And SecuritySecurity ModelMost agent tools protect you by sandboxing the whole agent — which also blinds it. Ambient does the opposite: the agent works on your real desktop, while the genuinely risky capabilities are the things that get contained.