In May 2026, Reflex, the Python full-stack framework, published a benchmark that pours cold water on the computer-use hype. The team had the same Claude model, the same dataset, and the same admin-panel task solved two ways: once by a vision agent clicking through the browser, once by the model calling structured APIs. The gap was roughly 45x on input tokens — the vision path burned an average of 550,976 input tokens and about 17 minutes, while the API path needed 12,151 tokens and 19.7 seconds. Reflex open-sourced the benchmark alongside the post, and on May 5 it reached the Hacker News front page, where it drew extensive discussion.
One Task, Two Interfaces
The experiment was designed to isolate a single variable. The task is a port of react-admin’s Posters Galore demo: against a pinned dataset of 900 customers, 600 orders, and 324 reviews, the agent must find the customer named Smith with the most orders, accept all their pending reviews, and mark their newest order delivered. The only difference between the two paths is the interface. The vision path runs Claude Sonnet through browser-use 0.12 in vision mode, operating the UI with screenshots and clicks. The API path runs the same Sonnet, but a Reflex 0.9 plugin auto-generates HTTP endpoints from the app’s event handlers, so the model uses tool calling against typed tools like list_customers and update_order. Same model, same data, same task — only the interface changes.
Where the 45x Gap Comes From
The head-to-head numbers are brutal. The vision agent averaged 53 steps (plus or minus 13), and every step re-sends a full screenshot into the model, accumulating 551k input tokens. The API agent takes a fixed 8 calls, consumes 12,151 input tokens, and shows near-zero variance (plus or minus 27 tokens). Wall clock tells the same story: 1,003 seconds versus 19.7. Swapping models barely moves the needle: Haiku finishes the API path in 7.7 seconds on under 10k input tokens. A better model lowers the cost per step, but the number of steps is set by the interface — and every screenshot is a fixed overhead of thousands of tokens.
The Hidden Costs of the Vision Path
Beyond the headline numbers, the benchmark surfaced two costs that don’t show up on an invoice. First, the vision agent actually failed the plain prompt: it found only 1 of the 4 pending reviews because the rest sat behind pagination the model never scrolled to. It only went 3-for-3 after the author added a 14-step written walkthrough of the UI to the prompt — and that hand-authored documentation is exactly the kind of invisible engineering work the vision route demands. Second, the vision path is wildly non-deterministic: three runs ranged from 407k to 751k tokens and from 749 to 1,257 seconds, so a single run proves nothing. The authors are candid about limits, too: samples of 3 to 5 runs, one task only, and although the endpoints were auto-generated, the tool descriptions fed to the API agent were still written by hand.
When Computer Use Still Wins
The benchmark is not a death sentence for vision agents; the deciding question is whether you control the app. For internal tools you build yourself, structured APIs win decisively — especially now that frameworks like Reflex can auto-generate the endpoint layer, eliminating the traditional cost of hand-building an API or MCP layer. But for third-party SaaS or legacy systems you cannot modify, screenshots and clicks remain the only way in. The same trade-off shows up in web data extraction, where a structured scraping API usually beats driving a browser agent — a pattern we noted in our Firecrawl scraping guide. And bot detection on the open web remains a permanent variable for anyone driving real browsers.
Three Takeaways for Developers
First, design interfaces for your agents instead of making agents adapt to human interfaces: every screenshot is a fixed cost, step count is set by the interface, and that is where the 45x really comes from. Second, evaluate computer-use costs by total tokens and variance, not per-step price; the benchmark’s code and raw results are public on GitHub, and the directional conclusion is solid. Third, a Haiku-class model with a clean API interface beats a flagship model behind a vision interface — budget spent on interface engineering pays back better than budget spent chasing a stronger model.
Sources
- Computer use is 45x More Expensive Than Structured APIs — Reflex
- reflex-dev/agent-benchmark — GitHub
- Computer Use is 45x more expensive than structured APIs — Hacker News
AI-assisted summary compiled from the sources above, reviewed by a human before publishing.