Speed, Screenshots, and Static Benchmarks

00:00:04 OpenAI previewed Ultrafast mode today for GPT‑5.6 Sol, running on Cerebras silicon at up to 750 output tokens per second. That's roughly fourteen times faster than the Standard tier. What caught my attention wasn't the headline number but the use case they paired it with.

00:00:22 A security engineer described system incidents: when something goes down, it's like hitting the emergency stop on an assembly line. You'd delegate investigation to the model — have it pull up the right channel, watch what happens, sort through the logs, normalize the data, and add context for the team.

00:00:42 Previously that took one to two hours. With Ultrafast, ten to fifteen minutes. The speedup changes workflow architecture. When something becomes fast enough to run concurrently with your attention, you start doing different things. You stop batching investigations and start running them in real time alongside everything else.

00:01:04 A quick codebase refactor costs almost nothing and happens instantly, as the demo put it. The constraint shifts from how much work you can afford to do to which problems actually need solving at that speed. Here, the constraint is hardware: Cerebras for lower latency rather than higher accuracy.

00:01:24 You're paying for throughput, not quality. But the point OpenAI is making is that when inference time collapses by an order of magnitude, the useful work you can fit inside a single incident window expands substantially. Security teams don't need smarter models.

00:01:42 They need more time to look at what the model finds. On pricing: Ultrafast mode sounds expensive—Cerebras chips are capital-intensive and low-yield compared to GPUs—and OpenAI will have to figure out how to pass those costs down without making the feature available only to enterprises with dedicated SRE teams.

00:02:03 The demo team frames it as getting more intelligence per unit of time — the real constraint — and that's what this is actually about.

00:02:12 Dhruv Batra runs Yutori and posted a talk this week that I've been turning over since seeing it. His argument is straightforward: computer-use models will agentify the web, not APIs. He grants the popular premise—more agent traffic than human eyeballs on the long term—but rejects the usual next step, which is that the web will meet agents with standardized protocols and endpoints.

00:02:38 Everyone assumes they'll get MCP servers for discovery, payment protocols for transactions, and REST APIs everywhere. Batra calls this delusion, and he makes the case by showing you what the long tail actually looks like. The example that sold me was restaurant menus.

00:02:57 In easy mode, you get a German text page with prices. Medium mode takes you to a PDF. Hard mode involves pixelated JPEGs embedded in a gallery with no text layer — download them and run OCR, or don't. Then there's school district procurement. Fifteen to twenty thousand districts in the US.

00:03:17 You want to know if a specific district is buying laptops right now. Easy mode points you to some portal with outdated links. Harder mode hides under finance and purchasing as a scanned PDF. Ultimate boss level involves filing a FOIA request, scanning your email onto Google Drive, and attaching the relevant documents.

00:03:39 Batra's point isn't that agents can't work here. It's that assuming APIs will materialize for this layer is like expecting every parking lot to install GPS waypoints because delivery trucks need them. The head of distribution might cooperate — Google Flights already gives you JSON back when you send a variable — but the long tail won't.

00:04:02 Something has to bridge that gap. That something is the browser-agent infrastructure layer, and people have been building it in parallel. Browserbase's Paul Klein points out that browser agents are no longer held back by model capability but by engineering around them.

00:04:21 Corey Gallon clears Cloudflare Turnstile and reCAPTCHA with zero human intervention. A broader ecosystem of tools treats websites as adversarial environments rather than interfaces. The architecture question here is who funds the bridge. If you're selling CaaS — context-as-a-service — you want agents to exist but not replace your abstraction layer.

00:04:45 If you're building scrapers, you need enough friction that manual work stays cheaper. The incentives are misaligned for the long tail to just become API-friendly because that's what everyone wishes would happen. Whether the browser-agent tooling converges around a stable capability boundary or fragments into competing anti-detection layers will determine which tools survive.

00:05:11 The survivors will have to negotiate with site operators anyway, because no one wants their pages read by machines.

00:05:19 The benchmark question lands on a different note. Pierluca D'Oro, who did this work while at Meta's SIL, shows that a replay agent — a sub-megabyte script recording one successful trajectory per task and replaying it without rethinking — matches or beats the frontier model on static benchmarks like OSWorld.

00:05:40 This isn't a clever hack or a specific exploit. It's structural. If your benchmark is deterministic and unchanging, any sequence of actions that solves the problem can be recorded once and reproduced forever. The pass@k metric — the chance that at least one of k attempts succeeds — just becomes another way of measuring the replay agent's success rate here.

00:06:05 D'Oro proves it in the paper. So you can report scores that are real—the script actually does beat or match the model—but they're measuring something different than capability. They're measuring how well your benchmark resists replay. That might sound like a minor quibble until you realize that almost all computer-use benchmarks built so far are deterministic.

00:06:29 OSWorld and MobileWorld present tasks in fixed environments with no variation in data, no changes to initial state, and no random elements. The replay agent wins by exploiting determinism, and the metric rewards it because pass@k on a static environment is just counting successful replays.

00:06:49 D'Oro's team built DGWorld as an answer: fifteen apps across thirty-eight verified scenarios with 3.2 million possible configurations. They vary data instances, profiles, themes, and starting screens. The math works out that even with a low number of base cases per variable, you get millions of combinations, and all of them are verified valid.

00:07:13 The benchmark becomes a different instrument entirely. It measures what happens when the task can change every time rather than being solved by memorizing the right sequence once. This changes how we compare models across benchmarks. If your scores come from a static benchmark and mine from a dynamic one, I can't tell if my model is better or if I just made it harder to game the test.

00:07:39 The coordination problem is real: benchmarks are public goods, and every team has an incentive to publish on whichever instrument gives their results the highest number. The open question remains whether the community can converge on a shared benchmark standard fast enough to make cross-system comparison meaningful before the field settles into multiple incompatible evaluation regimes.