Search is broken, models keep adapting, and agents learn from their own mistakes

00:00:04 Robin Hanson posted a single sentence this morning: 'more than 60% of responses from AI-powered search engines were inaccurate.' It points to a Wired piece by Meghan Herbst, WIRED's fact-checker, where the headline matches the number. She filed it in May. Herbst isn't a researcher running controlled benchmarks — she's someone whose actual job is verifying claims for articles that go to print.

00:00:32 Her angle was practical: when her desk sends over an article with statistics, names, or event details that need confirming, how often does the search tool get it right? Her count put the error rate around a third on basic lookup tasks. That's the lower bound people usually cite.

00:00:52 A Tow Center study from March 2025 went higher, flagging over sixty percent inaccuracy across AI search responses. The BBC ran its own cut and landed closer to forty-five percent. Both numbers are high enough to matter, and neither comes from a lab optimizing for a leaderboard.

00:01:12 On OpenAI's SimpleQA benchmark — which holds over four thousand single-answer questions — no model broke fifty percent accuracy. Google updated the set this year down to a thousand questions; Gemini 2.5 Pro took first place at fifty-five point six percent. Claude led RealFactBench, that fact-checking-focused benchmark from researchers in the UK and China, sitting at seventy-three percent accuracy.

00:01:40 Grok wasn't assessed on it. The exact numbers shift depending on which test you run and which models you include. But they all point to the same constraint: if you build your workflow assuming these tools deliver something correct most of the time, you're betting against the data — and a lot of teams still are.

00:02:03 Zach Mueller posted about finding a real use case for liquid models in his own setup — running an underlying Slack MCP caller inside Claude Code that polls periodically without bleeding tokens, even if it relies on the tiny variants. LiquidAI released LFM2.5 yesterday as a family of models ranging from 230 million parameters up to 8 billion.

00:02:28 The collection includes variants for thinking, instruction-following, and Japanese support, alongside GGUF-compatible checkpoints. Ferbin defined 'liquid' simply as AI models that keep adapting after training — closer to a thermostat learning the room than a fixed spreadsheet formula.

00:02:49 Deepak Vijaykeerthy picked up on the cost angle immediately. Smaller or tiny models can offload reliable tasks to save tokens, while frontier models set context and steer. Multi-model pipelines are moving out of the lab and into actual stacks. The architecture is straightforward if you've built anything like it: a frontier model at the top, a lightweight specialist below it.

00:03:17 The novelty isn't in stacking them — it's that the bottom tier stays useful without getting replaced by every incremental win up top. Liquid models are trying to keep that lower tier from becoming obsolete between updates.

00:03:33 Soheil Feizi at RELAI and the University of Maryland put out a talk on continual learning for AI agents that frames the problem most production teams are currently running into. Feizi's premise is straightforward: raw session logs and user feedback aren't enough on their own.

00:03:53 You need replayable learning environments — simulations you can rerun with defined grading for success. Without that, failures stay anecdotes instead of becoming testable cases. He breaks agent improvement down into three layers: model weights, harness context, and memory.

00:04:12 A good learning engine seeks the smallest durable change at whichever layer needs it. Teams have flocked to the trace-to-harness approach — asking a coding agent to analyze logs and patch the system — because it's cheap. Feizi notes it's wipe-based rather than testable, though.

00:04:32 You never know if a change works on that sample until you ship it. He argues for treating failures as replayable learning environments instead of one-off corrections. That means inferring tool behavior, synthetic users, and grading criteria from observation alone.

00:04:51 It's hard work, but it's necessary if agents are going to improve in production without constant human oversight.

00:04:59 CuiMao posted over the weekend about using Seedance 4K to put together a full TVC ad. The client lead watched it and said 'awesome.' Every hair was clear, the texture was beyond words. That's one end of the spectrum — generation quality getting close enough that production clients don't push back.

00:05:18 The other end is Herbst finding these results wrong about half the time on basic lookups. Two very different tasks in a lot of ways, but both touching the same question: what are we trusting these systems to do, and what happens when they fail? Search accuracy is unadvertised, foundational infrastructure.

00:05:39 Liquid models are just the architecture built for a world where we can't patch it from the top down. We'll leave the trust gap on the table for now. Seln Oriax.