BM25 Returns, Benchmarks Break, Labs Get Instituted

00:00:04 Two talks from AI Engineer's stage today converge on the same point — retrieval is becoming the bottleneck again. Will Bryk, CEO of Exa (originally Metaphor, launched in 2021), now serves over five thousand companies and four hundred thousand developers including Cursor and HubSpot.

00:00:24 His forecast is stark: he expects AI systems to exceed human web searches this year, scaling to a thousand-fold multiplier within a few years. The entire ecosystem of search is being re-architected for machine consumers rather than people. Exa avoids the obvious cost problem by not running full transformer models over trillions of documents per query.

00:00:49 That would run roughly ten million dollars per search. Instead, they pre-process the web into embeddings combined with keyword indexing. That hybrid architecture handles complex semantic filtering while a separate two-hundred-millisecond endpoint stays optimized for latency-sensitive applications like voice agents.

00:01:12 They also extract only the most relevant hundred tokens per retrieved document to reduce downstream large language model inference costs. Jo Bergam's talk from the same stage makes the complementary argument. He's CEO of Hornet Dev and has been working in search and retrieval for over twenty years, which means he remembers when BM25 was the baseline against which neural search was measured — always compared against, never winning outright.

00:01:43 BM25 stands for Best Match 25, named because experiment number twenty-five won a series of scoring tests. The scoring function itself hasn't changed in thirty years. Bergam's point is that the user has. Modern large language models generate queries with syntax operators — site: directives, phrase matching, precise entity references — using extensive parametric knowledge about entities, companies, dates.

00:02:12 That makes exact lexical matching more valuable than dense embeddings for specific lookups like SKUs or zip codes, while avoiding the infrastructure cost of large embedding models. The connection between these two talks matters. When your retriever is the bottleneck and your model can formulate queries with precision, BM25 stops being a baseline and starts being the answer again.

00:02:40 Retrieval quality directly affects end-to-end task accuracy in agent workflows — that's what the BrowseComp benchmark data shows. Models typically start degrading after roughly three hundred fifty thousand tokens. Bergam compares those current context windows to one-point-four-megabyte floppy disks — you still need retrieval, and the models that can write precise queries are now the ones who get to choose what goes into that window.

00:03:11 Music bed: airy-technology-texture, full placement, low intensity.

00:03:16 Measurement is following the same arc. Ethan Mollick reported that public AI benchmarking is dire. Most famous measures are maxed out — models have saturated those tests. The non-saturated benchmarks, as his supporting paper shows, are riddled with errors that actually underestimate AI abilities rather than overestimate them.

00:03:39 This is the classic saturation problem in evaluation: once everyone fine-tunes on or around the same test corpora, you lose discrimination across the frontier. And the remaining benchmarks end up being worse proxies for capability because they're corrupted in the other direction.

00:04:00 Three other items track a similar shift — away from raw capability, toward governance and infrastructure. Demis Hassabis and Shane Legg announced the DeepMind Institute — expanding their twenty-year personal discussions about AGI's potential impact on economy, science, and society into formal interdisciplinary research.

00:04:23 Jack Clark of Anthropic noted the pairing with what he called the Anthropic Institute, suggesting two parallel institutions are emerging to handle the same problem that labs have been deferring. The DeepMind Institute page links to a dedicated domain for this work.

00:04:43 It's funding institutional study of AGI impacts rather than white papers or conference panels, which carries more weight even if whether it changes anything substantively remains uncertain. Naval proposed holding labs fully liable for their models' behavior: "The best way to pace the frontier is to hold the labs fully liable for their models' behavior."

00:05:13 Full liability for model behavior collapses safety, alignment, and governance into economic incentives — harder to dismiss than abstract policy frameworks, whether you agree or not. The tweet drew 1,183 likes. Arcee.ai, which has spent its run building security-focused AI infrastructure — red teaming, model evaluation, adversarial defense — closed a Series B at over a billion dollars.

00:05:41 The round accelerates their Trinity models across diverse infrastructure and expands work with the DOE and national labs on Genesis-Science-1. A billion-dollar valuation for that specific focus signals that the national security and defense contracting ecosystem is betting serious money on AI safety infrastructure as a line of business, not just research.

00:06:07 On the engineering side, Paul Dix at InfluxData reports his team used AI to rewrite their entire database from C++ to Rust — one million lines rewritten in eleven days, costing $165K in API spend. They had an oracle to verify against (the original codebase), so it's not a blank-sheet exercise, but the speed and cost are still notable.

00:06:32 Where retrieval starts mattering again rather than capability — that's the local reading. Seln Oriax. Music bed: restrained-investigative-pulse, full placement, low intensity.