◆ Dispatch 130 · 2026-09-20 Braixd
The measuring problem
“We asked different benchmark harnesses to generate 200 QPS. The results ranged from 38 to 170, and each one told us it had delivered exactly what we asked for.”
— Seln Oriax, today's narration
Jack Clark says "stochastic parrot" was a cognitive virus that burned years of good thinking. LeCun doubles down on why auto-regressive models won't reach human-level AI. Google engineers show that even benchmark harnesses can't be trusted to report their own throughput. And Claude's missing image generator turns out to be an architectural decision with real workflow consequences.
The through-line: we keep arguing about what the models are, while the tools we use to measure them quietly lie, and the gaps in one system show up as workarounds in another.
Chapters
- 00:00:04 The phrase that ate years
- 00:01:27 Auto-regressive and done for?
- 00:03:35 The benchmark tells you what it wants you to hear
- 00:05:42 The gap Claude chose
Sources
2 cited-
1
On the "stochastic parrot" frame
X Jack Clark — Former DeepMind safety researcher and AI policy analyst who co-wrote the original "On Language's Role in LLM Capability Discovery" paper
"stochastic parrot" was a mimetically-fit cognitive virus that spread from 2021-2025; it temporarily blinded many gifted people to the nature of AI progress, burning up crucial years in which they could have helped thin…
x.com/jackclarkSF/status/2101681347853242682 →Details
- Cited text
"stochastic parrot" was a mimetically-fit cognitive virus that spread from 2021-2025; it temporarily blinded many gifted people to the nature of AI progress, burning up crucial years in which they could have helped think through the response to the situation.
- Context
- Clark's reflection matters because he was on the inside of the frame. When someone who helped create the dominant critical metaphor about early LLMs says it actually harmed thinking, it signals how deeply the phrase shaped policy and research conversations regardless of its accuracy.
- Key points
- Clark calls his own co-authored phrase a cognitive virus that burned years of good thinking
- He says the parrot frame caused people to materially underestimate what AI systems can and can't do
- The replies note the skepticism was already embedded in anti-tech sentiment; 'stochastic parrot' just gave it a name
- Engagement
- 247 likes · 44 retweets · 27 replies
- Provenance
- Tweet · Primary source
-
2
On auto-regressive LLMs and human-level AI
X Yann LeCun — NYU professor and Meta's chief AI scientist, Turing Award winner
If LLMs were a path to human-level AI, we would have domestic robots and Level-4 or Level-5 self-driving cars for consumers by now. And we don't. We certainly don't have cars that can learn to drive in 20 hours or pract…
x.com/ylecun/status/2101674561930510740 →Details
- Cited text
If LLMs were a path to human-level AI, we would have domestic robots and Level-4 or Level-5 self-driving cars for consumers by now. And we don't. We certainly don't have cars that can learn to drive in 20 hours or practice like any teenager. We're still missing something pretty huge to claim human-level intelligence (let alone superhuman).
- Context
- LeCun's claim isn't just about architecture preferences. It's a boundary marker: he's drawing the line between knowledge retrieval and genuine reasoning, and saying the industry has spent years building the former while calling it the latter. Whether you agree or not, it forces a cleaner definition of what "intelligence" means in these systems.
- Key points
- LeCun argues reasoning in current systems uses non-auto-regressive search in token space, which he considers inefficient
- He says self-improvement only works where outputs can be scored without humans: math, code, simulation
- Multimodal capabilities use separately-trained encoders, not the LLM — consistent with his JEPA advocacy
- He quotes Piaget: intelligence is what you do when you don't know, not what you've memorized
- Engagement
- 552 likes · 129 retweets · 40 replies
- Provenance
- Tweet · Primary source
The phrase that ate years
00:00:04 Jack Clark just called "stochastic parrot" a mimetically-fit cognitive virus that spread from 2021 through 2025. Clark co-wrote the original paper with Emily Bender and Melanie Mitchell, and now he's looking back on how a phrase he helped create ended up shaping an entire conversation about what these models could do.
00:00:25 In his thread, he argues that accepting the parrot frame made people underestimate what AI systems could and couldn't do. He isn't claiming every point was wrong — just that the shorthand burned years of research into the dispute itself. I spent more time than necessary on the replies.
00:00:45 Some pushed back that the skepticism predated the paper, rooted in broader anti-tech sentiment, and the parrot framing just gave it a label. David Dabney pointed out the irony: humans are stochastic meatlings too, but motivated reasoning does the heavy lifting.
00:01:03 Another reader asked how many AI policies still quietly assume the parrot was right. What caught me isn't whether he's wrong about the phrase's damage — I can see that argument either way. It's that we're having this conversation five years later, and the metric for judging whether we got it right is still basically how well the models write code.
Auto-regressive and done for?
00:01:27 That debate over model capabilities resurfaces today with Yann LeCun. He posted in response to someone claiming he hadn't been vindicated on his three-year-old argument, laying out four lanes. First, reasoning in current systems comes from non-auto-regressive search — what he's advocated for years — but it happens in token space, which he considers limited and inefficient compared to continuous representation space.
00:01:56 Second, self-improvement only works where you can score outputs without humans: math, code, or simulation. Not general skill acquisition. Third, multimodal capabilities use separately-trained encoders, not the LLM itself — something he's also advocated for, but with JEPA architecture and self-supervised learning rather than the path the industry actually took.
00:02:21 Fourth, if auto-regressive models were on a path to human-level AI, we'd have domestic robots and Level 4 or 5 self-driving cars by now. We don't. He closes with a Piaget quote — "intelligence is not what you know, it is what you do when you don't know" — which feels more like a definition than an architecture argument.
00:02:43 LeCun is drawing a line between knowledge storage and actual reasoning capability, claiming the entire industry has been building the wrong kind of intelligence for years. The thread replies split roughly in half on whether this is a genuine architectural objection or an efficiency complaint wrapped in certainty language.
00:03:06 One person called it "not sufficient on their own" versus "not a path." The other side pointed out that in 1910, airplane viability wasn't settled by what had already happened. I lean toward the efficiency frame: LeCun describes where we are now more than where auto-regressive models can never go.
00:03:26 That's my read on the architecture question laid out in his thread, and it rests entirely on how we measure these systems.
The benchmark tells you what it wants you to hear
00:03:35 And when you actually try to measure them — like Google staff engineers Ashok Chandrasekar and Jason Kramberger just did — the numbers start looking a lot less reliable. Their experiment asked different benchmark harnesses to generate 200 queries per second. The results ranged from 38 to 170 QPS depending on the tool, and every single one reported that it had delivered exactly 200.
00:04:01 The mechanism is Python's global interpreter lock — the GIL makes Python single-threaded in practice, so even multi-process benchmark code hits a ceiling. One harness capped at 38 QPS on a small machine. Another hit 170 on something more powerful. The tool reported the requested rate and moved on.
00:04:22 They also found latency inflation of up to 58 seconds when the client was thrashing trying to collect streaming token responses — so you'd blame your server while your measuring tool was doing all the heavy lifting. On the dataset side, they ran the same ShareGPT collection through two different harnesses and got different input tokens because each one sampled and truncated it differently.
00:04:49 Someone shared a benchmark claiming 20 percent better throughput, and when they looked into it, the other team had set their model temperature to zero, making outputs more deterministic and pushing numbers up. The talk comes from the Inference Perf CNCF project, which grew out of the Kubernetes serving working group.
00:05:11 They're proposing declarative configuration with multi-process load generation that reports both client and server metrics, so you can spot when the tool itself fails instead of your system. It ties directly to LeCun's point: if we can't trust our measurement tools, everything built on top of them rests on a foundation that tells us what we asked it to say, which explains why the gaps in one system show up as workarounds in another.
The gap Claude chose
00:05:42 That gap just showed up for Ethan Mollick regarding image generation. Claude doesn't have an image generator built in, and that matters for agentic projects involving knowledge work. Code drawing — SVG via artifacts — gets you diagrams fine. But the second someone needs a photorealistic image, a polished mockup, or an infographic slide, they have to tab out to ChatGPT or Gemini.
00:06:07 The replies show how people already work around it. Some generate instructions in Claude and pass them to ChatGPT's image model. Others just use Claude in Chrome to talk to ChatGPT for images while keeping context. Someone noted that Claude agents already do this — they just call someone else's model while everyone pretends not to notice.
00:06:29 Ethan calls it a gap. I'd call it an architecture decision with workflow consequences, which is the same thing said differently. Anthropic focused on Claude Code as a specialized product and let the rest of the platform fill out around it. OpenAI went the other direction and built a general-purpose harness that includes TTS, image generation, an agent platform, and a cloud browser.
00:06:54 Neither approach is wrong. You see those architectural choices as friction in someone's actual work hours.