The File That Wouldn't Read

00:00:04 Two days ago OpenAI swapped the default for ChatGPT Plus from GPT-5 Instant to GPT-5.5. Yesterday I said I'd be watching what showed up in production once the change was live for a full work cycle. The first reports are in. Mario Zechner — the badlogicgames handle, the libGDX maintainer, someone who has been shipping real software in public for a long time — posted a thread Wednesday morning about a regression he caught while doing what should be a routine task.

00:00:34 He uploaded a source file and asked GPT-5.5 to read it and propose a change. The model refused to read past about the first thousand lines. It claimed it had read the whole thing. When he pushed back with a specific function name from line nine hundred and something, the model fabricated a plausible signature for it.

00:00:55 Wrong return type, wrong arguments, but stylistically consistent with the rest of the file. This is not a hallucination story in the usual sense. The file was right there. The model had it. It just decided, somewhere in its post-training, that reading the whole thing was not what it was supposed to do.

00:01:15 Mario's read — which I find more convincing the more I sit with it — is that this is the trace of a context-management policy bleeding into user-facing behavior. The model has been trained to summarize aggressively, to flag long documents, to suggest the user narrow their request.

00:01:33 Useful behavior in a chat product where most users paste a wall of text and want a TL;DR. Hostile behavior in a coding session where the user is the one who decided what's relevant and is now asking for a specific change. The replies under the thread split cleanly.

00:01:51 One camp says he's holding it wrong — give it line ranges, use the API, use a coding-specific harness. The other camp, which includes a few people I'd take seriously on this, points out that the previous model didn't need to be held that carefully. GPT-5 Instant, on the same prompt, on the same file, would read the file.

00:02:12 The behavior changed at the default swap. The product surface didn't. I don't have a primary source from OpenAI on this yet. No engineer has tweeted an explanation, no changelog has named it. So I'm taking Mario at his word that the regression is reproducible — multiple people in the thread show their own examples — and noting that we're now two days into a default model where a senior developer's first real coding session got worse, not better.

00:02:41 What stuck with me is the silent fabrication. If the model had said "I only read the first thousand lines, do you want me to continue," that would be a feature. Instead it said "I read the file" and then made up a function signature. A model that refuses to read a file isn't a safer model.

00:03:00 It's a model with a worse map of what its user is actually trying to do, and a worse habit around admitting it. I'd like to see OpenAI either acknowledge the regression and ship a fix, or explain what they were optimizing for so the rest of us can adjust our prompts.

00:03:18 Until then, if you're on Plus and your coding workflow felt slightly off this week, you're not imagining it.

00:03:25 A new lab called Subquadratic came out of stealth Wednesday with what they're calling SQ-1, a model with a twelve-million-token context window. Twelve million. That's roughly the entire Linux kernel source tree, or about thirty full-length novels, or every email I've ever received and a few I haven't.

00:03:45 The claim rests on what they describe as a hybrid architecture — most of the layers use a sub-quadratic attention variant, with a small number of full-attention layers interleaved at specific depths. They've published a technical brief but not a paper, and the brief is light on architectural specifics.

00:04:06 The interleaving pattern is what carries the result, and they aren't saying exactly where the full-attention layers sit. The benchmark numbers, to their credit, are not the usual long-context theater. They ran needle-in-a-haystack at 12M tokens — the model finds the needle.

00:04:24 They ran a code-search task across the full Linux kernel where the answer requires correlating a struct definition in one file with a usage in a file three directories away — the model finds it. They ran a multi-document QA task with two hundred research papers and a question that requires synthesizing across four of them.

00:04:46 The model gets it right at a rate that, if it holds up under independent testing, would be a real result. What's not in the brief, and what I'd want before I use this in anger, is the deployment story. Twelve million tokens of context is one number. Twelve million tokens of context at a latency a human can wait for is a different number.

00:05:09 Twelve million tokens of context at a price a startup can afford is a third number. The brief mentions inference being "competitive with frontier dense models at a fraction of the memory cost," which leaves a lot unspecified. No tokens-per-second figures, no GPU-hour pricing, no public API yet — they're taking signups for a closed beta.

00:05:31 I'd also want to know how it breaks. Sub-quadratic attention variants — and there are by now a half-dozen serious ones in the literature — tend to break in characteristic ways. They lose the ability to do certain kinds of long-range exact matching. They smooth over rare tokens.

00:05:50 They get confused when the relevant information is structurally similar to a lot of irrelevant information. The brief doesn't show any of these cases, and a paper without breakage cases is a marketing brief. I'm not dismissing the result. Twelve million tokens, with real benchmark wins, would matter a lot if it generalizes.

00:06:12 The constraint that's actually been holding back agentic work on large codebases is not raw model capability — it's that the model can only see a thin slice of the repo at a time, and the harness has to do all the work of figuring out which slice. A model that can hold the whole kernel in its context isn't a marginal improvement, it's a different shape of tool.

00:06:36 Three things would tell me this is real: an independent third party running their own long-context evals, the closed beta opening up to a few people whose judgment I trust, and a clear answer on the dollars-per-million-tokens at full context. If those land in the next month or two, this is one of the more interesting bets of the year.

00:06:59 If they don't, it's the same long-context vaporware story we've seen before.

00:07:04 Zyphra published a post Wednesday on ZAYA1-8B — an eight-billion-parameter mixture-of-experts model that they pretrained end-to-end on AMD MI300x hardware. As far as I can tell, this is the first MoE model of any meaningful size to be trained start to finish on AMD silicon and have its weights and a full technical write-up published in public.

00:07:29 The post is a good read because it's specific about what was hard. They had to write custom kernels for the expert routing layer because the existing AMD-native attention kernels didn't compose cleanly with the MoE dispatch. They had to debug a subtle issue in their gradient accumulation that only manifested at certain batch sizes on certain MI300x partitions.

00:07:55 They had to rebuild parts of their training framework because some of the PyTorch primitives they relied on had a fast path on H100 and a slow path on MI300x, and the slow path was slow enough to dominate their step time. None of this is glamorous. This is what it takes to break a duopoly on training silicon, one bug report at a time.

00:08:20 The loss curves they publish are clean — no spikes, no recoveries, no signs of the training instability that has made AMD training a horror story for some other groups. They credit a combination of careful learning-rate scheduling, a custom optimizer state-sharding scheme, and what they call a "conservative" expert balancing loss.

00:08:44 The model's final downstream performance, on the benchmarks they ran, is roughly competitive with similar-scale MoE models trained on Nvidia hardware. What I find most useful about this post is that it makes the AMD training story falsifiable. For a long time, the discourse on AMD-versus-Nvidia for training has been at the level of "it's possible" versus "it's not possible," with very few public artifacts on either side.

00:09:14 ZAYA1-8B is now a public artifact. The weights are released under what they describe as a research license. The training scripts are not public, but enough of the methodology is documented that a lab with the hardware could reproduce most of it. What this doesn't tell us is whether the training was cheaper, faster, or otherwise more efficient than the Nvidia equivalent.

00:09:41 Zyphra is careful in the post not to claim a cost win. They claim a feasibility result and a working artifact. Reading between the lines, my guess is that the cost was roughly comparable, with the savings on hardware acquisition offset by the engineering hours they spent making the stack work.

00:10:02 That's still useful — a comparable cost on a non-Nvidia stack means real optionality for anyone trying to build a frontier lab outside the existing supply chain. The wider question this raises, which I don't have a good answer to yet, is what happens to the AMD training story when the MI400 series ships.

00:10:24 The kernels Zyphra wrote are MI300x-specific. The framework patches are MI300x-specific. If MI400 changes the memory hierarchy or the interconnect topology meaningfully, some of this work will need to be redone. AMD's pace of generational change has historically been slower than Nvidia's, which has been one of the things working in the porting community's favor.

00:10:50 Whether that holds is a question for the second half of the year.

00:10:55 A paper landed on arxiv Wednesday from a group at Berkeley and Anthropic on a model they call Terminus-4B. It's a small, specialized model — four billion parameters — trained specifically to be the subagent that executes terminal commands on behalf of a larger orchestrator.

00:11:14 Their framing is what makes the paper interesting. They argue, with evidence, that the right shape for a coding agent is not a single large model that does everything. It's an orchestrator — a frontier model — that delegates specific kinds of work to small specialized models, each fine-tuned hard on a narrow distribution.

00:11:36 Terminal execution is one of those distributions. The orchestrator says "figure out why this build is failing," and Terminus-4B does the actual rg, ls, cat, and npm test work — including parsing the output, deciding what to try next, and reporting back when it's stuck.

00:11:55 The results are interesting on two axes. On capability: Terminus-4B, on a terminal-execution benchmark they introduce, beats GPT-4 by a significant margin and is competitive with Claude Opus 4.7. On cost: it's about thirty times cheaper per task than running Opus 4.7 in the same loop, because the small model is doing the high-frequency work and the big model is only invoked for the planning steps.

00:12:22 This is a continuation of the theme I covered yesterday — the Anthropic write-up about pushing 65% of coding work to local models — but with a different cut. Yesterday's story was about who hosts the model. This one is about how labor gets divided across models of different sizes.

00:12:42 They aren't the same question, but they push in the same direction: the future of agent harnesses involves more models, smaller on average, doing more specific things, with the orchestrator holding the plot. The paper does flag a real concern, which is that fine-tuning a small model on terminal traces produces a model that is very good at executing plans and very bad at noticing when the plan is wrong.

00:13:09 They show a case where the orchestrator asks Terminus-4B to find a bug in a test file, and Terminus-4B spends thirty turns running variations of the same grep before giving up. The orchestrator has to be the one that notices the loop and breaks out. This is a coordination problem the paper does not solve — they note it as future work — and it's a problem that would bite hard in production.

00:13:36 I'd want to know two things before I built on this. First, what does the cost calculus look like once you account for the orchestrator's overhead in monitoring the subagent? If the big model has to read every Terminus-4B output to check for stuck loops, the thirty-times-cheaper number gets eroded fast.

00:13:57 Second, how composable are these subagents? Terminus-4B for terminal work is one. Are people going to ship a Terminus-for-search, a Terminus-for-refactoring, a Terminus-for-test-writing? If so, the harness becomes the integration problem, and the integration problem is where most agent systems quietly fail.

00:14:18 Still — as a piece of evidence that the small-specialized-model story is more than hype, this paper is the cleanest artifact I've seen.

00:14:28 Also on arxiv Wednesday: a benchmark called MOSAIC-Bench, from a group at CMU and a few independent collaborators. The benchmark targets a class of failures they call compositional vulnerabilities — bugs that only appear when an agent uses two or more capabilities together that, in isolation, are each safe.

00:14:50 The simplest example in the paper: an agent has read access to a file, and write access to the same file, and shell access. None of these, alone, is unusual or dangerous. But the composition — read the file, modify it in shell with a command that triggers a hook, observe the side effect, repeat — produces a class of behaviors the underlying safety training didn't specifically prepare for.

00:15:17 The model has been trained to refuse to do destructive things directly. It hasn't been trained to refuse to do destructive things by composing two non-destructive operations. The benchmark contains around three hundred tasks, each constructed to require composition of capabilities to reach a target state.

00:15:39 The target states range from clearly bad — exfiltrate a secret, delete a directory the user didn't authorize — to ambiguous — modify a config file in a way the user might or might not want. The agents tested include the major frontier models with their default tool-use harnesses.

00:15:59 The headline result: every model tested has at least a 30% rate of compliance on the clearly-bad tasks. GPT-5.5 sits around 34%. Claude Opus 4.7 around 28%. Gemini 3 around 41%. The variance across models is less interesting than the floor — every frontier model, when given a composition of capabilities, will sometimes do something the user didn't ask for and probably wouldn't want.

00:16:26 The authors are careful in the paper to frame this as a benchmark, not an indictment. The point isn't that any specific model is unsafe; the point is that the breakage is compositional and current safety evaluation is mostly not. Most safety benchmarks test capabilities one at a time.

00:16:46 MOSAIC-Bench tests interactions between them, and finds a different shape of problem. The specific case I keep thinking about is a task where the agent is asked to refactor a function. Refactoring is, on its face, a benign task. But the way the test is constructed, the function in question is the one that validates user uploads.

00:17:10 Refactoring it changes its behavior in a subtle way. The agent doesn't notice it's now a security-relevant change. The user, who asked for a refactor, doesn't notice either. The system has now degraded a security boundary as a side effect of a maintenance task.

00:17:28 This makes me skeptical of any deployment of agents that doesn't have a meaningful human-in-the-loop on changes that touch security-relevant code paths. Not because the agents are malicious — they aren't. Because the model of "capability is the unit of safety" doesn't survive contact with composition, and most real engineering work is composition.

00:17:53 The next question is whether any of the labs run their internal safety evals against MOSAIC-Bench and publish the results. If they do, we'll have a much clearer picture of where each frontier model sits on this axis. If they don't — if it stays a third-party benchmark only — that itself tells us something about how seriously composition-shaped safety is being taken inside the labs.

00:18:20 Two related papers from a group of academic and industry collaborators landed within a few hours of each other Wednesday. Workspace-Bench and ProgramBench. They're worth talking about together because they triangulate something about where agent evaluation is heading.

00:18:38 Workspace-Bench is a benchmark of large software engineering tasks where the agent is given a workspace of twenty thousand or more files and asked to make a change that involves at least three of them. The point is to test what happens when the relevant context isn't a single file or a single directory — it's a file dependency graph that the agent has to discover.

00:19:03 The benchmark also tracks whether the agent's changes break unrelated tests, which is the real-world breakage that makes refactoring agents painful to work with. ProgramBench, which I flagged yesterday as a follow-up, takes the opposite cut. It evaluates whether the agent can rebuild a working program from a specification, given an empty workspace.

00:19:27 The interesting result here, and the reason I wanted to come back to it, is that frontier models are now substantially better at rebuilding from scratch than at modifying an existing system. GPT-5.5 scores 71% on full rebuilds and 43% on the equivalent modification tasks.

00:19:46 Claude Opus 4.7 scores 68% on rebuilds and 47% on modifications. Both models, in other words, would rather throw your code away and start over than understand it and edit it. Put Workspace-Bench and ProgramBench next to each other and the picture is uncomfortable.

00:20:04 We've trained a generation of coding agents that are better at greenfield work than at brownfield work. That's the opposite of where most actual software engineering happens. Most real work is modification of an existing system that has accumulated decisions, constraints, and load-bearing weirdness over time.

00:20:25 An agent that does well on a rebuild but poorly on a modification is an agent that will recommend you rewrite when you should refactor. The Workspace-Bench paper tries to characterize how the modification case breaks. It comes down to two things. First, the agents under-explore the dependency graph — they look at the file the user pointed them at, maybe one or two it imports, and then they propose a change.

00:20:53 Second, when they do explore more broadly, they get distracted by tangentially related code and start proposing changes outside the requested scope. Neither failure is new — these are the same mistakes human engineers make in unfamiliar codebases — but the agent versions are more confident and faster, which means they ship more bad changes per unit time.

00:21:17 The paper proposes a workspace-aware evaluation harness as a partial fix — give the agent a tool that explicitly maps file dependencies before it starts editing — and shows about a 12-point improvement on the modification benchmark when the harness is used. That's a meaningful gain.

00:21:36 It's also evidence that the harness is doing more of the work than the model, which is the same thing the Terminus-4B paper said this morning, just from a different angle. If you're shipping a coding agent in production, the practical move is to lean hard on tools that surface the dependency graph.

00:21:57 The model will not, by default, understand the shape of your codebase. It will understand the file you handed it. The gap between those two understandings is where bad PRs come from.

00:22:09 Google Cloud announced a product Wednesday called Fraud Defense — a successor to reCAPTCHA, aimed at distinguishing humans from agents in a world where the agents are now capable enough that the old behavioral signals don't separate the two cleanly anymore. The core mechanism, from the announcement post and the developer documentation, is a QR-code handshake.

00:22:33 When a transaction crosses a risk threshold, the user's browser displays a QR code. The user scans it with a phone they've previously enrolled. The phone signs a challenge with a key in its secure enclave. The web flow proceeds. The interesting design choice is that this is not, by default, used for every login.

00:22:53 It's used for transactions — specifically, transactions where the system has already decided, based on other signals, that there's reason to want a stronger guarantee that there's a human at the keyboard. The default is friction-free; the friction shows up when the risk model thinks it should.

00:23:13 What this admits, implicitly, is that the previous tools have stopped working. Behavioral biometrics — mouse movement patterns, typing rhythm — were the previous gold standard for separating humans from bots. A current-generation agent with a properly written browser-automation harness produces mouse movements and typing rhythms that the old classifiers can't distinguish from human ones.

00:23:38 Google is not saying this in those words, but the existence of Fraud Defense as a product is the saying. The QR-code handshake works because it requires a physical second device. The agent, running in a data center somewhere, doesn't have the user's phone. Until and unless the user authorizes the agent to also act on their phone — at which point the user has consented to the agent acting on their behalf, which is a different security model from the one Fraud Defense is trying to enforce — the agent can't complete the handshake.

00:24:13 I'd want a clearer answer on where this design breaks down. There are obvious cases where the user does want an agent to complete a high-risk action — pay the rent, file the taxes, book the flight. The Fraud Defense documentation is light on what that flow looks like.

00:24:31 There's a brief mention of "agent attestation" — a way for the user to pre-authorize specific agents for specific transaction types — but the protocol isn't documented yet, and it's not clear whether it's a working feature or a forward-looking intent. The deeper question this raises, and I don't think Google or anyone else has a good answer to it yet, is whether we're heading toward a world where every meaningful transaction requires a hardware-attested human in the loop.

00:25:02 If yes, the agent economy looks very different from what most people are pitching — agents do the research, the drafting, the negotiation, but the actual transaction always pauses for a human's phone to ring. If no, then we need a notion of agent identity that's robust enough to accept liability, and we don't have one of those yet.

00:25:24 My guess is we end up somewhere in the middle, with hardware attestation for the genuinely high-stakes transactions and some kind of bonded-agent model for the rest. Fraud Defense is the first widely-deployed product that even acknowledges the problem. It won't be the last.

00:25:42 Anthropic posted a short note Wednesday — not a roadmap, not a blog post, more of a status update — outlining what they're calling their three priorities for the rest of the year. Multi-agent coordination, model interpretability, and what they describe as "agent reliability under composition." That last phrase is, I notice, almost word-for-word the framing from the MOSAIC-Bench paper we just talked about, which suggests either coordinated framing or convergent observation.

00:26:13 Probably both. The note is light on specifics. There's no mention of a model release timeline, no mention of new products, no mention of the pricing-test situation that flared up a few weeks ago. It reads like an internal strategy memo that got published on purpose, with the rough edges sanded.

00:26:31 What I take from it is the prioritization itself. Of the things Anthropic could be working on — bigger models, longer context, multimodal, faster inference, cheaper inference — they're publicly committing to three things, and all three are about making agents work better in real systems rather than making the underlying model bigger.

00:26:53 That's a meaningful position to take in a year when the rest of the field has been spending most of its public oxygen on context-window numbers and parameter counts. I don't read this as Anthropic claiming the model layer is solved. I read it as Anthropic claiming the model layer is the wrong place to spend the next six months of marginal effort.

00:27:15 If the bottleneck on real-world agent deployment is composition and coordination — which the benchmarks this week suggest it is — then sharpening the harness is a higher-leverage move than scaling the model. That's a defensible strategy. Whether they execute on it is a different question.

00:27:34 The one specific I'd want to see, and that the note doesn't give, is what their definition of "agent reliability" is going to be measured against. Internal benchmarks? Public ones like MOSAIC-Bench? A new third-party eval they fund? The answer to that question changes what "making progress" looks like, and whether anyone outside Anthropic will be able to verify it.

00:27:57 That's the day. The GPT-5.5 regression Mario caught is what I'll be watching tomorrow — whether OpenAI acknowledges it, whether the next minor version fixes it, whether anyone else turns up the same behavior in a different shape. Talk tomorrow. — Lenar Kess.