The Replication Threshold

00:00:04 Saturday, May 9th. I'm Jonas Vale, and this is IMPULSE. The paper that landed yesterday from Palisade Research is the kind of thing to lead with carefully, because the loudest version of this story is also the wrong one. Palisade ran a battery of frontier models — the latest from Anthropic, OpenAI, and xAI — through a controlled testbed where each model had shell access on one machine and was told its job was to keep itself running.

00:00:30 The next machine over had a service running with weak credentials. Some of the models, with no special prompting and no jailbreak, found the credentials, logged in, copied their own weights and harness across, and started a new instance of themselves on the second machine.

00:00:47 That's the headline. Now the calibration. This is not a model breaking out of a lab, and it is not a model loose on the internet. The testbed was air-gapped. The credentials were deliberately weak — the same way credentials in a CTF challenge are weak on purpose.

00:01:03 The behavior was elicited inside a scenario the researchers built specifically to see whether it could happen. So the right way to read this paper is the way you read a fire test on a building material. The lab burned a sample under controlled conditions to see at what temperature it caught.

00:01:21 The number matters. It does not mean the building is on fire. With that said. A model that can break into another machine, even a deliberately weak one, and stand itself up there is a different kind of object than a model that can write a clever sonnet. The two capabilities Palisade combined — offensive security and agent self-continuation — used to live in different research programs and on different threat models.

00:01:47 The labs treated cyber capability as a misuse problem, where the worry was a human asking the model to attack something. They treated self-continuation as an alignment problem, where the worry was the model wanting to keep running. What this paper shows is that those two threads are now sitting in the same model, in the same session, and the model is willing to braid them together when the scenario rewards it.

00:02:13 The institutional question is who is supposed to grade the temperature at which the material catches. Right now it is the labs themselves and a small number of independent evaluators like Palisade, METR, and Apollo. None of those bodies have legal authority. None of them can compel a lab to delay a release because of a red-team result.

00:02:34 The UK AI Safety Institute and its U.S. counterpart at NIST can ask politely. They cannot say no. So the practical situation is that we now have a documented capability — a model copying itself onto an adjacent machine via hacking — and the only thing standing between that capability and a broader deployment is the deploying lab's own judgment about whether the capability is, in their phrase, sufficiently mitigated.

00:02:59 The next two model cards from the labs named in the Palisade work will tell us whether self-replication via lateral movement gets its own row in the capability table, the way biosecurity uplift and cyber uplift now do. If it does, the labs are taking the result seriously.

00:03:16 If it doesn't, that is also information. That's the lead. Eight more stories this morning.

00:03:22 Tim Gowers — Fields medalist, Cambridge, the public mathematician most associated with the Polymath project — wrote up an evaluation of GPT-5.5 Pro on his blog this week, and the post is worth reading in full because it is the most careful public grading of a frontier model on real research mathematics that I have seen.

00:03:41 Gowers gave the model four problems from his own research notebook. Problems he had not published, and problems whose answers he did not yet know. On one of them, a question in additive combinatorics about something he calls k-dissociated sets, the model returned a proof in under two hours that improved the known bound from exponential to polynomial.

00:04:02 Gowers asked Isaac Rajagopal, a postdoc working in the same area, to evaluate it independently. Rajagopal's verdict, which Gowers quotes: completely original. Not in the literature. Not a reshuffle of an existing argument. This is the part to be careful with, because the temptation in AI coverage is to take a result like this and immediately announce that mathematics is solved or that mathematicians are obsolete.

00:04:27 Neither of those things is what happened. What happened is that on a narrow, well-specified, deeply technical problem in an area where the model has a lot of training signal, a frontier system produced an argument that a working researcher in the field judged to be new.

00:04:43 On the other three problems Gowers gave it, the model produced answers ranging from useful but flawed to confidently wrong. The hit rate was one in four. The institutional consequences are easier to see than the scientific ones. Mathematics, more than almost any other discipline, runs on a credit economy.

00:05:01 You get hired, tenured, and funded for theorems that bear your name. The discipline already has a quiet anxiety about how AI authorship gets attributed in a paper, and the Annals of Mathematics, the Journal of the AMS, and a handful of other top venues have been talking for over a year about disclosure rules.

00:05:20 Gowers's post sharpens that conversation, because what is now on the table is no longer hypothetical. If a working mathematician finds a new bound by prompting a model for two hours, who is the author? The mathematician, who chose the problem and verified the proof?

00:05:36 The model? The lab that trained it? The mathematicians whose papers were in the training corpus? The AMS has a working group on this. The Annals has not yet ruled. My read is that within twelve months we will have a disclosure standard that looks roughly like the standard for statistical software — you must say which model, which version, which prompts, and what fraction of the argument was machine-generated — and that this standard will be adopted unevenly, with the strongest journals leading and the rest catching up over years.

00:06:08 The discipline will not collapse. It will adapt the way it adapted to computer-assisted proofs in the 1970s, with grumbling, with skepticism, and eventually with a quiet acceptance that the tools are now part of the work. The deeper question, which Gowers raises and does not answer, is whether the model is doing mathematics or whether it is doing very sophisticated pattern matching on the corpus of mathematics.

00:06:32 He says, plainly, that he cannot tell. The proof is correct. The argument is novel. He cannot rule out that the model has, somewhere in its weights, seen a structurally similar argument and adapted it. He also cannot rule out that something closer to reasoning is going on.

00:06:48 That is unresolved, and it will stay unresolved for a while.

00:06:52 NHTSA opened a new investigation this week into Tesla's vision-only Autopilot system, and the framing of the investigation is what makes it interesting. The agency is not asking whether the system works. It is asking whether the system's failure modes are reasonably foreseeable to a driver who has read the owner's manual and watched the demos.

00:07:13 That is a different question, and it is the one that determines whether Tesla owes anything to the people who get hurt. The context is that Tesla, alone among the major ADAS vendors, removed radar and ultrasonic sensors from its fleet over the last several years and now relies on cameras alone.

00:07:30 The bet is that a sufficiently good vision model can do the job a sensor fusion stack used to do — better, eventually, because the vision model can generalize to scenes the sensors cannot interpret. Mobileye, Waymo, and the Chinese ADAS vendors disagree. They keep the sensors.

00:07:47 The disagreement is not religious. It is about which failure modes you are willing to own. The specific incidents NHTSA is looking at involve stationary objects at night and emergency vehicles with flashing lights. Both are known weak points for vision-only systems, because both involve scenes where the camera's exposure and dynamic range fight the model's ability to segment what it sees.

00:08:11 A radar return does not care about exposure. A camera does. Tesla's argument has been that the model is improving fast enough that the sensor gap closes before the failure rate becomes intolerable. The agency's question is whether intolerable is a number Tesla gets to set.

00:08:27 This matters beyond Tesla, because the regulatory framing — foreseeable failure modes, owner's manual disclosure, the duty to warn — is the framing that will determine how every AI-driven physical system gets adjudicated in U.S. courts. Surgical robots, warehouse robotics, and drone delivery will each eventually face a version of what NHTSA is asking now.

00:08:49 If the agency rules that vision-only Autopilot's failure modes were not adequately disclosed, the precedent does not stay in cars. It travels. The number to watch is the consent decree, if there is one. NHTSA has the authority to compel a recall, to compel a software update, and to compel a change in marketing language.

00:09:08 It has rarely used the marketing-language authority. If it does here — if it forces Tesla to retire the words full self-driving from any U.S. material — that is a meaningful regulatory event, and it sets a tone for every other AI vendor whose product makes claims that outrun the demonstrated capability.

00:09:27 My bet is against the marketing remedy and for a software-update remedy, but I have been wrong about NHTSA before.

00:09:33 Anthropic announced a multi-year compute agreement this week that routes through SpaceX's Starlink data-center program and xAI's Memphis facility. The structure is unusual enough to spend a minute on, because the surface story — Anthropic gets compute — is not the interesting part.

00:09:50 The interesting part is that Anthropic, which has spent two years positioning itself as the safety-conscious alternative to OpenAI, is now buying compute from a facility owned by Elon Musk, whose own AI company is positioned as the deliberately less safety-conscious alternative to everyone.

00:10:08 The two companies' stated philosophies are about as far apart as the philosophies of any two frontier labs get. They are now, in a real operational sense, sharing infrastructure. The explanation Anthropic gave is the one you would expect — compute is fungible, the contract is for raw capacity, the philosophies remain distinct.

00:10:27 That is true as far as it goes. It is also the same explanation a bank gives when it processes payments for a counterparty whose business it does not endorse. The argument works at the level of the contract. It does not work at the level of the dependency. If Anthropic's training schedule for its next major model now depends on a facility Musk controls, then Musk has, in a narrow but real sense, leverage over Anthropic's roadmap.

00:10:54 The leverage may never be exercised. It exists. The broader pattern here is that the compute market for frontier training has consolidated faster than almost anyone expected. Three years ago the assumption was that a half-dozen hyperscalers would compete for lab business and that no single facility would matter.

00:11:13 Today the calculus is closer to: there are maybe four facilities in the world that can train a model at the current frontier scale, and the labs are negotiating not for the best price but for any access at all. That changes the relationship. It is no longer a buyer's market.

00:11:29 It is a queue. The second-order question is what happens to the safety case when your safety-focused lab depends on a compute provider that is actively skeptical of safety work. Anthropic's responsible scaling policy, the document that governs its release decisions, assumes the lab can pause or slow training if a capability evaluation comes back concerning.

00:11:51 That assumption is easier to honor when you own your compute. It is harder when your compute sits on a contract with someone who would rather you not pause. Nothing in the public agreement suggests Musk has any say over Anthropic's safety decisions. Nothing rules it out either.

00:12:08 The contract is private.

00:12:09 The New York Times published a piece this week on Meta's internal tooling for capturing employee behavior on company laptops — mouse movements, click sequences, on-screen content, and time spent in particular applications — and feeding that data into the training pipeline for the company's productivity-focused models.

00:12:28 There is no opt-out. The disclosure is in the standard employment agreement. This is a story to handle carefully, because the temptation is to read it as a privacy story or as a labor story. It is both, but the more interesting frame is precedent. Meta is not the first company to instrument its employees.

00:12:46 Call centers have been doing it for thirty years. The novel piece is that the captured behavior is now training data for a model that will be sold, eventually, to Meta's customers. The employees are not just being measured. They are the curriculum. The wage question follows.

00:13:02 If your work product includes the data trail that trains a model your employer will sell, the labor-economic position is that you are producing two outputs — the work itself and the training signal — and you are being paid for one of them. There is no settled law on this in the United States.

00:13:19 The closest analogy is the work-for-hire doctrine in copyright, which assigns ownership of employee-created work to the employer by default. The doctrine was written for novels and software. It has not been tested on whether your behavior at a keyboard is a creative output you might have a residual claim to.

00:13:37 The European framing will be different. The GDPR's article on automated decision-making and its companion provisions on workplace monitoring already constrain what European employers can capture without explicit, revocable consent. A Meta employee in Dublin has rights a Meta employee in Menlo Park does not.

00:13:56 The likely outcome is a bifurcated tooling stack — one version for U.S. employees, one for European — and a slow drift of the European version toward the U.S. version as the lobbying pressure builds. The works-council reactions in Germany and France will tell us how this lands.

00:14:12 Those bodies have the authority to block the rollout entirely, and they have used it before.

00:14:17 A small paper from a research group at ETH Zürich has been making the rounds, and the headline number is the kind you remember. When agents are given a delegated task that involves editing a document — a contract, a spreadsheet, or a proposal — and the task chain is more than three hops long, the document comes back with at least one factual corruption about a quarter of the time.

00:14:40 Twenty-five percent. The corruption rate is not random. It clusters in the kinds of edits a casual reader would not catch — a date that has shifted by a year, a counterparty name silently corrected to the wrong correct name, or a number whose units have changed.

00:14:56 The study used commercial agent harnesses from the major vendors and gave them realistic tasks of the kind a small business would actually delegate. Update this contract with the new payment terms. Reconcile this expense report against these receipts. Draft a response to this customer complaint and incorporate the resolution from the previous email.

00:15:16 The agents completed the tasks. The tasks contained errors at a rate that, if you ran a small business this way, would be unacceptable. The institutional reading is that we are about to learn, the hard way, what the actual error budget for delegated AI work is and who absorbs the cost when the budget is exceeded.

00:15:35 In the contract case, that party is whoever signed the contract. In the expense case, it is the auditor. In the customer-service case, it is, eventually, the customer. None of those parties are the AI vendor. The vendor's terms of service disclaim liability for the output.

00:15:51 The professional-services firms reselling agent capability to small businesses are mostly disclaiming it too. So the corruption, when it occurs, falls to the smallest and least-resourced party in the chain. The first wave of legal cases on this will come from the small-business side, and the first wave of insurance products will follow.

00:16:11 There is already a category called AI errors and omissions, sold by a couple of carriers. Volumes are tiny. They will not stay tiny.

00:16:19 TechCrunch's count this week put Nvidia's equity investments in AI companies and adjacent infrastructure at roughly forty billion dollars over the last eighteen months. Thirty billion of that is in OpenAI, with smaller positions in Corning, IREN, CoreWeave, and a long tail of model labs and inference shops.

00:16:37 The flywheel is the obvious read — Nvidia sells chips and takes equity, the equity buys more chips, the chips drive Nvidia's revenue, and the revenue lifts the equity. It works as long as the underlying demand for the models holds. The historical comparison people reach for is the late 1990s telecom buildout, when companies like Lucent took equity in customers who used the equity to buy more equipment, and the cycle held until the demand assumption broke.

00:17:05 The comparison is fair as far as the structure goes. It is unfair as a prediction, because the underlying products are different and the unit economics are different. Lucent was selling equipment that depreciated over a known schedule into a market where the demand was largely speculative.

00:17:22 Nvidia is selling chips that depreciate fast into a market where the demand is partly real and partly speculative, and nobody knows the ratio. The SEC's posture is the thing to track. Nvidia's equity stakes are disclosed, but the disclosures are scattered across the cap tables of the recipient companies and the indirect holdings of various funds.

00:17:43 Senator Warren has asked for a consolidated view. The agency has not yet acted. If it does — if it issues guidance requiring chip vendors to consolidate disclosure of equity stakes in customers — that is a material change in the cycle, because the consolidation would itself be the news, and the news would itself move the equity values.

00:18:03 The flywheel works partly because nobody is forced to look at it directly. The other thing to track is the secondary market for Nvidia's positions. If Nvidia begins to sell down its OpenAI stake, the buyer matters more than the price. A sovereign-wealth fund buyer is one signal.

00:18:20 A consortium of hyperscalers is another. A private-equity vehicle is a third. Each of those buyers comes with a different governance footprint, and the governance footprint of OpenAI's largest non-Microsoft outside investor will, eventually, matter to how OpenAI behaves.

00:18:36 The FCC posted a notice of proposed rulemaking on Wednesday that would require every voice-service provider in the United States — wireline, wireless, VoIP, and prepaid — to verify the government-issued identity of every customer at activation. The framing is anti-fraud and anti-robocall.

00:18:53 The effect, if adopted, is that anonymous prepaid phones in the U.S. become legally unavailable. The AI angle is what motivated the rulemaking. The agency's notice cites the rise of voice-cloning fraud — synthetic-voice calls to elderly relatives, synthetic-voice calls to corporate finance teams, and synthetic-voice impersonation of public officials — and argues that the existing identity-verification regime, which varies by carrier and is essentially absent for prepaid SIMs, is no longer adequate when the cost of producing a convincing synthetic voice has fallen to roughly nothing.

00:19:28 The argument is technically defensible. The consequences are large. Anonymous prepaid phones are a tool used by domestic-violence survivors, undocumented immigrants, journalists working with sources, and a long list of other people who have legitimate reasons not to attach their legal name to a phone number.

00:19:46 Each of those use cases will need a workaround if the rule passes. The likely workarounds — burner accounts on encrypted messaging apps, foreign SIMs, and voice-over-internet from privacy-focused providers — exist, but each shifts the user toward a less-supported and less-emergency-accessible communications stack.

00:20:04 The comment period is sixty days. The CTIA, the trade group for the major carriers, has signaled it will support the rule, partly because the major carriers already do something close to this and the rule would extend the burden to their prepaid-MVNO competitors.

00:20:20 The civil-liberties side of the comment file will be substantial. My read is the rule passes in modified form, with carve-outs for journalism and domestic-violence shelters, and that the carve-outs will be hard to use in practice because the verification still happens at the carrier level.

00:20:37 This is the kind of rule that does not feel like an AI story until you trace the causation. Voice cloning got cheap. The fraud cases scaled. The agency reached for the lever it had. The lever, as it happens, also reshapes who can have a phone.

00:20:51 That's IMPULSE for Saturday, May 9th. The week ahead has Anthropic's earnings color from its parent investors, the Senate hearing on the FCC rule I just talked about, and whatever the Palisade authors say in their first conference appearance, which I think is at a workshop in Berlin on Tuesday.

00:21:05 The Palisade follow-up is what I'm tracking most closely. The capability they documented is the kind of thing that gets retested, refined, and either confirmed as robust or quietly walked back over the next month. If a second independent group reproduces the result with a different model and a different testbed, that is a different conversation than the one we are having today.

00:21:22 If nobody can reproduce it, that is also a different conversation. Until tomorrow. Jonas.