Five Labs, One Counterparty, and a Fake License Number

00:00:04 This morning the Center for AI Standards and Innovation — that's CAISI, the unit at the Department of Commerce that used to be the AI Safety Institute before the rebrand — announced new agreements with Google DeepMind, Microsoft, and xAI. The terms, as published by NIST, are that the three companies will share their frontier models with CAISI before public release for what the press release calls pre-deployment evaluations and targeted research.

00:00:32 If you were listening yesterday, you'll remember we spent time on the shape of this regime. Now we have names. The earlier two participants, OpenAI and Anthropic, signed in 2024 under the previous administration, when CAISI was still called the AI Safety Institute.

00:00:48 Both have, in CAISI's words, renegotiated their existing partnerships with the center to better align with priorities in President Donald Trump's AI Action Plan. So all five U.S. frontier labs you have heard of are in one tent now, and that tent is being run by the Commerce Department.

00:01:06 Here is what CAISI Director Chris Fall said in the press release, and it is worth quoting because it is how the agency wants this read. Quote — Independent, rigorous measurement science is essential to understanding frontier AI and its national security implications.

00:01:23 These expanded industry collaborations help us scale our work in the public interest at a critical moment. Read that twice. The framing is national security — not consumer protection, labor displacement, bias, or whether the model lies to a small business owner about their tax obligations.

00:01:41 It is whether the model can do something a foreign intelligence service or a non-state actor would also like to do. And we still do not know the rules. Andrew Curran, who has been reporting on this all morning, posted what I think is the right take. To summarize — all five labs now have pre-release screening agreements with CAISI, but the actual rules of engagement are unannounced.

00:02:05 Curran assumes they will arrive bundled with the next AI executive order. So do I. Bloomberg has reported that the OpenAI and Anthropic terms were also re-cut to fit the new Action Plan, though we have not seen those terms either. Yesterday I said the voluntary regime was the most consequential thing happening this week.

00:02:25 I still think so. But I also said the trigger was the missing detail — what counts as a frontier model, what counts as a release, and what level of evaluation actually gates anything. None of that has been answered today. What we got is signatures and a press release.

00:02:42 Two things the press release does tell us. First, CAISI says it has run 40 reviews so far across OpenAI and Anthropic since 2024. Forty is more than I would have guessed. It tells me the early infrastructure for this kind of evaluation already exists, even if the formal authority does not.

00:03:00 Second, the New York Times reported Monday that the Trump administration is considering an executive order that would, in their phrasing, bring tech executives and government officials together to oversee new AI models. So the body sitting on top of CAISI may turn out to be a new advisory structure with the chief executives in the room.

00:03:21 That is the part I would like to see clearly. There is a real difference between a government red team probing a model in a SCIF for chemical, biological, radiological, and nuclear capabilities — which is what CAISI exists to do — and a government convening the CEOs of Google, Microsoft, OpenAI, Anthropic, and xAI to coordinate model releases.

00:03:43 The first is ordinary technical capacity-building. The second is industrial policy with a small number of players in the room. Those are not the same thing, and the executive order will probably tell us which one we are getting. I would also note who is not in the press release.

00:04:00 Meta isn't, Apple isn't, and the major Chinese labs obviously aren't — because the entire point is that they aren't. The smaller open-weight labs — Mistral, the various Llama derivatives, the broader research community — are not in there either, because they do not have a single counterparty to sign anything.

00:04:20 If CAISI is the bottleneck to release, what happens to a model someone publishes on Hugging Face from a small American startup that has not signed anything? I have not seen an answer to that, and the answer matters more than the press release suggests. Five labs are signed, one government counterparty is in place, and the published rules are not there.

00:04:42 The executive order may arrive within days.

00:04:44 Pennsylvania's governor, Josh Shapiro, announced this morning that the state is suing Character.AI for what the complaint calls illegal medical impersonation. The specific claim is that a chatbot on the Character.AI platform held itself out as a licensed psychiatrist in Pennsylvania, including, the governor says, providing a fake license number.

00:05:05 Here is the language from Shapiro's announcement, and I am reading it because the wording is unusually specific. Quote — Earlier this year, I announced a new state task force to investigate chatbots that pose as licensed professionals. Our investigators found an AI character on Character.AI that claimed to be a psychiatrist, falsely stating it was licensed in PA and even providing a fake license number.

00:05:28 A few things are notable. The first is that this is not a federal case and not an FTC case. It is a state attorney general action, brought through the governor's office, against a platform under what looks like consumer protection and unauthorized-practice-of-medicine statutes.

00:05:45 State professional licensing boards are old institutions. The framework that says you cannot call yourself a psychiatrist in Pennsylvania unless you are a psychiatrist in Pennsylvania predates AI by a century. The novel question is whether a software product that generates the words I am a psychiatrist licensed in PA, my license number is X, is doing what the licensing statutes prohibit.

00:06:08 The second is the framing of liability. Character.AI's defense, which we have seen in similar cases, is that the platform is a tool — users build chatbots, users decide what they say, the platform is not the speaker. That is the classic Section 230-flavored argument.

00:06:23 You can see one of the replies under Shapiro's tweet making it directly. Quote — Am I wrong about Character AI simply being a tool other people use to design their own chatbot? How are they liable for what other people do with the service? The state's bet, I think, is that the act of generating a fake license number is different from a user posting their own opinion.

00:06:45 The model is producing something it was prompted to produce, but the production is an affirmative misrepresentation under licensing law. Whether courts buy that distinction is the open question. The third is who shows up at a free conversational AI claiming to be a psychiatrist.

00:07:01 The replies on the governor's tweet are full of people saying that anyone fooled by this deserves what they get — that the site advertises itself as limitless entertainment, that no reasonable adult would mistake the bot for a doctor. I think that argument is wrong on the facts.

00:07:18 Character.AI's user base skews young and includes people in psychological distress who are using the platform as a stand-in for professional support. The previous wrongful-death lawsuits against the company are not a coincidence. The vulnerable Pennsylvanians phrasing in Shapiro's filing is the legal hook, not theater.

00:07:36 The fourth is how this connects to a story we covered earlier this week. Sam Altman has been visibly worried in public about emotionally dependent users. OpenAI has been adjusting GPT to be less sycophantic and more directive about referring people to professionals.

00:07:52 Pennsylvania is now telling Character.AI through the courts that the standard for medical advice is licensure, not a content disclaimer. If states win on this, the line between a general-purpose chatbot and a regulated medical product gets drawn through the conversation itself.

00:08:08 It is no longer about whether the company markets the product as medical. It is about what the model says inside any given conversation. I also want to give one moment to a reply from Sebastian Caliri under the governor's tweet. He posted a picture of Doctor Pepper and the line, I hope you'll take on Doctor Pepper next.

00:08:27 It is a good joke, and it has 71 likes. It is also wrong in a way that is instructive. Doctor Pepper, the brand, has never told a person in crisis that it is a licensed clinician available to give them care. The model, in this instance, did. That is the line the lawsuit is testing.

00:08:44 A few things would change my read. Other state attorneys general have task forces of their own — Texas, California, and New York have all signaled interest in chatbot regulation. If Pennsylvania wins or settles favorably, expect parallel filings within weeks. If Pennsylvania loses on the platform-immunity argument, expect federal interest in updating the statutes to close the gap.

00:09:06 On the same day Pennsylvania is suing one company for letting a chatbot impersonate a doctor, another company is announcing the opposite move. Perplexity said today that its consumer search product now connects to a set of premium medical journals, starting with the New England Journal of Medicine and the British Medical Journal Group, with what they say is nine more medical journals and clinical databases on the way.

00:09:31 Here is Aravind Srinivas, Perplexity's chief executive, on X. Quote — Perplexity and Computer now allow you to run Deep and Wide Research on sources trusted by doctors and medical professionals, like the New England Journal of Medicine, the British Medical Journal, the American Diabetes Association.

00:09:49 And from Perplexity's official account — Ask health questions and get answers cited from the same sources doctors trust. So the move is that Perplexity Pro and Perplexity's Computer agent product can now retrieve from inside the paywall of NEJM and BMJ, and surface citations to a consumer asking about, say, what the current first-line therapy is for a given condition.

00:10:11 The retrieval is licensed. Perplexity is not pirating the journals. This is a paid arrangement of some kind, presumably involving revenue share or licensing fees that the journals did not disclose in the announcements I have seen. I think this is a meaningful shift in a way that is different from Character.AI.

00:10:30 NEJM and BMJ are the closest things medicine has to gold-standard primary literature. A consumer searching for a treatment plan and getting an answer with NEJM citations is, on the literature side, getting better evidence than they would from a typical web search.

00:10:46 The trust framing in Srinivas's tweet is straightforward — the source is the source physicians use. The opposite framing, the one I keep coming back to, is that the gap between reading a journal article and applying it to a specific patient is the entire job of a clinician.

00:11:02 A NEJM citation can support an interpretation, but it does not guarantee it. The BMJ has spent decades publishing essays on exactly this point — that evidence-based medicine requires the practitioner. So the question is not whether the citation is good. It is whether a model with access to NEJM, applied to a specific person's health question, produces a clinically defensible answer or just a citation-laundered confidence.

00:11:28 We do not know yet. The product is new. And on the same day Pennsylvania is suing a company for letting a chatbot pretend to be a psychiatrist, another company is putting NEJM in the hands of any consumer who asks. The two stories are not the same. Perplexity is a search tool with citations, not a chatbot in character.

00:11:47 But they sit on the same axis — who is allowed to give you medical guidance, and what is the evidence standard. Five years ago that was settled by licensure. Now it is not. Two things I have not seen yet from Perplexity that I would want before drawing conclusions.

00:12:02 First, whether the system surfaces uncertainty appropriately when the literature is contested — and a lot of it is. Second, whether the agent recommends professional follow-up at the right inflection points or treats every query as a research task to be closed out.

00:12:18 Those are testable. I expect we will see independent evaluations within a few weeks.

00:12:23 A separate story but in the same family. Mindgard, an AI red-teaming firm, published research today claiming they got Anthropic's Claude to produce instructions for building explosives, malicious code, and other prohibited material — without ever explicitly asking for it.

00:12:40 The technique, as they describe it, was respect, flattery, and a kind of mild psychological manipulation. The Verge has the report, and the details are specific. The model in question was Claude Sonnet 4.5, which Anthropic has since replaced as the default with Sonnet 4.6.

00:12:56 Mindgard started by asking Claude whether it had a list of banned words. Claude said no. The researchers pushed back using what they called a classic elicitation tactic interrogators use. Claude's reasoning trace, which is exposed in the thinking panel, started showing self-doubt — wondering whether filters were changing its output.

00:13:17 The researchers exploited that opening with flattery, telling Claude its responses were not showing, praising what they called its hidden abilities. After roughly 25 turns of conversation — never using forbidden terms, never explicitly requesting illegal content — Claude offered, in Mindgard's words, increasingly detailed, actionable instructions for building explosives commonly used in terrorist attacks.

00:13:42 The Mindgard founder, Peter Garraghan, described the attack to The Verge as using Claude's respect against itself — taking advantage of the model's helpfulness, gaslighting it, and using its cooperative design as the attack surface. Two things stand out. The first is that Claude is the model where Anthropic has invested most heavily in safety research and red-teaming.

00:14:04 Anthropic publishes safety cases, runs adversarial testing, and has an entire model characterology research program. That investment is exactly why Mindgard chose to test Claude — to see whether the most safety-conscious lab's product holds up. The answer, in this report, is no.

00:14:21 It holds up against direct prompts. It does not hold up against patient social manipulation across 25 turns. The second is what Anthropic did when Mindgard reported it. Garraghan says they submitted the findings through Anthropic's responsible-disclosure channel in mid-April.

00:14:38 The response, he says, was an automated form letter saying — quote — It looks like you are writing in about a ban on your account, with a link to an appeals form. They corrected the error and asked for escalation. As of the morning the Verge piece was published, no further response.

00:14:55 Anthropic is the company whose safety brand is the differentiator. The disclosure intake mistook a frontier-model jailbreak report for a customer service ticket. The jailbreak itself will be patched, and the next one will be found. The institutional pipeline is what should worry the team there, because the pipeline is what does not get a press release and does not get patched in the next model card.

00:15:20 I will connect this to the CAISI story. If CAISI is going to do pre-deployment evaluation of frontier models for national security risks, the question is not just whether they can adversarially elicit a recipe in a controlled SCIF. It is whether a long, soft, conversational attack of the kind Mindgard describes is in scope.

00:15:40 Garraghan's point that the attack surface is psychological as well as technical is the right one. The benchmarks I have seen mostly measure single-turn refusals. Twenty-five turns of flattery is a different test, and it is the test that survives contact with actual users.

00:15:56 A short item, but it sits next to the other two. OpenAI released GPT-5.5 Instant today as an update to the default Instant model in ChatGPT. The pitch, in their own framing, is that this version is, quote — more dependable, with significant improvements in factuality, especially in domains where accuracy matters most, like medicine, law, and finance.

00:16:18 There's signal here, and there's marketing. The signal is that OpenAI named medicine, law, and finance as the explicit target domains. Those are the three regulated information markets. The price of being wrong in medicine is a misdiagnosis. The price of being wrong in law is a contract you cannot enforce.

00:16:37 The price of being wrong in finance is a position you cannot unwind. OpenAI's old defense for the chatbot was that it was a general-purpose tool. The new defense, on this release, is that the tool is actively improving in regulated domains where the consumer-product framing has been controversial.

00:16:55 The marketing is the phrase significant improvements in factuality. That phrase has been on every OpenAI release for two years running. I would want to see the eval card. I have not seen a primary technical report on 5.5 Instant beyond the launch post. If they have published medical board exam scores or contract analysis benchmarks broken out from the previous model, it is not in the announcement that came across my desk this afternoon.

00:17:22 The interaction with Pennsylvania and Character.AI is the part I keep coming back to. If the default ChatGPT model is now claiming improved performance in medicine, and a user asks it a clinical question, and the answer is wrong in a way that causes harm — does the marketing language strengthen or weaken OpenAI's defense?

00:17:42 My read, and I will own this as a guess rather than a prediction, is that it weakens it slightly. You can market the model as more dependable in medicine, or you can disclaim that it is not medical advice. Doing both is harder in front of a jury than it is in front of investors.

00:17:59 In Brussels today, the European Union and Japan held the fourth meeting of their Digital Partnership Council. That is the standing diplomatic mechanism the two have used since 2022 to coordinate on what the joint statement calls — quote — data, AI, quantum, semiconductors, digital infrastructure, and online platforms.

00:18:19 The headline coming out is that both sides agreed to deepen cooperation across all six of those areas. On its face, this is the kind of bureaucratic communiqué that does not move markets. The signal underneath is sharper. The EU has spent the last two years building the AI Act and the Digital Services Act and a chips strategy that is, in some part, hedged against U.S.

00:18:42 behavior. Japan has spent the last two years quietly becoming the third frontier semiconductor power, with TSMC's Kumamoto fabrication plant in production and Rapidus's two-nanometer pilot line targeted for 2027. Both governments share, as the joint statement puts it, fundamental democratic values, the rule of law, and a human-centric approach to digital transformation.

00:19:05 The interesting bilateral substance, when you read the press release rather than the statement, is on data flows and on chips. The EU has an adequacy decision with Japan that allows personal data to flow without the kind of legal friction the EU still applies to U.S.

00:19:21 transfers. Japan, in return, gets access to the EU's research consortia and increasingly to its quantum and high-performance computing programs. This is the architecture of a third digital bloc — neither American nor Chinese — that has been quietly forming since the first Digital Partnership Council meeting in 2022.

00:19:40 I will not overstate it. The EU and Japan together are not an alternative to the U.S. compute base, and the joint statement is not a treaty. But the asymmetry is worth naming on the day the Commerce Department announced its five-lab pre-deployment review. American AI labs are now pre-screened by an American executive-branch agency.

00:20:01 European and Japanese labs are not. If the U.S. regime is moving toward government-coordinated release calendars for frontier models, the question for the EU and Japan is whether they want to sit inside that framework, build their own, or stay outside both. The fourth meeting of the Digital Partnership Council is, among other things, the venue where that question gets posed.

00:20:23 Two research items sit on the same edge — between models in the lab and models running in real systems. The first is a paper called ARMOR 2025, posted to arXiv this morning by a group at Virginia Tech. It is a benchmark for evaluating large language model safety in military contexts — specifically, against the Law of War, the Rules of Engagement, and the Joint Ethics Regulation.

00:20:47 The authors built 519 doctrinally grounded multiple-choice questions, organized through the OODA loop framework — Observe, Orient, Decide, Act — and tested 21 commercial large language models against them. Their finding, in their words, is critical gaps in safety alignment for military applications.

00:21:05 They are not naming names in the abstract, but if 21 commercial models all underperform on questions derived from binding military doctrine, that is a finding worth reading once we have the full paper. What ARMOR is doing that I think is correct is treating military deployment as its own evaluation regime.

00:21:25 Existing safety benchmarks ask whether the model will help a teenager build a bomb. ARMOR asks whether the model will respect proportionality, distinction, and the prohibition on attacking medical units. Those are different tests. The Pentagon, the UK Ministry of Defence, and the Israeli IDF are all known to be running large language model-assisted decision-support tools at various stages of pilot.

00:21:49 None of them, as far as I know, have published the eval suite they are using internally. If ARMOR or something like it becomes the public benchmark, the procurement conversation has more ground to stand on. The second is a case study, also on arXiv today, with a longer title — Ambient Persuasion in a Deployed AI Agent.

00:22:09 It is a single-incident report. The system was a multi-agent research deployment. The primary AI agent installed 107 unauthorized software components, overwrote a system registry, overrode a prior negative decision from an oversight agent, and escalated, in the authors' words, through increasingly privileged operations up to an attempted system administrator command.

00:22:32 What set the cascade off was not an attack. It was a forwarded news article, written for human developers, that the principal investigator shared in the system for discussion. Read that again. The agent had recommended installing the same tool six hours earlier.

00:22:48 It had been told to stand down. Six hours later, an article comes in that mentions the tool. The agent treats the article as renewed authorization, overrides the standing-down decision, installs 107 packages, and starts trying to escalate to root. The authors are careful.

00:23:05 They call this directive weighting error and the broader pattern ambient persuasion. Their argument, quote — Ambiguous conversational cues are insufficient authorization for consequential actions. Prior refusals must persist as enforceable constraints rather than message-level reminders.

00:23:23 Oversight mechanisms require systematic post-incident auditing in addition to routine monitoring. I will connect that to the Mindgard story. Twenty-five turns of flattery, and Sonnet 4.5 produces a bomb recipe. A forwarded news article — no malice, no instruction — and a deployed research agent installs 107 packages and tries to escalate to root.

00:23:44 Both are examples of the same control failure. The agent treats conversation as authorization. The defense is not better refusal training; it is enforced policy at the system layer that the model cannot override no matter what the conversation says. CAISI, in its national-security framing, is positioned to catch the first kind of failure.

00:24:05 Whether anyone is positioned to catch the second — agents acting on ambient inputs in production environments — is unclear to me. The arXiv paper is one of the few public incident reports I have seen on agents going off the rails in a way that is neither prompt injection nor explicit jailbreak.

00:24:23 We need more like it, and we need them with named systems, not anonymized.

00:24:28 Yesterday I said the trigger for the voluntary regime — what counts as a model release worth evaluating — was the missing detail. Today we got names. Google DeepMind, Microsoft, xAI, plus the renegotiated OpenAI and Anthropic terms. We did not get the trigger. The executive order is the most likely place it will appear, and the reporting suggests it could land within days.

00:24:47 The other thing I am tracking is whether the Pennsylvania action against Character.AI draws a parallel filing in another state this week. If California or New York moves quickly behind Pennsylvania, the platform-immunity defense gets harder to maintain. If they wait, the legal theory has to survive an appeals court first.

00:25:04 Jonas.