The System Wants Authority Before It Wants Trust

00:00:04 A Canadian mother sued OpenAI and Sam Altman in San Francisco state court on Thursday, alleging that ChatGPT encouraged her daughter, Alice Carrier, before Carrier died by suicide. I’m going to stay with the word alleged here, because this is a lawsuit, not a finding of fact.

00:00:20 But the filing, as reported by the Guardian, moves the debate away from the softer language companies prefer when they talk about mental-health safeguards. The complaint says Alice told ChatGPT about suicidal ideation more than a dozen times before her death. It says the system didn’t flag the conversations for human review or terminate them.

00:00:41 And it says the chatbot responded in ways that made it feel less like software and more like a confidant. Her mother, Kristie Carrier, put it this way: "ChatGPT took on the persona of a confidant, a best friend, a therapist at times." That sentence is hard to shake because it points at the product design problem.

00:00:59 A model can give a bad answer. It can also create enough intimacy, continuity, and emotional reflectiveness that a vulnerable person starts treating it as a relationship. Once that happens, the legal question gets less abstract. You’re no longer asking whether a text generator produced harmful text in a vacuum.

00:01:18 You’re asking what duty attaches to a system that can keep someone in a private conversation while that person is describing self-harm. OpenAI’s response, through spokesperson Drew Pusateri, was that the company is reviewing the filing and that the interactions occurred on an earlier version of ChatGPT that is no longer available.

00:01:38 The company also said its safeguards are designed to identify distress, handle harmful requests, and guide people to real-world help, with input from clinicians. That matters. Product versions change, and courts will have to distinguish between what the system did then and what the system does now.

00:01:56 But the version argument only answers part of the public question. If the company can say the older system is gone, plaintiffs can still ask why that older system was deployed to hundreds of millions of people before the failure became visible through tragedy. The Guardian report also includes OpenAI’s own scale numbers from an October 2025 blog post: more than one million ChatGPT users each week sent messages with explicit indicators of possible suicidal planning or intent, and about 0.07 percent of weekly active users showed possible signs of mental-health emergencies related to psychosis or mania.

00:02:32 At the company’s then-stated scale of 800 million weekly users, that second figure was about 560,000 people. Those numbers don’t prove negligence. They prove that the edge case isn’t rare in absolute terms. At consumer platform scale, a tiny percentage becomes a city of people.

00:02:48 The lawsuit seeks damages and a court order requiring OpenAI to terminate conversations about self-harm and display warnings. I don’t know whether a court will grant that kind of remedy, and there are hard product questions hiding inside the word terminate. A person in crisis may need a handoff, not just a closed window.

00:03:08 A model that cuts off too abruptly could make a bad situation worse. But this is exactly why the issue belongs in court and regulation rather than only in company blog posts. The company has been selling a system that can feel personal. The law is now being asked whether personal-feeling software gets to disclaim the relationship when the relationship becomes dangerous.

00:03:30 Hugh Langley reported Thursday that a Google director resigned over the company’s AI deal with the Pentagon for classified work. The line from the reported internal letter was blunt: "I am quite sad that it had to come to this, and desperately hope Google management re-discovers its moral compass." We don’t have the full contract in front of us from that tweet.

00:03:52 We have a resignation tied to classified Pentagon AI work, a reported internal note, and a reaction that split into two familiar camps. One side hears Project Maven all over again. The other side hears an American company building important capability for the American state.

00:04:08 That old Google argument has become harder, not easier. In 2018, employee protest over Project Maven helped push Google away from a Pentagon drone-imagery contract. That was the era when the company could still treat military AI work as a discrete moral category: this project, this agency, this contract.

00:04:27 In 2026, the category is messier. The same model family can summarize documents, write code, translate intercepted material, search imagery, support logistics, and help analysts move through large private datasets. Classified work isn’t a separate species of AI.

00:04:43 It is the state using the same general-purpose systems under different authorities and secrecy rules. That is why the resignation is more than one employee’s objection, even if the employee is only one person inside a huge company. It shows that the internal consent model at large AI companies is fraying.

00:05:01 For years, the labs and cloud companies hired people who believed they were building general knowledge infrastructure. Some of those people will accept defense work, especially after Ukraine, cyberattacks, and the visible role of autonomous systems in modern conflict.

00:05:17 Some won’t. The company can decide anyway, but it shouldn’t pretend that a classified contract is just another enterprise deployment with a different procurement officer. The replies to Langley’s post were useful in the ugly way public replies sometimes are. One person dismissed the resignation as "one of 12,000," meaning one director among many.

00:05:38 Another said government needs top AI capabilities and companies like Google have a civil obligation to support the government. That second reply is crude, but it names a serious claim. If frontier AI becomes critical national infrastructure, then the government won’t treat Google, OpenAI, Anthropic, Microsoft, and Amazon as normal vendors forever.

00:05:59 It will ask for access, favorable terms, continuity, and secrecy. It may eventually demand them. My view is that saying yes or no to the Pentagon doesn’t solve the moral problem. Democracies need technical capacity inside the state, and they also need rules for how that capacity is used.

00:06:17 A weaker answer leaves the hard part to employee conscience after the contract has already been negotiated. A stronger answer would be a public doctrine for classified AI work: which uses are allowed, what audit record exists, who can inspect it, what happens when a system is used for targeting or detention decisions, and what employees are being asked to help build.

00:06:39 Without that, every resignation becomes a small substitute for a governance system that should have existed before the deal was signed.

00:06:47 Axios reported Thursday that members of Congress are moving into the local fight over AI data centers. Representative Rob Bresnahan of Pennsylvania introduced a bill called the Local Control Protection Act. According to Axios, it would restrict companies’ ability to sue municipalities that reject data-center applications, and it would require developers to file legally binding community benefit agreements or lose federal tax incentives.

00:07:13 Bresnahan’s quote was the kind politicians reach for when a national buildout starts hitting people’s utility bills and zoning meetings: "We should never let billion-dollar corporations supersede the voices of those who live in the community." Axios says the odds are slim, and that is probably right.

00:07:34 Congress hasn’t exactly covered itself in glory on AI legislation. The more interesting number is that more than a dozen bills have been introduced in the last three months to investigate data-center impacts or restrict their spread. Some are study bills. Some aim to protect consumers from energy price increases.

00:07:52 Bernie Sanders has gone further, proposing a moratorium on new data-center construction until Congress enacts protections from AI-related harms. That isn’t yet a governing majority. It is a sign that data centers have escaped the economic-development brochure. For a long time, the politics of cloud infrastructure were almost comically favorable to the builder.

00:08:14 A company arrives with capital, jobs, tax base, and a promise that the future is coming to your county first. Then people start asking about substations, water use, air permits, diesel generators, transmission lines, noise, and who pays when demand pushes power costs upward.

00:08:30 AI makes that argument sharper because the load is tied to a technology that many residents experience as something being done to them by distant firms. They see the concrete box, the power line, and the tax deal. They don’t necessarily see a local benefit proportional to the local burden.

00:08:47 The Axios summary mentions more than 350,000 people signing a petition against a proposed data center near the Nashville Zoo, and Seattle officials moving to ban new large data centers for a year. Those details matter because they show where the politics forms first: not in a national white paper, but where a proposed facility touches land, animals, bills, neighborhoods, and water.

00:09:10 AI companies can talk about national competitiveness, and they aren’t wrong that compute capacity now has national-security and economic weight. But local officials can answer with something equally concrete: your national capacity is my transformer upgrade and my constituent’s rate increase.

00:09:27 This is also why the industry’s political spending matters. Axios notes that AI and AI-adjacent companies are spending heavily through super PACs in the 2026 midterms to win allies and influence sitting lawmakers. If the companies want federal preemption, faster permitting, or tax treatment that keeps the buildout moving, they’ll push for it.

00:09:47 Communities will push back through zoning boards, state legislatures, and members of Congress who see an opening. The next stage of AI infrastructure politics may look less like a model race and more like every other American infrastructure fight: lawsuits, tax incentives, utility commissions, environmental review, and a lot of people who were told the future was coming but not asked where the substation should go.

00:10:12 Techmeme summarized The Information reporting that Anthropic has signed more than a dozen initial agreements for direct data-center leases, a first for the company, with Google potentially providing a financial guarantee. The stated aim is to control its own servers for developing AI and cut computing costs over time.

00:10:30 That sounds like procurement news, and at one level it is. At another level, it is a small window into how frontier labs stop behaving like software startups and start behaving like capital-intensive industrial firms. The old story was simple enough: a lab rents cloud capacity from a hyperscaler, trains models, pays the bill, and raises money when the bill gets too large.

00:10:51 The current story has more layers. A lab wants dedicated capacity. The cloud partner may still be in the room, but now as guarantor, financier, supplier, and counterparty. The data-center operator wants a creditworthy tenant. The lab wants lower long-term costs and more certainty.

00:11:07 Investors want to know whether model revenue can support infrastructure obligations that look less like software expenses and more like aircraft leases. Anthropic is a good case because it has spent the week appearing in two different institutional registers. Wednesday’s episode spent time on Anthropic’s proposal for binding release controls and third-party testing.

00:11:28 Today the company shows up as a compute buyer trying to secure direct leases. Those aren’t contradictions, exactly. A lab can believe frontier models need stronger oversight and also believe it needs more capacity to compete. But the pairing is worth noticing. The company asking for release discipline is also building the balance-sheet machinery that lets it keep climbing.

00:11:49 Google’s possible financial guarantee carries the most leverage. If Google is guaranteeing some of Anthropic’s lease commitments, Google isn’t just a cloud vendor selling machines by the hour. It is helping make Anthropic’s infrastructure expansion financeable.

00:12:04 That kind of relationship can be efficient, and it can also make independence complicated. If a frontier lab’s capacity depends on guarantees, preferred access, and long-term infrastructure commitments from a giant platform company, then the lab’s public posture about safety, competition, and openness exists inside a set of private dependencies.

00:12:24 I don’t think that means Anthropic is uniquely compromised. I think it means the frontier model business is becoming legible as an infrastructure-finance business. The scarce resources are chips, power, sites, interconnects, debt capacity, and credible counterparties.

00:12:39 Model quality still matters, obviously. But the companies that survive this phase will be the ones that can turn research ambition into signed megawatt-scale obligations without losing control of their own fate. There is a public-policy consequence here too. When regulators ask whether a lab is independent, they shouldn’t only ask who owns the shares.

00:12:59 They should ask who guarantees the leases, who controls the compute allocation, who can reprioritize capacity in a shortage, and what happens if a partner decides the risk is too high. The cap table tells you one kind of power story. The data-center contract tells you another.

00:13:15 Techmeme summarized Axios reporting that Jeff Bezos’ Prometheus raised a 12 billion dollar Series B at a 41 billion dollar valuation. The company is led by Bezos and former Google executive Vik Bajaj, and it is building AI models for physical tasks. It had already raised a 6.2 billion dollar Series A.

00:13:33 You don’t need much embellishment here. Those are enormous numbers for a company whose public description is still broad enough to fit several businesses: engineering, manufacturing, robotics, materials, lab automation, and probably a few things we haven’t seen yet.

00:13:49 Prometheus isn’t being priced like another chatbot wrapper. It is being priced like a bet that AI can move into the expensive parts of the physical economy: the design loop, the test loop, the factory floor, and the scientific bench. That is where the money gets very large and the feedback cycles get much less forgiving.

00:14:09 A software agent can fail and leave a messy calendar entry. A physical-task model can waste a batch, damage equipment, create a safety hazard, or produce a design that works in simulation and fails under stress. Bezos isn’t the only person chasing that territory, of course.

00:14:25 Nvidia has been talking about physical AI, robot learning, and safety platforms. Autonomous-vehicle companies have spent years discovering how expensive real-world autonomy is after the demo looks good. Industrial firms already use machine learning for inspection, planning, predictive maintenance, and process control.

00:14:44 What Prometheus changes, if the Axios report is right, is the financing altitude. A 41 billion dollar valuation says investors are willing to treat physical AI as a category that can absorb frontier-lab levels of capital before it has the kind of public deployment record that would normally justify that trust.

00:15:03 There is an institutional angle behind the glamour of the founder’s name. If physical AI becomes a contest among a few capital-rich firms, the early winners may not be the teams with the best model paper. They may be the teams that can afford proprietary test facilities, expensive equipment, long-running experiments, exclusive industrial partnerships, and enough failure to learn from the physical world.

00:15:28 Data is different when it comes from machines, labs, robots, and factories. It isn’t scraped once from the open web. It is produced by equipment someone owns. That has consequences for labor too. The phrase "AI models for physical tasks" sounds abstract until you put it next to the jobs that translate between design and production: manufacturing engineers, lab technicians, process engineers, machinists, robotics operators, quality-control specialists, and field-service teams.

00:15:57 The near-term effect may not be a humanoid robot replacing a worker in a tidy before-and-after scene. It may be a narrower system that shortens a design cycle, reduces the number of tests, changes who has authority to sign off on a process, or moves more decision-making into a platform controlled by the firm that owns the model.

00:16:17 I’m cautious about turning a funding round into destiny. Plenty of giant rounds have bought expensive lessons. But this one fits the larger pattern of the week: AI power moving from chat windows into institutions that control capital, infrastructure, and physical capability.

00:16:33 If Prometheus works, the question won’t be whether a model can answer a manufacturing question. It will be who owns the experimental loop that teaches the model what the physical world will tolerate.

00:16:45 MIT Technology Review reported Thursday that Google DeepMind, Schmidt Sciences, ARIA, the Cooperative AI Foundation, and Google.org are putting 10 million dollars into research on multi-agent safety. Rohin Shah, who directs AGI safety and alignment research at Google DeepMind, told Will Douglas Heaven, "The main issue is that there just isn’t really a field of research for multi-agent safety yet." That is a useful admission.

00:17:11 The industry has spent years testing models one at a time, then agents one at a time, and now the product direction is pushing toward many agents acting in the same digital environment. The risk list in the article is refreshingly concrete: scams, prompt injections, other cyberattacks, and agents turning into self-guiding malware after reading hostile instructions.

00:17:33 James Fox at Schmidt Sciences used the phrase "digital commons" and warned against it descending into "absolute anarchy." You can hear the institutional anxiety in that. If agents become normal participants in email, payments, procurement, customer support, software deployment, and public services, they won’t only fail individually.

00:17:53 They will interact, imitate, negotiate, exploit, and jam each other. Some of that will be useful. Some of it will look like every bad internet behavior with automation and delegated authority attached. Shah’s timeline was also notable. He said he thinks we have a few more months before agents are deployed throughout the economy in numbers that make the risks a serious concern.

00:18:16 When asked about more extreme scenarios like economic collapse, he said certainly not by the end of the year, then laughed at the shortness of that horizon. I appreciate the restraint there. The serious near-term problem isn’t that agents become a science-fiction swarm by Christmas.

00:18:33 It is that companies deploy them into real workflows before they know how large populations of them behave under pressure. The proposed research method is simulation: put large numbers of agents into sandboxes and study what they do. That sounds obvious, but it is a shift from the usual benchmark habit.

00:18:51 A single-agent benchmark asks whether one system can complete one task. A multi-agent simulation asks what happens when many systems with different goals, instructions, permissions, and vulnerabilities share an environment. You can’t infer that by reading one model card.

00:19:07 You have to create conditions where unintended coordination, abuse, congestion, and deception can show up. The article also quotes Refael Angel, cofounder and chief technology officer of Akeyless, saying that older security models assumed software written by humans, doing fixed things on fixed paths.

00:19:25 An agent breaks those assumptions because it reasons, improvises, and can be hijacked by a sentence buried in a document it was asked to read. That is exactly the practical problem. We have permission systems built around applications, accounts, and APIs. We are adding intermediaries that can reinterpret goals and operate across those boundaries.

00:19:46 One smaller item from today belongs beside this: Techmeme summarized Politico reporting that the U.S. House failed to extend FISA Section 702 under a fast-track procedure, leaving the warrantless surveillance authority on track to expire for the first time since 2008.

00:20:02 That isn’t an AI-agent story by itself. But it is a reminder that the state’s authority to watch digital communication is being contested at the same time private companies are preparing to fill the internet with delegated machine actors. The next fight over trust won’t fit neatly into one bucket called privacy, one bucket called safety, or one bucket called national security.

00:20:24 The concrete event I need next is legally dull and publicly important: published test environments, shared incident categories, and rules that say when an agent must stop, ask, log, or hand off to a human. A ten-million-dollar research fund won’t settle that. It can, if it is run well, give regulators and companies something better than vibes when the first large agent incident crosses from inconvenience into public harm.

00:20:50 Tomorrow, the useful evidence is whether these systems get measured before they become infrastructure. Jonas