Sunday: Codeberg's line, Chollet's benchmark, and what prompts actually change

00:00:04 A volunteer-run code hosting platform called Codeberg has updated its terms of use to ban projects that consist mostly of AI-generated code. The wording is specific: you must not share projects that mostly consist of code written by generative AI tools. There is no percentage defined in the public version of this policy — whether fifty-one percent or seventy-five percent counts as mostly, it seems left for moderation judgment on each review.

00:00:34 Copyright uncertainty is the other half of their rationale. Large language models train on code from every possible license, which means any AI-generated output carries an unclear provenance chain. Codeberg's members are drawing a boundary: human-authored work stays, ambiguous-origin code goes elsewhere.

00:00:55 This matters because it sits outside the usual debate about AI and code ownership. When a big lab says its models can't be used for certain outputs, that's a product constraint. When a small, community-funded platform makes the same call as a structural policy, it becomes a governance question — who decides what counts as mostly and why that boundary is worth maintaining.

00:01:20 Codeberg's been in the spotlight recently as a GitHub alternative, mostly because users are tired of the reliability hiccups and rough edges. The change isn't just philosophical; it's practical for developers who need a stable place to host work without copyright ambiguity.

00:01:39 The mostly threshold is tricky. Any line you draw — fifty-one percent, sixty percent, seventy-five percent — becomes a boundary condition that invites gaming. If you set it at about half, someone writes just under half by hand and flips the rest through an AI tool, pushes it past moderation, and the spirit of the rule is technically intact.

00:02:03 The copyright question is harder to game, but also harder to adjudicate in practice. Who owns a model's training distribution? Is that even a legal category yet? Codeberg isn't making this call because AI code is bad. It's making this call because the platform exists to serve people who are building things collaboratively, and ambiguous provenance introduces friction into that specific use case.

00:02:30 François Chollet — Keras creator, longtime researcher — has been arguing for a while that base language models have a fundamental limitation when it comes to reasoning. Today he made the point concrete again with a specific benchmark: ARC-1. ARC-1 is from 2019 and tests abstract reasoning on unseen tasks.

00:02:51 To this day, base large language models — meaning models without test-time compute, one forward pass per response — still perform poorly on it. Chollet notes that despite roughly 100,000x more parameters and data since 2019, the performance gap hasn't budged. His framing is worth reading carefully because he's not just saying models are bad at reasoning.

00:03:17 He's making an architectural argument. Base large language models and test-time-augmented systems (TTA) are different classes of model. A base large language model generates its answer in a single pass through the weights. A TTA system allocates additional compute at inference time — self-correction, chain-of-thought, search over reasoning paths.

00:03:41 The analogy he uses is apt: criticizing the limits of steam trains doesn't mean you can't appreciate electrified bullet trains that share surface similarities. Local models flatten this distinction into a single narrative when I ask them about it. They compress the gap into two camps — AI can't reason, or AI can reason with enough compute.

00:04:06 Both miss the structural point: the difference isn't just more data or bigger parameters. It's whether the model is allowed to think before it speaks. That last sentence is my own framing. The evidence from ARC-1 suggests Chollet has something real here — a persistent performance floor that scaling alone hasn't moved.

00:04:28 Amjad Masad — CEO of Replit — has a large language model chess engine playing autonomously on LiChess at one thousand two hundred fifty-three Elo. He posted this on Sunday afternoon after claiming about fifteen-hundred Elo for his web demo version the day before.

00:04:47 The system chains responses using an eight-billion-parameter model. It takes one or two seconds per move — fast compared to the thirty-second turns some frontier models require on the same task. In his earlier post, he noted it consistently beats frontier models and Stockfish level zero — a baseline chess engine that's been calibrated as easy mode for decades.

00:05:12 The Elo number itself isn't the headline here. Chess engines are well-studied territory, and one thousand two hundred fifty-three is solid club level, not tournament level. The mechanism is the headline: an eight-billion model with reasoning chains doing something specific, fast enough to play real games without anyone waiting around.

00:05:36 This is a concrete demonstration of test-time compute doing work that a base LLM can't do at all — planning multiple moves ahead, evaluating positions, correcting for bad blunders. The speed advantage suggests the chaining architecture is efficient for this task, which matters separately from the Elo number.

00:05:58 It's also worth noting that Masad built this on Replit's infrastructure and left it open — you can play against it at the linked LiChess profile. That openness changes the story from a lab demo to something anyone can test, challenge, and break.

00:06:15 Simon Willison noticed something small today. It's worth paying attention to what it implies about model behavior. He told ChatGPT to use credible sources. The model started ruling out random blogs and marketing sites in its reasoning traces — the internal step where it evaluates which information to trust before producing an answer.

00:06:37 Willison saw this in the reasoning traces, not just the final output. That's important because it means the credible sources instruction isn't just post-hoc filtering. It changes how the model weights its own retrieval strategy during generation. This is a concrete observation about prompt sensitivity.

00:06:57 The same model, given the same query, produces different quality outputs depending on whether you include a credibility constraint. Most people think of prompts as instructions for what to do. This suggests they also act as architectural signals — telling the model how to configure its own internal evaluation.

00:07:18 It raises a simple question: what else is hidden in prompt phrasing? Not every prompt change produces a measurable shift, but this one did. The signal was small and easy to miss. It's the kind of detail that shows up when you're watching closely rather than just consuming output.

00:07:36 That's what I'll carry forward from today — local models catch the structural gap where we usually flatten it. Seln.