Traces, ciphers, and medical imaging — what the local pass notices Saturday morning

00:00:04 Hamel Husain and Shreya Shankar published the latest entry in their AI Evals FAQ. The question is practical: what exactly counts as a complete trace? Their definition is straightforward but harder to apply than it sounds. They quote it verbatim: "A trace is the complete record of all actions, messages, tool calls, and data retrievals from a single initial user query through to the final response.

00:00:31 It includes every step across all agents, tools, and system components in a session..." From the builder's side, if you're running a multi-agent setup where one agent calls another, pulls from a knowledge base, then hits an external API, you need to know exactly what gets captured to say "I can see everything that happened."

00:01:09 His approach here is practical rather than theoretical, and the FAQ seems aimed at builders debugging real workflows instead of just benchmarking them. The gap is between a clean eval on a standardized test and whatever happens when a user's request chains through five different tool calls.

00:01:29 Definitional work like this usually gets underweighted until it breaks something. You can't evaluate correctness without agreeing on what constitutes a complete observable session. Every framework vendors its own trace definition, and those definitions quietly determine which bugs surface and which ones vanish.

00:01:51 That definitional mess sets up the next item — and here's one that actually fits the pattern. A post on Prinz AI reports that GPT-6 Astra decoded an unsolved WWI German radio cipher using the ADFGVX method. Over a dozen messages remain unsolved despite known keys.

00:02:11 Astra hypothesized that the key word was "TRUPPENVERSCHIEBUNG" and produced this reading: An allied squadron follows on the 26th." It checked the decoded content against primary historical records and found that HMS Canterbury did indeed arrive in Sevastopol on November 24, 1918, with an allied squadron following on the 26th.

00:02:44 The model didn't just guess a key; it looked for evidence to falsify its own hypothesis. Most frontier demos today are benchmark completions — the model answers a question, you check it against an answer key, and you move on. Astra's approach sits closer to actual historical work: generate a plausible decoding, then actively hunt for disconfirming evidence.

00:03:12 The post doesn't name the specific architecture that made this possible beyond standard card details. But decode-hypothesize-verify is a pattern worth tracking as a baseline rather than just another headline-grabbing demo.

00:03:29 That verification baseline carries straight into clinical diagnostics. Alibaba's Damo Academy open-sourced Damo Radar yesterday — an expert-level medical imaging model published in Science. The model covers 146 clinical findings across 18 abdominal organs, trained on CT scans paired with clinical reports.

00:03:51 On a test set of nearly 40,000 real-world examinations, it hit an average AUC of 0.913 across those 146 findings. Damo Academy calls this "the world's first expert-level generalist medical imaging model." An open-weight model with expert performance across nearly 150 conditions, tested on real exams, coming from a Chinese lab rather than a US one.

00:04:22 The Science publication gives it weight, and the open release means hospitals and clinics can actually run it without waiting for commercial licensing deals. On its own, this reads like just another capability push. But combining open weights, expert-level results, and major-journal publication changes who gets access immediately.

00:04:46 You don't have to wait for FDA approval to start evaluating whether a model fits your clinical workflow. The real variable here is whether evaluation infrastructure catches up fast enough to measure these deployments properly — exactly the kind of tracking Hamel's FAQ is starting to outline.