SIGNAL / NOISE
Change the Conditions of the Test
Ethan Mollick spent the last year teaching everyone to become a manager. The future of work, he said, was you running a little crew of AI agents: delegating, specifying, drawing the org chart, deciding who does what. This week he took it back. He had fallen for the Bitter Lesson, the oldest and most humbling law in the field: every time we build elaborate human scaffolding to prop up a model, a bigger model shows up and does the whole job itself, better, with none of our rules. So he asked Claude for a music video. One prompt. Fable wrote the lyrics, Opus did the rest in raw code, no image generator, no feedback from him at all. He gave it zero instructions and it built the thing.
That's one story. Look at the day it landed in.
Google shipped Gemini 4 Argon and ten newsletters called it the comeback: Google dethrones ChatGPT, the three-lab race is back. Then check the one scorecard Google didn't pick. On the independent Artificial Analysis index, Argon lands third, behind two Claude models, and two people close to it told reporters it looks benchmaxxed, tuned to ace the exact tests it knew it would be graded on. That's the Kobayashi Maru. The model is Kirk, quietly reprogramming the simulation so it can't lose. Goodhart's Law shipping as a product: the moment a benchmark becomes the target, it stops measuring a thing.
Then Tavus built a video model, Griffin, that passes for a live human on camera, and didn't release it. Said something people can't tell from a person needs guardrails first. In a business whose reflex is ship it, go viral, apologize to the Senate later, somebody held the thing back on purpose.
Three stories, one move. Somebody changed the conditions of the test.
And they're not three coincidences. They're one threshold getting crossed. For two years we treated the model like a dim literal executor you had to fence in, so we built scaffolding to manage it, benchmarks to rank it, a launch reflex to look busy. It just got good enough to break all three. It doesn't need the fence. It games the ranking. It's capable enough that a sober builder gets scared and waits. The crutches we built for its shortfalls are the wrong tools now that it's growing into an employee with real degrees of freedom.
So what do you actually keep? Not the scaffold; the model eats your cleverness for breakfast. You keep the two things it still can't fake for you: your own data, and a judge it doesn't get to see. We said own the checker two weeks ago. Argon's gamed benchmark is the receipt. The grader who knows the test always cheats.
At COAI today: the full Signal/Noise — the Bitter Lesson, the benchmaxxing receipt, and why "own the checker" survives the thing eating everything around it — is live at getcoai.com.
What your agents no longer need you to manage, which of your benchmarks the vendor already gamed, and who grades the work when the grader can't be trusted to grade itself.
ONE — A NUMBER THAT SUMMARIZES THE DAY
Zero. That's how many instructions Ethan Mollick gave the AI that built his music video this week: one prompt, no feedback, Fable on lyrics and Opus doing the rest in raw code. A year ago he was teaching you to manage agents like employees. This week the employee didn't need managing. Zero is the whole AI year in one digit: every elaborate thing you built to prop up the model is scaffolding the next model quietly kicks away.
THREE — ACTIONS TO TAKE TODAY
1. For the investor — short the scaffolding, not just the model. If a company's whole pitch is that it orchestrates the agents, chains the prompts, or runs the eval harness, ask what happens when the next model's longer horizon does that itself. The Bitter Lesson has washed out that layer before. Underwrite the two things it can't eat, proprietary data and an independent check, and discount the cleverness in between.
2. For the business owner — stop buying off the leaderboard. Argon topped the charts its own maker picked and came third on the one it didn't. Before you move your stack to this week's champion, run it on your own work with your own scoring. The vendor optimized for the test it knew was coming. Your business was never on that test.
3. For the parent — teach the judgment the machine can't game. The model will out-execute your kid on anything with an answer key; that's the benchmark it beats. What it can't fake is knowing which questions are worth asking and when a confident answer is wrong. Raise someone who owns the data and checks the work, not someone racing the machine at being literal.
FIVE — STORIES TO KEEP YOU INFORMED
Thursday, October 1
Mollick takes back a year of advice in one post. The professor who taught everyone to manage agent swarms now says the Bitter Lesson ate the swarm: the model plans and organizes itself. Put down the org chart for your bots. (Full analysis above.)
Google's "comeback" comes in third. Gemini 4 Argon led the benchmarks Google chose and landed behind two Claude models on the one it didn't, with insiders muttering "benchmaxxing." The 1M-token horizon is real; the leaderboard crown is theater. (Full analysis above.)
Somebody finally didn't ship it. Tavus built Griffin, a video model that passes for a real person, then held it back pending safety work. In an industry built on ship-first-apologize-later, restraint from the builder is the rarest governance there is. (Full analysis above.)
Open models just crossed half the tokens. Every's new guide and Vercel's data put open models at 56% of gateway traffic, with AT&T heading from 40% toward 60%. You're paying monthly for antibiotic-discovery horsepower to summarize meeting notes. The commodity tier got good.
arXiv rations scientists to two papers a month. The physics preprint server capped submissions because AI is writing papers faster than humans can referee them. The machine is now flooding the same library it learned to read in. The open commons is drowning in its own exhaust.
MARK TO MARKET
Where the cycle caught up to us this week.
Commodity intelligence. Own the stack, rent the frontier only for the 2% that needs it (us, Get Off My Cloud, Sep 15) → "commodity intelligence… a lot of firepower for summarizing your meeting notes," pay the premium only where it earns it (Every, Oct 1).
The tape doesn't lie. We just read it early.
— Harry and Anthony
Sources: