SIGNAL / NOISE

Say What You Mean

Trail of Bits ran an experiment this week that should stop every founder mid-sentence. They pointed their best coding agents at 130 real bugs in open-source software, let the agents write the fixes, then went back and counted what else moved. Twenty-one regressions. The agent patched the hole in front of it and quietly broke twenty-one things nobody asked it to touch. Not malice, not a hallucination. It did exactly what it was told, and filled in the parts it wasn't told with judgment of its own.

That's the whole story of the week, and it's older than any of us.

Think about how we've talked to machines for seventy years. Punchcards, then assembly, then C, then Java, then HTML, and now plain English: "build me a dashboard that flags the invoices that are late." Every one of those steps was a move in the same direction, one layer further from the metal, each letting us say less and get more. We told ourselves that meant we needed to know less. It's the reverse.

Picture an onion. The machine doing the actual work sits dead center. Every layer we've added is a skin around it, and your instruction now lands on the outermost skin, further from the core than any generation before us has stood. The further out you are, the more skins your intent has to pass through to reach the center, and at every one, anything you left vague gets decided for you by something that is not you. Punchcards were brutal and literal; they did precisely what the holes said. English is generous and loose, so the machine covers the slack with a guess. Most guesses are fine. The twenty-first one wipes your production server.

Which is why the labs just shipped the thing that makes it worse. Claude Duet and Meta's Muse both launched this week as agents that run while you're away: more layers, more autonomy, more slack to fill. And the same week, AI found a flaw sitting in 70% of the internet, work OpenAI says took one researcher a week instead of a team months. The power of the new layer is real. So is the blast radius.

Here's the part nobody's saying out loud: the work didn't disappear. It moved. It used to live in getting the syntax exactly right. Now it lives in getting the intention exactly right, and loading enough context that the machine isn't guessing in the dark. That's a harder job than the one it replaced, not an easier one. The people who win the agent era are the ones who figure out that precision didn't die with the semicolon. It moved to the sentence.

The machine does exactly what you tell it. The entire job now is telling it precisely enough.

At COAI today: the full Signal/Noise — the whole climb from punchcards to Duet, and why English is the first interface you can't read back — is live at getcoai.com.

What your agents are allowed to touch, how much context you actually loaded before you hit go, and who reads the twenty-one things it changed that you never asked for.

ONE — A NUMBER THAT SUMMARIZES THE DAY

21. That's how many separate things Trail of Bits' AI agent broke while fixing 130 bugs it was actually asked to fix — regressions it introduced on its own, in code nobody flagged. One in three of the problems it later flagged weren't even real. The machine does what you say. The space between what you say and what you mean is where it improvises, and this week that space had a number: twenty-one surprises, hiding in the patches that were supposed to be the safe part.

THREE — ACTIONS TO TAKE TODAY

For the investor — price the verification layer, not the autonomy. The money is chasing agents that act on their own. The bottleneck, and the next fortune, is the checking layer: whoever catches the twenty-one things the agent changed. Underwrite the companies selling proof, tests, and audit trails, not just the ones selling speed. Autonomy is table stakes now. Verification is the moat.

For the business owner — your new job is the spec, not the task. Before you turn an agent loose on your invoices or your codebase, write down what you mean precisely enough that a literal-minded genius can't misread it, then load the context it needs. The companies that get burned this year won't have bad agents. They'll have vague instructions and nobody reading the diff.

For the parent — teach the kid to say exactly what they mean. The skill that mattered in 1995 was learning the machine's language. The skill now is making your own thinking clear enough that a machine can't warp it. Precise intention, said out loud, is the literacy of this era. Have them ask for exactly what they want, then notice when they got something else instead.

FIVE — STORIES TO KEEP YOU INFORMED

Thursday, October 22

  • The agent fixed your bug and changed 21 other things. (Full analysis above.) Trail of Bits shipped two agent plugins and the receipts with them: 21 regressions across 130 merged patches, and only 36% of flagged issues were real. The tell of the whole era — the machine is literal, and your spec is the safety system.

  • AI found the hole in 70% of the internet before the bad guys did. OpenAI, Wiz, and Cloudflare used frontier models to surface PHP2Shell, a Heartbleed-scale flaw; governments across Europe took sites offline to patch. OpenAI calls it a week's work that used to take a team months. The new layer cuts both ways. (Feeds the analysis above.)

  • Claude Duet and Meta's Muse both launched the always-on agent this week. Anthropic's runs on Opus 5.5 and pauses for your sign-off; Meta's browses, fills forms, and negotiates on its own. Two bets on how much slack you'll hand a machine that fills the slack with judgment. (Full analysis above.)

  • 90% of companies already had an AI-agent security incident. ServiceNow surveyed 3,700 security leaders: nine in ten took a hit in the past year, and fewer than half have rules written for agents. Manufacturing, the one sector with real safeguards, got hit the least. Governance isn't the tax. It's the permission slip.

  • Anthropic's S-1 leaks: $16.3B revenue, $80B loss, $600B in compute bills. Staggering in both directions. Most of the loss is non-cash accounting charges; the compute commitments are real, and larger than all of Intel is worth. The whole thing is a bet that revenue holds its 15x pace.

— Harry and Anthony

Sources:

Reply

Avatar

or to participate