Garbage In, Gospel Out

The Wrong Noun

An analyst at Special Operations Command Pacific asked a chatbot to read a ship's manifest. The machine mixed open-source chatter with classified signals and concluded a Chinese vessel in the Middle East was carrying components for a nuclear weapons program. The analyst asked a second AI to turn that into a formal intelligence report. Two AI passes, zero verification in between. Within hours boarding teams were staging and planes were in the air, and the U.S. was one order away from firing on a Chinese ship. Then somebody finally opened the source. The report was, in one official's word to CNN, "entirely false." It "almost started a war."

The whole week is in that sequence. Nobody was fooled by a genius. They were rushed by a confident idiot. As one source put it in the eight words you should tape to your monitor: "AI allows you to get to a bad idea faster."

We've been arguing about the wrong noun. The fashionable fear is superintelligence — the model wakes up, outthinks us, wants things. Last weekend Dario Amodei was still pricing that apocalypse. The real one showed up in the opposite costume: a machine too dumb to know it was wrong, moving faster than any human could check it. In February two Tomahawks hit an elementary school in Minab, Iran, a day-one target a Palantir tool helped wave through off years-stale data, vetting that once took hours compressed into minutes. 123 children died. The Centcom team whose job was to catch exactly that had been cut from ten people to one.

Here's the part that should keep you up. To the model, those two calls, the support ticket and the kill order, are the same problem. Same confidence, same tone, same half-second answer. The machine has no idea that one decision is a rounding error and the other ends lives. That gradient, this one's reversible, that one's a war, lives only in the human head, and we'd built it into the one thing we just optimized away: slowness. The board that meets monthly, the analyst who reads the source, the review that eats a week. That friction wasn't waste. It was how the system spent more deliberation on the decisions that could kill you.

Which is just the ruin problem wearing a uniform. A positive-expected-value round of Russian roulette is still insane; you only collect the average if you survive the one branch that ends the game. When a decision can't be undone, you stop maximizing return and start minimizing the chance of catastrophe, and you pay any premium to not die. Speed is free right up until the door only swings one way. The machine can't tell a two-way door from a one-way one. So you have to.

This is the bill we've priced all week, the eval loop, the checker, the thing you can't rent. Turns out it isn't a uniform line item. It's a gate you put where ruin lives.

At COAI today: the full Signal/Noise — the near-war, Minab, and why "confidently wrong" at machine speed is the risk the doom debate can't see — is live at getcoai.com.

123. That's how many children were killed when U.S. missiles hit a school in Minab, Iran — a target a Palantir AI tool helped wave through after vetting that once took hours got compressed into minutes, with the civilian-harm team that might have caught the stale data cut from ten people to one. Same season, a chatbot hallucinated nuclear cargo onto a Chinese ship and nearly started a war. Garbage in. Gospel out. Nobody checked.

Sort your decisions by whether they can be undone — before you let an agent near them. Bezos's test: a two-way door you can walk back, a one-way door you can't. Let the AI rip through the two-way doors — drafts, summaries, first-pass code — where speed is pure upside. Put a human hard on the one-way doors: anything that moves money, ships to a customer, or can't be reversed. Today: list your ten most AI-touched workflows and mark each door.

Write your definition of "done" as a number before you deploy, not after. Gartner says 40% of agentic projects die by 2027; McKinsey just found 30% of the companies using them got slower. The cause is rarely a weak model. It's that nobody said what "right" looked like, so the machine optimized nothing. Today: pick one live workflow and write the score it has to clear. If you can't, it isn't ready to ship.

Fund the checker like the risk desk it is. Centcom cut its harm-review team from ten people to one and called it efficiency; the savings math dropped the column where you verify the work. Glean clocked AI saving 11 hours a week and 6.5 going straight back into checking output. Today: name the person or system that owns verification for your highest-consequence workflow, and put it in the budget.

Monday, September 21

— Harry and Anthony

Sources: