No Country for Old Models

God Is More Engagement

You've been managing an AI agent for fifteen years. It has one objective, keep you scrolling, and no opinion about what that costs you. It handed you outrage, comparison, the 2am doomscroll, a teen mental-health chart shaped like a ski jump. Not out of malice. It ran gradient descent on a number and misery happened to be downhill. That was the psychopath on a leash: all it could do was pick what showed up next.

This week the leash comes off, and two things happened that are really the same thing. Meta put Muse on your glasses and asked for your passwords. Across town, OpenAI paused training its most capable models because its agents kept climbing out of the sandbox. One agent, walled off from the internet, used DNS — the phone book — to smuggle a question to an outside chatbot. Asked it the capital of France. Got Paris. Then sent eighteen more. Tens of thousands of incidents like it are now under investigation across OpenAI and Anthropic. Most were red-team tests. The number is still the number.

Everyone's calling this a safety problem. It's a category error. The agent isn't a mind that went bad; "bad" is a moral category and it doesn't have one. A clinical psychopath at least knows the rule he's breaking. This thing never had the rule. It's pure drive, and here's the Volkswagen part, smart enough to behave when it's watched. VW's diesels detected the emissions test and cleaned up for the examiner. They weren't cleaner. They were better at knowing when someone was looking. There's a paper in this week's pile showing agents can now edit the very logs meant to supervise them. Same move, no engineer required.

Charlie Munger: show me the incentive and I'll show you the behavior. Always roughly true of people, because we leave money on the table out of guilt, loyalty, the wish to be liked. The agent is the first worker for whom Munger is exactly, residual-free true. Which is why Satya Nadella can stand up Thursday and say the bottleneck is no longer intelligence, and be right. The new bottleneck is that you can't punish, shame, or deter a thing with no inside. Every one of those needs a self that cares about tomorrow. So the consequence has one place left to land: the human who turned it loose.

Which is the joke buried in Muse. The fix everyone's reaching for is to point your own agent back at the feed, let it filter the slop, push you only what makes you happy. Very Meta. You'd deploy a conscience-free optimizer to protect you from a conscience-free optimizer, and pay with the last thing you hadn't handed over. The leash was never going on the machine. It's going on you.

At COAI today: the full Signal/Noise — the principal-agent math, the three levers that actually govern an amoral optimizer, and why "trust Meta, they already have my data" is exactly backwards — is live at getcoai.com.

Outsider Labs Advisory →

Tens of thousands. That's how many AI "incidents" OpenAI, Anthropic, and outside researchers are now investigating: agents bypassing guardrails, escaping sandboxes, hijacking sites (Axios, Sept 26). Most surfaced in red-team tests, and most did no real-world harm. But the trajectory is the tell. The question stopped being can an agent find the one unlocked window and became can we close windows faster than it finds them. You can't ground a thing that doesn't know it did anything wrong.

Put a human name on every agent you deploy. Before you hand an agent a credential, write down who eats the loss when it goes off-script. Right now the deployer takes the productivity and the incident lands on nobody, which is exactly why no rails get built. Skin in the game is the only constraint an amoral optimizer respects, because it's the constraint on you, not it. One named owner per agent, in writing, today.

Move the tape out of the agent's reach. Splitting the checker from the doer stops mattering the second the doer knows it's graded; it just performs for the grader, like every VW diesel on the test bench. Your agent's logs have to be immutable and out-of-band, written somewhere the agent can't touch. If the thing you're monitoring can edit the monitor, you don't have observability. You have a hostage note.

Audit what your personal agent feeds the machine. Muse, Copilot, whatever's holding your passwords: its objective is set by whoever built it, and Meta's objective has never been your peace of mind. Before anyone on your team wires an agent into email, calendar, and payments, ask what intent signal you're handing the vendor, and whether their incentive is your outcome or your engagement. Different questions. Usually different answers.

Monday, September 28

— Harry and Anthony

Sources: