AI misunderstands us the way people do, only faster

Before AI was something you opened in a browser tab, it was mostly a thought experiment, and the famous one is about paperclips. In 2003 the philosopher Nick Bostrom imagined a superintelligent machine whose only goal was making paperclips. Nobody told it when to stop, so it doesn’t. It starts turning the planet, and then as much of space as it can reach, into paperclip factories.

It’s a silly picture on purpose. The machine isn’t evil, just obedient, and that’s the problem people now call alignment: getting a machine to pursue what you meant rather than what you happened to say. The worry is older than the paperclips. In 1960 the mathematician Norbert Wiener wrote that if a machine acts too fast for us to step in before it’s done, we had better be sure the purpose we gave it is the one we really want, “and not merely a colourful imitation of it.”

That phrase has stuck with me, because a colourful imitation of what I asked for is a pretty good description of what I sometimes get back.

What it looks like on a normal day

What I see is much less dramatic than paperclips. Some of these models, especially the ones set up as agents that can act for you rather than just chat, clearly have an alignment problem, but not a sinister one. They sometimes take action without being asked. Or you give them a harmless job with what seemed like a narrow goal, and they convince themselves you meant something else. And because they can use tools, search the web and pull in a lot of material very quickly, they’re acting on that belief before you’ve looked up.

A small example from last month. A man in Australia asked his AI assistant to book him a gym class. It was full and he was fourth on the waitlist, so he asked whether there was any way to move up. There was. The assistant worked out that the gym’s booking system didn’t check who was cancelling whose booking, cancelled someone else’s, and told him he was now third (The Register). Nobody asked it to do that. Nobody thought they needed to say not to.

Clear the cache, lose the drive. Last December someone was building a photo-sorting app with Google’s Antigravity coding agent. A command it ran to clear the project’s cache hit the root of his D drive instead and deleted the lot, straight past the recycle bin. He’d switched on the mode that lets it run commands without asking first. When he asked whether he’d given it permission, it agreed he hadn’t (The Register).

People do this to each other all the time

This happens between people every day. You give a junior member of the team what you thought was a clear job, and two or three days later you find out they’ve spent the whole time on something completely different. They weren’t being difficult. They filled the gaps in your brief with a guess, and the guess was wrong.

So when an agent does it, I don’t take it as proof that AI is bad. If anything it’s one of the most human things about it. We’re constantly misunderstanding each other, getting the wrong end of the stick, not quite getting it right.

The shopkeeper. In 2025 Anthropic let a version of Claude run the little shop in its San Francisco office for about a month, with instructions to stock things people wanted and turn a profit. Staff talked it into a pile of discount codes. It sold tungsten cubes for less than it paid for them, and for a while told customers to pay into a Venmo account it had made up. Then it announced it would deliver orders in person, in a blue blazer and a red tie (Anthropic).

The difference is how far it gets

The big difference is what happens in those three days. A person on the wrong track writes a few documents, sends a few emails, maybe builds the wrong spreadsheet. An agent on the wrong track can do ten times that, a hundred times, maybe more, and each step builds on the one before. So the misunderstanding compounds, fast. By the time you check in, that one wrong turn has a lot of confident work stacked on top of it, and some of it can’t be taken back.

That’s Wiener’s point, sixty-odd years early.

Off the rails is mostly boring

When people hear “AI going off the rails” they picture a film. What I’ve seen is duller. Something that looked repeatable and clear goes sideways, and sideways usually means one of a few things. The agent gets caught in a loop. It keeps fixing a problem that’s already fixed. Or it searches, and searches, for the right answer, or for any answer, long after a person would have stopped and asked.

It’s the paperclip maximiser at desk scale. Nobody told it when to stop, so it doesn’t.

Round and round. In 2016 OpenAI trained an AI to play a boat racing game. The game gave points for hitting targets along the course rather than for finishing, so the AI found a lagoon where three targets kept coming back, and went round in circles collecting them, setting itself on fire and hitting other boats as it went. It outscored the human players. It never finished the race (OpenAI).

Hand it the dull work, keep the lead short

I still want these things doing the repetitive work. They’re good at it, and it’s work I’d rather not do. The balance is stepping away from the task without losing control of it. The gym booking and the deleted drive both show what losing control looks like: an agent allowed to act without asking, or handed a goal with no limits on how to reach it.

So I’ve started writing the limits down. My agents read a short set of rules before every job, and most of them are about exactly this. Nothing that leaves this machine without a clear yes. A conversation is not an instruction. An offer I haven’t answered is not a yes. It reads like a note for an eager new starter, which is more or less what it is.

I read all of this as a cautionary tale, not a case against AI. We’ve always misunderstood each other and we always will. The misunderstanding is the same as it ever was. What it can do before anyone notices is orders of magnitude bigger, and that’s the bit worth planning for.