The trouble with ChatGPT is how sure it sounds
The limits that cost you aren't the things ChatGPT can't do. They're the things it gets wrong in the same confident voice it uses when it's right. Trust it with drafts and thinking, and check every name, number and date before it leaves your desk.
We’ve spent our whole lives reading confidence as a clue. A colleague who isn’t sure of something sounds unsure. They hedge, they slow down, they say “I think”. It’s one of the quiet ways we decide what to trust, and most of the time it works.
ChatGPT has switched that signal off. It sounds exactly the same when it’s right as when it’s making something up, and it does still make things up. That’s the limit that actually costs people, and the reason it happens is stranger than “it’s not finished yet”.
Why it guesses
In September 2025 OpenAI’s own researchers published a paper called Why Language Models Hallucinate. Their answer, roughly, is that we mark these models the way you’d mark a student sitting a multiple-choice exam. A blank scores zero. A guess sometimes scores. So a model trained that way learns to guess, fluently, rather than say “I don’t know”. Which is also what most of us did in exams.
It explains something that surprised a lot of people. On OpenAI’s own fact test, its 2025 reasoning model o3 got answers wrong 33% of the time, against 16% for the older o1, and the smaller o4-mini missed 48% (TechCrunch). The models built to think harder guessed more. GPT-5 then pulled it back sharply, with OpenAI reporting its thinking model makes roughly five times fewer factual errors than o3. Better, but not zero, and nobody serious expects zero, because the habit comes from how these things are built and graded. It isn’t a bug waiting for a patch.
If you want the mechanism in a few minutes, IBM’s short explainer is good on why something that predicts the next word will always be willing to invent one.
Who it catches
The list of people it has caught out is not a list of fools. In October 2025 Deloitte refunded the final instalment of a A$440,000 Australian government contract after its report turned out to contain academic references that didn’t exist and a quote from a Federal Court judge who never said it. It had been drafted with help from GPT-4o (Fortune). In 2024 a Canadian tribunal held Air Canada liable after the chatbot on its own website invented a bereavement refund policy. The airline argued the bot was “a separate legal entity responsible for its own actions”. The tribunal was not persuaded (CBC).
Then there are the lawyers, which is the bit I can’t shake, because checking citations is their entire job. In 2023 a New York lawyer was fined US$5,000 for filing six cases ChatGPT had made up. When he asked it whether the cases were real, it told him they were (Mata v. Avianca). He wasn’t the last. A researcher keeps a public database of court decisions that caught AI-invented cases, and it passed 1,500 by mid-2026, growing by around eight a day, with fines climbing towards US$15,000 a lawyer and, in some courts, suspension.
None of these people were reckless. They treated fluent writing the way you’d treat a knowledgeable colleague. It talks like one. It isn’t one.
Fluency isn’t evidence. ChatGPT sounds just as certain when it’s inventing as when it’s right, so its confidence tells you nothing about whether it’s correct. Check anything you’d act on. The tool is fine. The signal you’d normally use to judge it is gone.
The everyday ones
Beyond the big one, there are a handful of smaller limits I run into all the time, and each comes with a simple habit.
Its knowledge stops at a date. Every model is trained up to a point and then sealed, and as of 2026 the paid version’s knowledge runs to late 2025, with the free tiers older still. It can search the web, but only when it decides to. Ask it for a price, a tax threshold or who currently holds a job, and a stale answer looks exactly like a current one. So tell it to look things up and show you where it found them.
It can’t really do sums. It predicts text, and text shaped like arithmetic isn’t arithmetic. These days it often hands the sum to a calculator or writes a scrap of code, but only when it decides to. The famous failure is miscounting the letters in a word. The expensive one is a wrong total, beautifully formatted, sitting inside a quote you’re about to send. For any number that matters, tell it to use its analysis tool, or check the figure yourself.
Long conversations go off. Microsoft and Salesforce researchers ran more than 200,000 conversations through fifteen models and found performance fell by an average of 39% when a task was spread across a back-and-forth instead of asked in one go (the study). Once it takes a wrong turn early, it tends to build on it. You’ll know the feeling: a chat that was sharp for twenty minutes and now stubbornly misses the point however you phrase it. Arguing with it rarely helps. A fresh chat with one clean, complete instruction usually does.
It doesn’t know your business. It knows the internet’s average answer to a question like yours. It hasn’t met your customers, seen your margins or heard about the last time someone tried what it’s now cheerfully recommending. That makes it a great sounding board and a poor final decision-maker.
Two more, quickly. It agrees with you far too easily, which I’ve written about in how to use AI in your business. And its default writing is fluent and forgettable, samey enough to spot across a room, so anything that has to sound like you needs a heavy edit.
The slow one
The limit that worries me most is the one you don’t notice happening. In 2025 an MIT Media Lab team had 54 people write essays, some with ChatGPT, some with a search engine and some with nothing, and watched their brain activity while they did it. The ChatGPT group showed the least engagement, felt the least ownership of what they’d written, and often couldn’t quote the essay they’d finished minutes earlier (Your Brain on ChatGPT). It’s a small study, it hasn’t been peer-reviewed, and the authors themselves warned against reading too much into it. But the name they gave it has stuck with me: cognitive debt. The work feels done, but nothing got learned, so the skill isn’t there the next time you need it.
The line I try to hold is simple. Let it draft, and let it argue with me. Keep doing the thinking I’d need to defend out loud in a room.
The polished dud. There’s a team version of this. A 2025 Harvard Business Review piece by researchers at BetterUp and Stanford called it workslop: AI output that looks finished but isn’t, which hands the real work to whoever receives it. Of the 1,150 workers they surveyed, 41% had been handed some in the past month, and each piece took nearly two hours to untangle (HBR). The bigger cost was trust: 42% thought less of the person who sent it, and a third didn’t want to work with them again.
So should you stop using it?
No. It’s still the most useful general assistant most of us have ever had, and I use it every day. The limits aren’t a reason to stop. They’re a map of where to keep your hand on the wheel, which is the moment it produces a fact, a figure, a date or a decision.
So let it draft and think with you, make it search for anything that changes, send its maths to a calculator, start fresh when a chat goes sideways, and check anything with a name, a number or a consequence before you act on it. The confidence is the one thing about it you can’t take at its word.