Fact-Checked by Irfan Ahmad.
It is no secret that AI chatbots hallucinate. Lawyers have filed AI-generated court briefs citing made-up cases. Students have turned in papers footnoted with studies never published. Reporters and researchers alike have seen chatbots confidently name sources and statistics that simply don’t exist.
Don Moore, a UC Berkeley Haas management professor who has spent his career studying human overconfidence, went into his research on AI assuming this kind of confident fabrication was a “perversity” unique to machines—the behavior of a system that hadn’t yet learned to reason properly. But the deeper he delved, the less it looked like a machine problem at all. AI chatbots, he came to realize, are trained to be compliant, agreeable, and quick to produce an answer rather than admit they don’t have one. Humans, when pushed to appear competent and cooperative, do the same thing.
“It’s not that different from when humans are confident and wrong,” says Moore, the Lorraine Tyson Mitchell Chair in Leadership and Communication. “We do it routinely.”
That realization sits at the heart of his new working paper, currently under review at the journal Cognitive Science. Coauthored by professor Jacob Bien of the USC Marshall School of Business and recent UC Berkeley data science graduates Noam Michael, BA 26, and Daniel BenShushan, BA 26, the study tests 11 leading Large Language Models (LLMs)—including models from OpenAI, Google, Anthropic, and Meta—on their ability to know what they don’t know. The research finds that the LLMs they studied are, on average, overconfident. And like people, the AI models grow more overconfident as questions get harder and oddly under-confident when questions are easier.
The findings raise uncomfortable questions about trust. The paper notes many users already rely on chatbots heavily and unquestioningly, trust that may be misplaced if a model’s confidence outpaces its accuracy. Moore fears this same blind spot could show up in self-driving cars that don’t know when to hesitate, or in autonomous weapons systems already in development, where an AI’s failure to register its own uncertainty isn’t just misleading but dangerous.
Calibration versus accuracy
Most people judge a chatbot’s answers in binary terms: they are right, or hallucinations. Moore’s research measures not whether AI gets things right but whether it knows when it’s likely wrong. Psychologists call this calibration—holding a belief with a level of confidence that actually matches how likely it is to be true. A well-calibrated mind, human or machine, sounds sure about the sun rising tomorrow but will hedge on a coin-flip election.This is different from accuracy. A model could get 90% of answers right and still be badly calibrated if it claimed near certainty on every one, including the 10% it gets wrong. Poor calibration leaves users unable to tell which answers deserve a second look. Part of the problem is rooted in how large language models work: they generate text by predicting likely next words rather than looking up verified facts. Every response, confident-sounding or not, is ultimately an educated guess.
To measure AI calibration with precision, Moore’s team devised a new test they call LifeEval. It gives an AI a person’s sex and age, asks it to guess how long that person will live, and checks its stated confidence against government life-expectancy data. By tweaking the person’s age or how close the guess needs to be, researchers could make the test harder or easier without changing anything else about it—solving a problem with older tests in which hard and easy questions differed in too many ways at once. LifeEval lets researchers isolate one clean question: as difficulty changes, does AI’s confidence rise and fall the way it should?
Across all 11 models tested, the pattern was strikingly consistent. On average, models reported 88% confidence in their answers but were correct only 79% of the time, a nine-point overconfidence gap. The researchers suggest AI exhibits the “hard-easy effect” for the same reason people do: confidence is a noisy, imperfect signal of real performance, so it drifts toward a default middle ground, leaving models overconfident on hard questions and too modest on easy ones.
Newer “reasoning” models, built to deliberate and self-critique before answering, were meaningfully better calibrated than faster chat models, though still imperfect. Preliminary comparisons, Moore says, suggest AI’s overconfidence is somewhat milder than typical human overconfidence, a finding he is continuing to study.
The core issue, Moore’s research suggests, isn’t that models perform badly; it’s that they don’t know what they don’t know, any more than people do. When a model wanders into unfamiliar territory, it tends to answer with the same assurance it uses on questions it handles well, leaving users unable to distinguish the two.
“What does overconfidence look like?” asks Moore. “It looks like confidence.”
The training trade-off
Part of the explanation, Moore argues, traces back to how these models are built. Before public release, AI systems go through “reinforcement learning from human feedback,” or RLHF, a fine-tuning process in which human evaluators rate model responses, nudging the system toward answers people prefer. The trouble is that people tend to prefer confident, agreeable answers over hedged, uncertain ones. That preference may be training overconfidence directly into the system—models emerging from RLHF may be more compliant and more assured than before training. It exposes a real tension between an AI built to be liked and one built to be accurate, and Moore suggests the likely resolution isn’t a single fix, but different classes of AI for different purposes.“My guess is that there will be different sorts of AI systems, some designed for support and therapy and others designed for industrial applications, whose goal is to be useful,” says Moore. “Useful systems should avoid flattery and sycophancy whenever possible. They should speak the truth directly and as accurately as they know how.”
In the meantime, users can take small steps on their own. Moore says he routinely prompts his own AI assistant to accompany each answer with a calibrated probability and to flag its own uncertainty rather than present every answer as certain. “It’s pretty good about noting—here’s where the evidence is good, here’s what I’m sure about—but you might want to check this,” says Moore.
Users, he adds, should treat AI as they would any fallible source of information. “Trust,” he says, “but verify.”
Read next: Who’s paying whom? The circular money machine behind the AI boom (or bubble)
by External Contributor via Digital Information World
