AI Hallucinations: Why ChatGPT Invents Facts and Citations

AI invents facts and citations because guessing pays. Language models are trained and graded in a way that rewards a confident answer over an admission of uncertainty, so when a model has no real source to give, producing a plausible-looking one scores better than saying nothing. The invention is not a malfunction. It is the incentive working as designed.

That is a different question from whether AI output is trustworthy in general. If what you want is a routine for checking answers before you use them, our practical checklist for fact-checking ChatGPT covers exactly that. This guide goes underneath it: why the invention happens at all, what the research says about how stubborn the problem is even in tools built specifically to stop it, and one structural change that removes the most common opportunity for it.

Why do language models hallucinate?

The clearest explanation comes from a 2025 paper by researchers at OpenAI and Georgia Tech, Why Language Models Hallucinate (Kalai, Nachum, Vempala, and Zhang). Their argument is not that models are broken or that the training data was bad. It is that hallucination is the predictable output of the way models are trained and evaluated.

The analogy they reach for is one every student already understands from the inside. Picture yourself facing a hard exam question you genuinely cannot answer. Leaving it blank scores zero, guaranteed. Writing something plausible might pick up a mark. So you write something. Language models are graded on much the same logic, and they have learned the same lesson: a confident attempt sometimes wins, and an honest blank never does.

Two things follow from that. First, hallucinations arise as a natural statistical consequence of training, because incorrect statements cannot be reliably distinguished from correct ones during the training process. Second, the behaviour is then actively reinforced, because most evaluation benchmarks grade models like test-takers, rewarding confident answers and giving no credit for a hedged or uncertain one. A model that hedges honestly ranks below one that guesses boldly, so guessing boldly is what gets optimised for.

This reframes what an invented citation actually is. It is not the model lying, and it is not a glitch to be patched. It is a system that has learned that producing something always beats producing nothing, applied to a question where it has no genuine source available. The researchers’ proposed fix is telling: they argue for changing how existing benchmarks are scored rather than piling on more hallucination-specific tests, because the scoring is the root cause and the tests are only the symptom.

How often do AI tools invent or misuse sources?

Often enough that purpose-built products have not solved it. The most useful public evidence comes from Stanford’s HAI and RegLab, who benchmarked AI legal research tools in a study titled AI on Trial: Legal Models Hallucinate in 1 out of 6 (or More) Benchmarking Queries. Their earlier work found that general-purpose chatbots hallucinated between 58% and 82% of the time on legal queries. The more revealing result came next.

They tested specialised legal research tools from LexisNexis and Thomson Reuters, systems built on retrieval-augmented generation, meaning they look answers up in a real document collection rather than recalling them, and marketed explicitly as hallucination-free. Even those produced incorrect information more than 17% of the time in the cases of Lexis+ AI and Ask Practical Law AI, and more than 34% of the time for Westlaw’s AI-Assisted Research.

Read those figures carefully, because they are easy to misuse. They come from legal research queries put to legal-specific AI tools, where the questions are narrow, the sources are technical, and a wrong citation carries professional consequences. They are not a measurement of how often ChatGPT gets your history homework wrong, and nobody should quote them as if they were. What they do establish is something more useful to a student: hallucination is a structural property of this technology rather than a bug some sufficiently good product has already fixed. If teams building specialised, retrieval-grounded, professionally marketed tools still land in double digits, then assuming any AI is simply reliable about sources is not a safe default.

What is a misgrounded citation, and why is it worse?

The same Stanford work draws a distinction most people never think to check for. It separates hallucinations into two types. The first is the obvious one: a response that is simply factually incorrect. The second the researchers call misgrounded, meaning the AI cites a real source, one that genuinely exists and that you can go and find, but that source does not actually support the claim it has been attached to.

They flag the second type as potentially the more dangerous of the two, and the reason is about how people verify. A fabricated citation is fragile: you search for it, find nothing, and throw it out. A misgrounded citation survives precisely the check most students perform. You search the title, the paper is real, the authors are real, the journal is real, and you move on satisfied. The reference is genuine. The link between that reference and the sentence it is propping up is the part that was invented, and that part stays invisible unless you open the source and read it.

For anything you will be graded on, that raises the bar for what counts as checking a citation. Confirming that a source exists is not verification. Verification is finding the specific passage that says what your sentence claims it says. That is slower, and it is the only version that actually protects you. It is worth remembering that an unchecked citation in a graded essay is an integrity problem as well as an accuracy one, a separate line explored in where using AI to study becomes cheating.

Why generating from your own source is structurally safer

There is a real difference between two things that both feel like using AI, and it matters more than which model you choose. Asking an open-ended chatbot to tell you about a topic is a free-recall task: the model has to produce content from patterns in its training data, and every gap in that memory is an opening for something invented to fill the space. Handing the AI a document and asking it to build questions from that document is a different job entirely. The material is already there. There is no blank page to fill.

This narrows the risk rather than removing it, and the distinction is worth being precise about. An AI working from your uploaded chapter can still misread a sentence, drop the qualifier that changed the meaning, or attach a definition to the wrong term. What it cannot easily do is invent a reference to a paper that does not exist, because it was never asked to supply one from memory in the first place. That is why source-grounded tools sit in a different risk category: NotebookLM, for instance, answers only from the sources you give it and points back to the passage behind each claim, and our guide to studying with NotebookLM walks through how that grounding behaves in practice.

The same logic applies to how you generate practice material. If you upload a file, paste your notes, or paste a link, and the tool builds a quiz, flashcards, or recall prompts from that material, the questions are constrained by the content you supplied rather than assembled from general AI memory of the subject. That is the design GeniusPal is built around, and the free tier includes a small number of study-set generations for the life of the account, which is enough to feel the difference. It also makes the review step fast: you are checking questions against a document you already have open, not auditing claims against the whole internet.

None of this is an argument for avoiding AI. It is an argument for knowing which task you are handing it. Ask it to explain, rephrase, or quiz you on material you already hold, and you are on solid ground; the broader workflow for that lives in the guide to using ChatGPT to study. Ask it to recall facts, supply references, or produce citations from memory, and you have given it exactly the job its training incentives make it worst at. Keep the AI close to your own material, verify by opening sources rather than confirming they exist, and the failure mode this post describes largely stops being your problem.