A neural network can confidently tell you a date that didn't exist, quote a study that no one did, and refer to an article of the law that doesn't exist in the code. At the same time, the tone of the answer will be exactly the same as when it tells the truth, and this is what creates the main danger.

The phenomenon is called hallucinations, and it is not a breakdown or accidental failure. It follows from how the training of these systems is arranged, so it is worth understanding mechanics, and only then proceed to practical consequences.

Why does this happen

In September 2025, OpenAI researchers published a paper that provides the most understandable explanation I have come across. Their thesis is that models hallucinate because standard learning and evaluation procedures reward guesswork instead of recognizing uncertainty.

The simplest analogy here is as follows. Imagine a multiple-choice exam where points are not deducted for an incorrect answer, and then the rational strategy for the student is to always answer, even when he does not know, because silence is guaranteed to give zero, and guessing gives at least some chance.

Language models are in this mode all the time because they are evaluated by hundreds of accuracy tests that punish evasion and reward guessing. Because of this, a model that honestly says «don't know» loses in evaluation to one that invents a plausible answer.

The authors give a case study which illustrates well the scale of the problem. They asked in a modern model the date of birth of one of the researchers, explicitly stating in the request to answer only if the model knew it, and in three separate attempts obtained three different incorrect dates.

There is also a deeper technical reason worth mentioning. Generating the correct answer is statistically more difficult than recognizing the incorrect answer, so a model can correctly define a statement as false if you show it to it, and yet generate the same false statement yourself.

The main thing to understand

Confidence of tone is in no way related to the authenticity of the content, and this is the most important sentence in the entire material. A person usually betrays his uncertainty with intonation, pauses, or warnings, and the model does not, because he does not experience uncertainty as an inner state. It generates the most likely continuation of the text, and the most likely continuation always sounds confident, regardless of how reliable the data is at the heart of the answer. Because of this, the usual skill of evaluating the reliability of the interlocutor in a manner does not work here at all, because the invented answer looks exactly the same as the correct one.

Where hallucinations occur most often

Most often, fictions appear in specific numbers and dates, that is, in statistics, percentages, years and sums, because the model reproduces the order of magnitude well and the specific value poorly. References and quotes, which are the most dangerous category of all, deserve special attention. The model can generate the title of the article, the name of the author, the name of the journal, and the year of publication, composing it from plausible parts, and the result will look like a completely normal bibliographic reference, although the work itself never existed. The situation is similar with little-known people and organizations, because the model usually knows enough about the public figure, and about the person mentioned in the network several times, he will complete the biography from what seems logical in this context. Legal rules pose a separate risk, since article numbers, law titles and court decisions are invented just as easily, and here this has already led to real consequences when lawyers submitted documents to the court with references to non-existent precedents and received sanctions for it. Local specifics suffer the more, the narrower the context, so Ukrainian realities, local procedures and names of institutions have to be checked especially carefully. Risk categories also include technical details such as function names in libraries, command options, and menu items in specific versions of programs, as well as everything that happened after model training was completed.

Where the risk is low

Explaining established concepts works well, working with the text you yourself have provided, including retelling, structuring, editing and translation, as well as brainstorming, generating options and drafts that you will reread anyway. The general rule reads as follows: a model is strong where it requires reasoning or working with the provided material, and weak where it requires accurate reproduction of a specific fact from memory.

What to do in practice

The most important habit is to check everything you are going to quote, because if you use a number, date, quote or link, it is necessary to find them in the primary source in any case. Asking for sources is useful because it gives you a specific item to check, but the link itself can also be invented, so it is mandatory to open it.

Let the model not know, because wording like «if you're not sure, so say» really reduces the amount of fiction, although it doesn't completely eliminate it. Researchers explicitly propose making the confidence threshold explicit in the instruction itself, and this is one of the few techniques with theoretical justification. Ask one question twice in different words, and if the answers differ in details, then these details are invented. The reception looks primitive, but works much better than it seems at first glance. Do not formulate the question as if the answer definitely exists, because a query like «which three studies support this thesis» almost guarantees three names even when there are no such studies, while the question of whether there are studies on this topic leaves the model the opportunity to answer in the negative. Let's give the context ourselves, because when working with the provided text, the risk of fiction drops sharply, because the model relies on specific material, and not on its own memory. Break complex queries into parts, because a long task with many conditions gives much more room for error than a few short consecutive questions.

About searching in neural networks

Modern models know how to search on the Internet, and this significantly improves the situation with facts, although it does not completely remove the problem. A model can find a page and translate it incorrectly, can take outdated material, can refer to a source that contains an error, or can mix information from multiple pages into a statement that none of them have.

The practical conclusion is that the presence of links in the answer makes verification possible, but does not make it superfluous, so you have to open the link and check the content in any case. Separately, it is worth remembering the summaries that search engines show over the results, since they synthesize a response from several pages and present it as an established fact without a visible hierarchy of sources. I have already analyzed a real case when such mechanics helped a fraudster to confirm a completely fictional biography.

When not to use AI at all

There are categories where the price of error is too high to rely on ex post facto verification. These include medical issues relating to a specific treatment or dosage, legal documents submitted to official authorities and financial calculations on which real money depends. The most important item on this list is anything where you don't have enough competence to notice a mistake. The model is useful as an assistant for someone who understands the topic and can evaluate the answer critically, and dangerous as the only source for someone who does not have such an opportunity.

About responsibility

One thing is worth saying directly, because it applies to any use of these tools.

The person is responsible for what is published, sent or used, not the tool, so the explanation «so answered neural network» does not work either in court, or in front of the editor, or in front of the client. Because of this, the scope of the check must match the price of error, and a quick reading is enough for a draft of the letter, while for material that thousands of people will see, a check of each individual fact is required.

Hallucinations are not a temporary flaw that will be fixed in the next version, because they stem from the way these systems are trained, and as long as models are evaluated with guess-rewarding tests, the incentive to guess will not go anywhere.

This does not make the tool unusable, but only makes it a tool that requires verification, roughly like a regular search output or an article on the network, only without visual clues about the reliability of the source.

The most useful habit is to treat the neural network response as a draft from a understanding but inattentive assistant who should take ideas and wording from, while checking each specific fact independently.


Sources: Kalai A, Nachum O, Vempala S, Zhang E. Why Language Models Hallucinate, OpenAI Blog