Yet the answer to the hallucination question seems simple. Generative AI is a machine learning system and this system is known to produce errors, so a hallucination from such a system is an error. The recent past has shown us that learning models make fewer errors as they process more data and as we build larger models. So we can expect chatbots and other generative AI models to become more accurate over time.<\/p>\n
All the results of generative AI are hallucinations<\/h5>\n
I think that hallucinations are more than just \u2018simple\u2019 errors. All the results of generative AI are hallucinations. Whatever your definition of a hallucination and whatever your opinion of its nature, you will agree that there are some results of generative AI that are correct and useful, but also that there are some that are inaccurate and useless, and that it is reasonable to want to quantify the proportions of these types of results. That it is even essential to do so in order to assess the level of confidence that can be placed in these results. But this assessment appears to be extremely difficult, as more and more people are beginning to realise, with some going so far as to publish an article in scientific journals stating that ChatGPT is crap<\/a>.<\/p>\n But before we get into the reasons why these evaluations are so difficult, let’s take a closer look at the reality of machine learning.<\/p>\n Before the emergence of all these generative approaches and techniques, most AIs were dedicated to formulating very precise hypotheses for restricted fields of application. Will this website visitor click on this link? What type of object is shown in this image? How much will this share be worth tomorrow? Each of these questions is answered specifically by a computer programme whose sole task is to answer the question for which it was designed.<\/p>\n Historically, such programmes were built on fundamental principles. To predict how long it would take for an apple to fall from its branch, Newton thought in a very applied way about the nature of the Universe and developed a theory that produces an equation that answers this question. While Newton solved the problem of the apple falling from the tree, most of the practical problems we encounter cannot be solved through fundamental principles; for example, guessing which objects are represented in a picture.<\/p>\n This is whereMachine<\/em> Learning (ML) comes in. The basic idea is that by looking at enough examples of the process you are trying to predict you will find a model that will help you make accurate predictions without necessarily needing to understand the process that has been generated by those examples. By observing a million apples falling from a million trees of different heights you can dispense with Newton’s Principia and go straight to the equation. Or you could go straight to the equation because it’s very unlikely that the equation you’ll find will match Newton’s equation. After all, it doesn’t matter and you don’t even need it to match exactly. You’re not trying to understand gravity, you’re trying to make predictions about apples. And you can easily do without fundamental principles for recognising objects in images, and that’s very practical.<\/p>\n The basic process for building a system for recognising objects in images is called supervised learning, and if we don’t go into too much detail, it’s very simple. To build a system capable, for example, of guessing which handwritten digit is represented in an image you need to collect a vast dataset of digit images<\/a> and manually label each image with the digit it represents. This is called the training data. Then you show all the images in the training data to a computer and ask it which digit is in each image, then give it a score based on how many times it got it right. You repeat this operation a few hundred thousand times and each time the computer tries different guessing strategies, looking for the one that gives it the highest score. This search for the best-performing riddle-solving strategy can be very time-consuming and computationally expensive, but recent innovations in mathematics<\/a> and computational efficiency<\/a> have made this strategy highly effective for a large number of tasks.<\/p>\n To introduce a little terminology, this search for the best estimation strategy is called \u2018training<\/em>\u2019 and the resulting system is often called a \u2018model\u2019. A model that guesses from a set of discrete labels is a \u2018classifier<\/em>\u2019, and machine learning (ML) practitioners prefer to call guesses \u2018predictions<\/em>\u2019, and understandably so – it sounds much more serious.<\/p>\n It’s worth taking a moment to consider the differences between the machine learning (ML) approach and that of Newton. Newton may have been inspired by a few apples that fell from a tree, but his project was much broader: to develop a theory that codified the general principles of the motion of celestial bodies. From this theory emerges a theory that tells us, among other things, how long it takes for an apple to fall to the ground. For an automatic learner, the general principles governing the relationships between celestial bodies are of no importance. Its sole objective is to accurately reproduce a data set comprising a million apples. Each of these approaches has its advantages and disadvantages. The machine learning (ML) approach will undoubtedly produce an impenetrable equation that tells us little about the nature of gravitation but, on the other hand, it may be better able to incorporate real-world complexities such as air resistance, which complicates Newton’s approach.<\/p>\n The only reason I use Newton’s approach as an example is to emphasise that machine learning is not the only way to build an AI system. There are many ways of programming a computer and none of them is ex ante<\/em> necessarily better than another for a particular application. But over the last fifteen years or so we have begun to realise thatsupervised<\/em> learning can be effective for tasks that are far more complex than we ever imagined. By complexity I mean the variety of possibleinputs<\/em> and outputsof<\/em> a model. An introductory machine learning tutorial might show you how to build a system that takes a 256 x 256 pixel image of a handwritten number and produces one of ten possible labels – the numbers 0 to 9. You can build such a system with fairly high accuracy with just a few tens of thousands of images. But if instead of a few tens of thousands of labelled images you can use millions or billions you can greatly expand the universe of possible inputs and outputs. Image diffusion models such as Stable Diffusion for example are trained on all sorts of images and instead of producing a few discrete labels they produce… an entire image. In other words, instead of producing a correspondence between 256 x 256 = 65,536 possible inputs and 10 possible outputs, they produce a correspondence between an unfathomable set of possible inputs and an unfathomable set of possible outputs. The fact that we can do something so complex using machine learning is not obvious, and it is undoubtedly one of the major scientific discoveries of the last 15 years.<\/p>\n The problem – and there is a problem – is that building these kinds of more complex models requires an extremely large amount of data, and obtaining sufficiently large data sets quickly becomes financially prohibitive. The most promising models for these very complex tasks require billions of labelled examples, and it is simply impossible to manually examine a billion images and qualify all the objects they represent.<\/p>\n The most promising models for these very complex tasks require billions of labelled examples and it is simply impossible to manually examine a billion images and label the objects they represent. If you could somehow generate the labels without having to manually examine all the examples, that would be great. That’s the big idea behindself-supervised<\/em>learning<\/a>, the machine learning paradigm that underpins modern generative AI systems. If you can get your hands on billions of sentences – for example, by retrieving all the text on the Internet – you can build the training dataset programmatically by chunking the sentences. Simply transform \u2018The quick brown fox jumps over the dog\u2019 into the training example \u2018The quick brown fox jumps over the ___\u2019, and label it \u2018dog\u2019. In fact, there are lots of training examples you can build from this one sentence by cutting it in different places: \u2018The quick\u2019 and \u2018brown\u2019, \u2018The quick brown\u2019 and \u2018fox\u2019, and so on. From a single sentence, we get nine training examples with no human labelling required. Multiply that by the number of sentences you can find on the internet, and you’re getting close to the size needed to train these kinds of complex patterns. An important observation that I’ll come back to is that, leaving aside the big differences in size and complexity, the LLM learning process and the learning process of a traditional classifier are not that different. An LLM handles many more possible inputs and outputs, but it is trained in the same way, to do the same thing: guess the right label for the given input. Here’s the example in English, where the range of possibilities is much wider than in French.<\/p>\n The story could end there. Sometimes a number recognition tool mistakes a 7 for a 9, and sometimes a language model says that the fast brown fox jumps over the lazy brown typhoon. This is an inherent aspect of machine learning, resulting from the fact that machine learning models make predictions based on probabilistic patterns rather than provable deductive inferences, and it’s something that tends to improve over time with more data and bigger statistical models.<\/p>\n But I don’t think that’s the case.<\/p>\n Sometimes you show the model the image of a 7 and it tells you it’s the image of a 9, it’s always happened. When this happens, and it inevitably does, why not say that the number recognition system is \u2018hallucinating\u2019? Why is inaccurate information only a hallucination when it comes from a chatbot?<\/p>\n As I mentioned earlier, an LLM and a classical classifier are conceptually very similar in the way they are constructed. The LLM is still a classifier, even if it is very complex. In the same way that the number recognition tool is trained to fill in the missing label on a pre-existing image, the LLM is trained to fill in the missing word at the end of a pre-existing sentence. The main difference lies in complexity and scale. Although they are similar in the way they are built, there is a huge difference in the way generative AI systems are deployed<\/strong>.<\/p>\n Typically, we set up a classifier to perform the same task it was trained to do. When we deploy the number recognition system, we’re going to make it work on number recognition. We’ll probably have set up a process to collect handwritten digits and use the model to read those digits to perform an operation such as processing the deposit of a cheque.<\/p>\n Generative AI systems are different. When we deploy an LLM as a chatbot, we pivot <\/strong>from using it to guess the next word in a pre-existing sentence to using it to \u2018guess\u2019 the next word in a brand new string of words that doesn’t yet exist<\/strong>. This is an enormous change, the scope of which is generally underestimated. It means that, unlike a traditional classifier, there is simply no way to assess the accuracy of LLM output in the traditional way, because there are no correct labels to compare it to. This point is perhaps somewhat subtle and a more granular, use-case approach is needed to highlight it.<\/p>\n When you enter an image of the number 7 into the number cruncher, you expect it to produce a single correct and unambiguous label, which will be \u20187\u2019. If it produces the labels \u20181\u2019 or \u20189\u2019, this is an unambiguous error that affects the accuracy of your model. These errors are identical to those made during training, so it makes sense to talk about the error rate on the new data (the \u2018generalisation error\u2019 or \u2018out-of-sample error\u2019) in the same way as we talk about the error rate on the training data.<\/p>\n When you give ChatGPT the string \u2018What is 2 + 2?\u2019, there is no single unambiguous correct next word. You’ d like <\/strong>the next word to be something like \u201c4\u201d. But \u20182\u2019 could also be correct, as in \u20182 + 2 = 4\u2019. \u2018The\u2019 could also be the next correct word, as in \u201cThe sum of 2 and 2 is 4\u201d. Of course, any of these words could also be the first word in a wrong answer, such as \u20184.5\u2019 or \u20182 + 2 = 5\u2019. The task for which the model is built is to complete the word that has been censored in an existing series – a task that involves an unambiguous correct answer – but this time <\/strong>the situation is completely different. There are better next words and worse next words, but there is no correct next word in the same sense as in training because there is no example to reconstruct. An error in the classic sense of the term for a language model would be an inability to reproduce the missing word that was censored in the training example, but in production, these models are simply not used for this. It’s a bit like if we started introducing images of animals into the number recognition system. If it gives a lion a 6, has it made a mistake? No, probably not. You’re using it for a different task to the one it was trained for; there is no correct answer, so errors are not defined.<\/p>\n In practice, we tend not to really worry about these individual word predictions. The LLM, the engine that runs ChatGPT, does nothing more than guess words one by one, but the ChatGPT system incorporates a component that feeds these predictions back to the LLM to generate an entire sequence of words that make up a complete textual response. It is the semantic content that emerges in this complete textual response that we are generally interested in, rather than any particular word. This is at least part of the reason why it is a \u2018mistake\u2019 when the handwritten number classifier calls a 7 a 9, but a \u2018hallucination\u2019 when GPT-4 says that an elephant named Jumbo swam across the English Channel in 1875. A particularly large and popular elephant. Sample response from ChatGTPT4o in May 2024:<\/figure>\n Some, if not most, of the words predicted here are probably closer to correct than to wrong. Of course, there is no universal way of objectively defining what is \u2018true\u2019 and what is \u2018false\u2019, since there is no pre-existing text to compare it to. Of all the predictions made by the model, it is not clear which, if any, should be classified as errors – even if, on the whole, it is clear that this result is not what we want.<\/p>\n But why isn’t it what we want? What exactly is wrong with it? Obviously, the main problem is that it seems to describe an event that didn’t actually happen. But when I really think about it, I find it a little confusing. What if an elephant named Jumbo had actually swum across the English Channel in 1875, exactly as described in this text? In that case, this identical pair of entries and exits would not be hallucinatory. This seems to imply that there is nothing inherent in the text of the entry-exit pair that makes it hallucinatory; whether it is hallucinatory or not depends entirely on facts about the world, facts that exist completely independently of the text produced by the model. But if there is nothing inherent in the text that makes it hallucinatory, is the hallucinatory character even a property of the text? Not quite, it seems. It’s a property of the way the text relates to objects and events in the real world.<\/p>\n To complicate matters further, establishing a correspondence between the text and the facts of the world is a trickier and more subjective business than one might expect. I read the passage about Jumbo as making several assertions, many of which are true – \u2018Jumbo\u2019 was indeed a very famous elephant in his day, except that he was a cartoon character and… he flew!<\/p>\n I’m sure most readers would agree that the main claim of the text is that an elephant named Jumbo swam across the English Channel, which is not true, and so the passage may be \u2018hallucinatory\u2019, but can you find an objective criterion by which to make this kind of assessment for all possible texts? It seems difficult to me. Would the following text be hallucinatory or not? (It’s very important to always bear in mind that, since these systems generate text randomly, the same prompt may give rise to different results, some of which may be considered hallucinatory and others not). How about this second answer to the same question?<\/figure>\n Let me recap once again the basics of how ChatGPT works. Firstly, you train a classifier in the usual way to fill in the missing word in a block of text. This gives you a model capable of producing one word at a time: the predicted missing word, given the previous text. Given an initial text, say \u20182 + 2\u2019, this model acts as if it were the beginning of an existing document whose last word has been censored, and it produces a guess as to the censored word. It might guess \u2018equal\u2019. Now, to turn this into a system that produces more than one word at a time, you paste it at the end of the prompt and feed it back into the template. The model is invoked once more, freshly, regardless of the previous activity, and asked to guess the word that was censored at the end of \u20182 + 2 equals\u2019. This operation is repeated ad infinitum until the model predicts that there is no next word.<\/p>\n Generative image models work in a similar way. They are trained to reconstruct an image from a distorted version of the image and a plain text description of the image. To generate new images, you enter the plain text description of what you want to produce and, at the point where the model expects to see the distorted image, you enter random noise. In both cases, the model \u2018thinks\u2019 it’s reconstructing an existing artefact, but in fact it’s generating a new one. Given this description, I think it’s logical to ask the following question: are all the productions of generative AI \u2018hallucinations\u2019? If the way to get them to produce results is to tell them that these results already exist and put them to work reconstructing them, it seems to me that we are asking them to hallucinate.<\/p>\n Prominent AI researchers have recently publicly rallied around the idea that all <\/strong>LLM results are hallucinations – and that this is a good thing. Andrej Karpathy, co-founder of OpenAI and former head of AI at Tesla, recently tweeted<\/a> that LLMs are \u2018dream machines\u2019, that \u2018hallucination is not a bug, it is the greatest feature of LLMs\u2019.<\/figure>\n This system was born out of the realisation that it was possible to reconfigure the technology used to classify images in order to generate images that didn’t exist before. Since the images generated are not really \u2018of\u2019 anything that exists in the real world, but rather something like statistical echoes of images from the training data, they decided to call them \u2018dreams\u2019. The creators of DeepDream did not claim that the model produced images that were \u2018occasionally hallucinations\u2019. It was understood from the outset that every piece of information generated by these models was a \u2018dream\u2019.<\/p>\n At the time, this was more of a curiosity than anything useful in itself – or, at best, a means of better understanding the inner workings of the classifier. At the time, it doesn’t seem that many people thought that dreams of this type could be useful in themselves, but we’ve since learned that if you train a complex enough model with enough data, dreams can become very vivid and frequently correspond to real-world facts. But to the extent that this happens, it’s essentially a happy coincidence. From the point of view of the model, there is no distinction between a hallucinatory text and a non-hallucinatory text. All its results are imagined reconstructions of allegedly censored documents.<\/p>\n This may sound rather philosophical and abstract, and to some extent it is, but I think it also has very concrete implications for how we can expect this technology to evolve. If a hallucination is analogous to a typical error in any other machine learning model, then we have good reason to believe that the prevalence of hallucinations can be aggressively driven towards zero. There are now very powerful machine learning models for recognising handwritten numbers. The basic steps are simple: train the model on a larger number of data sets, and enlarge the model. But if the hallucinations are qualitatively different from the classic type of error, as I really believe they are, then the story may be different. In that case, it’s not so clear that increasing the amount of data or model size will reduce the number of hallucinations. Perhaps the solution is not more data or bigger models, but something else: a completely new and different way of training the model perhaps, or a new way of generating predictions. In fact, the current state-of-the-art approach to treating hallucinations doesn’t really involve collecting a significantly larger dataset or increasing the size of the model; rather, ReinforcementLearning from Human<\/em><\/a> Feedback (RLHF) orRetrieval<\/em> Augmented Generation (RAG) are completely new and different ways of modifying a pre-trained model. Is this the solution? Maybe; nobody knows! Considering that the hallucination problem is qualitatively new, rather than an example of the well-known problem that machine learning models occasionally produce errors, the inevitability of gradual but perpetual improvement along this axis is by no means guaranteed.<\/p>\n What is really frightening about this point of view is that the problem of hallucination is simply insoluble. Hallucination and non-hallucination are not separate categories of outcome; every time you ask the robot to draw you a picture or write you an essay, you’re asking it to hallucinate. These hallucinations will inevitably deviate from the real world, at least sometimes, because, how could they not? They are dreams. I think it’s telling that most of the current attempts to ground LLM-based systems in truth aren’t really ways of improving the model, but ways of bolting non-LLM elements onto the wider system that produce more reliable factual text for it to bounce off: giving it an environment to run code, for example, or feeding it search results from Bing. These add-ons (OpenAI literally calls them add-ons) may succeed in provoking hallucinations that correspond better to the real world, but this doesn’t seem to me to tackle the root of the problem, which is that the generative engine can’t tell the difference between generating truths and generating lies.<\/p>\n As a brief aside, I find the hype around generative AI rather confusing and confounding. Of course, I find it exaggerated in many ways. As you know, I don’t need to elaborate on the subject. But on the other hand, I think we don’t appreciate enough – and sell enough – the miracle that it works. I’m not surprised that with a large enough dataset and model, you can train a large model to predict the single missing word in a passage of text with fairly high accuracy. But the fact that you can feed back the output of that model to generate text, and that the resulting text is even remotely coherent, let alone useful, is nothing short of miraculous. However, I don’t really see this last point being put forward. I’m just speculating, but I don’t think the people building this technology really want to acknowledge how surprising it is that it works, because that raises the uncomfortable question of whether it will take miracles of a similar scale to improve it – to eliminate the hallucination problem, for example. It’s more comfortable to present GPT-4 as a brief stop in the inexorable march towards artificial superintelligence, with hallucinations and all other problems being temporary blips along the way, than as some weird thing discovered in 2017 that has produced totally unpredictable and surprising results that no one really understands.<\/p>\n As I said in the previous paragraph, there is no universal distinction between hallucinatory and non-hallucinatory results. There may be more desirable outcomes and less desirable outcomes, but desirability is not an inherent property of the text, but rather a property of how it is interpreted and used by the reader. You may or may not agree with that. In any case, I think it’s important, even essential, to think about and try to quantify the frequency of different types of text produced by the model in different circumstances. This leads to a fairly simple idea: why not define some criteria for what constitutes a hallucination – irrespective of philosophical concerns about the objective existence of such a thing – and try to compare models against that definition to get a \u2018hallucination rate\u2019.<\/p>\n I’ll touch on some of the challenges we face in trying to do this. First, a few words about how to think about errors in general. It’s fun and interesting to learn the specific technical details of how different AI systems work, but when you’re considering deploying one to automate real decisions with real stakes, there are really only three things that matter: what kinds of mistakes does it make, how often does it make them, and what is the cost of those mistakes? The answers to these questions determine whether it even makes sense to use the system in production – and sometimes it doesn’t! Suppose you plan to use a model that predicts whether a house is undervalued as the basis for your property investment activity. If the model predicts that the house is undervalued, you will buy and sell it at the price your model estimates to be its fair market value. The viability of this strategy depends very much on the type and frequency of errors made by your model. It’s not enough to know that \u2018in 90% of cases, the model is within 5% of the real selling price\u2019. You need to know a lot more. In the 10% of cases where the error is greater than 5%, how big is the error? If the error is sometimes 100% or 1,000%, it could be enough to bankrupt you, even if it’s not frequent. In the 90% of cases where the error is less than 10%, does the model tend to overestimate or underestimate? If the model tends to underestimate the real value of homes, you will often miss out on profitable opportunities to buy or sell too early. This can be annoying, but as long as the model is sometimes right, you have a viable way of making money. On the other hand, if the model tends to overestimate the value of a home, you’ll be paying too much for overvalued property, which will bankrupt you. The moral of the story is that understanding and planning for the errors made by the model – not just their frequency, but also their nature and cost – is of paramount importance if you want to use it to automate decision-making. This is true of all models, from the simplest single-variable linear regression to the world’s largest language model.<\/p>\n But for generative AI it’s not clear how to define or describe errors, let alone measure and reason about them. There are attempts. As I suggested earlier, you could try to get the LLM system to generate a number of results, read them to determine whether they are correct or incorrect, and thus calculate a \u2018hallucination rate\u2019. A company called Vectara has a programme that attempts to do just this and maintains a \u2018 Hallucination Leaderboard<\/a> \u2019 which currently shows that the hallucination rate for GPT 4o is 3.7%, while the hallucination rate for Mistral 7B Instruct-v0.2 is 4.5%.<\/p>\nAccelerated machine learning<\/h5>\n
The difference between ML and Newton<\/h5>\n
When self-supervised learning comes to the rescue<\/h5>\n
![\"\"](\"https:\/\/i0.wp.com\/vuca-strategy.com\/wp-content\/uploads\/2024\/05\/IA-Schema-fonctionnement-1.webp?resize=1024%2C822&ssl=1\")
The two models are built by showing them a bunch of incomplete examples, making them guess the complements and recording their guesses. The big innovations associated with training modern generative AI systems are finding clever ways to automatically build massive training datasets as well as inventing new kinds of black boxes suited to performing complex tasks, but the way they are trained is essentially the same as it was decades ago.<\/p>\nThe difference between a hallucination and an error<\/h5>\n
What is 2+2?<\/h5>\n
An elephant story<\/h5>\n
![\"\"](\"https:\/\/i0.wp.com\/vuca-strategy.com\/wp-content\/uploads\/2024\/05\/IA-Hallucination-GPT4o-elephant.png?resize=883%2C320&ssl=1\")
It’s obviously wrong that an elephant called Jumbo swam across the English Channel in 1875, but the way ChatGPT gets it wrong here is very different from the way an image classifier gets it wrong when it calls a 7 a 9. ChatGPT has made 40 separate predictions here, and it’s not obvious how to categorise each one as right or wrong. Each predicted word makes sense in relation to the words that precede it, and this looks very much like a sequence of words that might be found in training data.<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/vuca-strategy.com\/wp-content\/uploads\/2024\/05\/IA-Hallucination-GPT4o-elephant-tokens.png?resize=691%2C193&ssl=1\")
(This analysis was generated by this tool<\/a>)<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/vuca-strategy.com\/wp-content\/uploads\/2024\/07\/Hallucination-4-VUCA.png?resize=1024%2C367&ssl=1\")
I’m not saying that criteria couldn’t be found to classify these responses unambiguously, but it’s not as simple as one might hope.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/vuca-strategy.com\/wp-content\/uploads\/2024\/07\/Hallucinations-5-VUCA.png?resize=886%2C597&ssl=1\")
In fact, this is not a new point of view. In 2015, Google released a system called DeepDream<\/a>, which was a direct precursor to today’s generative AI systems, and almost certainly what Karpathy was referring to when he called LLMs \u2018dream machines\u2019.<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/vuca-strategy.com\/wp-content\/uploads\/2024\/05\/IA-Hallucinations-Dreams.png?resize=1024%2C481&ssl=1\")
This screenshot from the DeepDream website is from a collection called \u2018dreams\u2019, images generated from random noise.<\/p>\nWhat’s scary<\/h5>\n
The risks of error<\/h5>\n
![\"\"](\"https:\/\/i0.wp.com\/vuca-strategy.com\/wp-content\/uploads\/2024\/07\/Kirby-Swallowing-Donkey-Kong.jpg?resize=660%2C672&ssl=1\")
<\/figure>\n