ChatGPT generates text answers by finding the most probable next word in the given context. For example, if you ask ChatGPT ‘When was the Great Pyramid built?’, the system calculates what would be the most likely first word in the answer. Here the most likely first word is ‘The’. So it generates ‘The’. Now, what will be the most likely next word for the given question and the already generated words of the answer? It’s ‘Great’. The 3rd word will be ‘Pyramid’. Similarly, it keeps generating words and adding them to the answer until it reaches an end.
The Learning
How does it know what is the most probable next word? Because of the training that ChatGPT has undergone. ChatGPT belongs to a type of machine learning model called the Large Language Model (LLM). These models are trained on billions or even trillions of lines of text data collected from all over the internet and other sources. The probability calculation will be based on text patterns learned from its training data.
The Temperature
But why doesn’t it give the exact same answer for the same question every time? Because it doesn’t always add the next word with the highest probability. It may sometimes add the next word with 2nd or 3rd highest probability. Whether it always chooses the highest probable word or other words with lower probability is controllable through a variable called temperature. If the temperature value is 0, it will always take the highest probable word. It can be any decimal value between 0 and 1. As the value increases, the randomness of the answers will also increase.
A computer program is expected to produce the same output for a given input. Getting slightly different answers every time may give you the impression of thinking like a human being. But it’s just the temperature value. The creators of ChatGPT must have set this value to something higher than 0, which is why we don’t get the exact same answer. The web-based version doesn’t allow us to customize this value. But we can customize it while accessing ChatGPT through its API.
Sounds Good! Let’s Hallucinate!
So answers from ChatGPT are not based on any real understanding of concepts. It is just a probability calculation. This is kind of similar to answering questions based on what sounds good. If it sounds good, then it must be right. If you keep answering based on sounding good, without a real understanding of the concepts, then you may say something that doesn’t make any real sense. This is called Hallucination in artificial intelligence.
Hallucination is the main problem in LLMs. Out of many LLMs created by different companies, ChatGPT is the one that has successfully reduced the hallucination problem to a great extent, which is why it gained popularity.
Will it take over the world?
LLMs will keep getting better and better at generating answers, but that’s all it can do. It may take away your job, but it can’t take over the world. So you need not worry about LLMs. But, there is a different kind of AI that is still in the very early stages of development. It’s called Artificial General Intelligence(AGI). That’s something you may need to worry about.
Reference Links:
https://www.semrush.com/blog/how-does-chatgpt-work
https://ai.meta.com/tools/system-cards/ai-systems-that-generate-text
https://en.wikipedia.org/wiki/Artificial_general_intelligence