TL;DR ChatGPT is bad for accuracy, good for inspiration
This is counterintuitive of our science fiction expectations of AI being an all knowing inhumanly accurate deity. This misunderstanding results in comments criticizing ChatGPT. For example:
I find this paragraph dull, generic, and uninspired
ChatGPT can't even answer <basic fact>
The code doesn't run
Well just like if you tried using your calculator to inspire your writing, ChatGPT won't work well if it's not the right tool for the job.
So what are the reliable use cases and when should you use caution? To answer that let's take a high level overview of the mechanism behind ChatGPT.
What's the GPT in ChatGPT?
GPT = Generative Pretrained Transformer
Generative is referring to how given a text prompt the machine learning model will predict the next words in the sequence based on the surrounding words.
'PT' stands for Pre-trained Transformer
Transformer is referring to a type of machine learning model. It's a mechanism for using the sequence of previous tokens to predict the next token. Tokens in this context are the words in the text. The model was trained by having it predict missing words in text from across the internet using the surrounding words.
When writing new text the AI is not forming thoughts, it's using statistics to predict the next word based on the previous words. However since the model was pre-trained on a large part of the internet the patterns it picks up yield very impressive results. The the latest language models use a large number of the surrounding words to predict the next word resulting in more realistic generated text.
With that in mind, let's go over what to be cautious about when using ChatGPT.
When to be Cautious Using ChatGPT
There has been a lot of hype surrounding using ChatGPT as a search engine. It's possible that ChatGPT will generate a response that answers a question correctly. However you have to keep in mind words appearing in the response are select based on probability. If a fact is repeated many times across the internet most likely the question will be answered correctly. ChatGPT has no rules or logic to reason about things. The only rules are those derived from studying the statistical relationships between words in the English language.
Since words are selected based on probability it's possible that a more commonly appearing word will be substituted for the correct word. The probabilities of the words appearing in the answer are similar, however the meaning to us humans is vastly different.
Let's take a look at an example.
Donald Trump's second wife was not Melania Trump, his third wife, rather Marla Maples. The child from his second wife was named Tiffany Trump. Melania Trump and her son occurred far more frequently in the media. It would make sense that the probability of Melania and Barron occuring in the context of the words Donald, Trump, wife, and child is greater than that of his actual second wife and her child.
It's also important to keep in mind just because the factual question was answered correctly once doesn't mean that related question, or ever that same question paragraphs will be answered correctly.
These are examples of how the response is terribly wrong factually because of the statistical nature of words being chosen. There are currently no mechanisms to prevent this, perhaps this will change in the future.
ChatGPT and related technologies such as Copilot are great for providing inspiration. The same way words may be substituted incorrectly causing factually incorrect responses, tokens in the code may also be substituted resulting in vastly different results. For example undefined variables and more nuanced issues related to security and performance which would not throw an error immediately.
Good Use Cases
Avoiding a blank screen is probably the best use case for ChatGPT. It's a great tool if you're planning to spend time thoroughly reading and editing the generated text. It's an amazing tool for writing boring things such as routine emails, applications, mundane school work, and about a million other things.
However if you're trying to draft something highly creative, such as copy for a product you care about, you might want to think of creative ideas yourself. Because with ChatGPT you'll be getting content similar to what has literally been written millions of times before.
Answering Evergreen Factual Questions
Evergreen refers to facts that are consistent over time and have been subject to lot's of discussion on the internet. As we discussed, what ChatGPT writes is based on statistics, if there's lots of accurate text relating to the topic, the generated text is more likely to be factual. For example historical facts or well known scientific knowledge.
Hopefully this helps you form a mental model for when to use ChatGPT.
If you know someone who has an incorrect idea of how ChatGPT works please consider sharing this article with them.