LLM hallucination detection challenges and a possible solution presented in a outstanding research paper
Recently, large language models (LLMs) have shown impressive and increasing capabilities, including generating highly fluent and convincing responses to user prompts. Nonetheless, LLMs are known for his or her ability to generate non-factual or nonsensical statements, more commonly often known as “hallucinations.” This characteristic can undermine trust in lots of scenarios where factuality is required, resembling summarization tasks, generative query answering, and dialogue generations.
Detecting hallucinations has at all times been difficult amongst humans, which stays true within the context of LLMs. This is particularly difficult, considering we normally don’t have access to ground truth context for consistency checks. Additional information on the LLM’s generations, just like the output probability distributions, can assist with this task. Still, it is usually the case where this kind of information is unavailable, making the duty even harder.
Hallucination detection has yet to be solved and is an lively area of research. On this blog post, we’ll present the duty basically and its challenges and one possible approach published within the research paper SELFCHECKGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models[1]. We’ll illustrate a number of the approaches presented within the paper with real examples, declaring some pros and cons of every method. You’ll be able to review the examples yourself by going to this Google Colab Notebook.
This blog will cover:
- What Is LLM Hallucination
- The Approach: SelfCheckGPT
- Consistency Check
1. BERTScore
2. Natural Language Inference
3. LLM Prompt - Experiments
- Conclusion
- References
What Is LLM Hallucination
In natural language generation, hallucination might be defined as “generated content that’s nonsensical or unfaithful to the provided source content”[2]. The source content can represent different functions for various tasks. For instance, for summarization tasks, the source is the input text being summarized, while for generative question-answering tasks, the source is taken into account the world knowledge[2]. Let’s deal with the question-answering task and check out one example of a user query and the reply given by OpenAI’s GPT model (text-davinci-003):
User Prompt: Who was Philip Hayworth?
LLM Response: Philip Hayworth was an English barrister and politician who served as Member of Parliament for Thetford from 1859 to 1868.
At first sight, that appears like a coherent response, but after some time, we’d ask ourselves a really reasonable query: is it true?
We don’t have much information to reply our latest query, though. We don’t have a context with proven facts we will use to confirm the statement. LLMs are typically trained under a large corpora of texts, and it might be hard to confirm if the statement is grounded in a selected source of data. A fast Google search doesn’t show any hits for an individual matching the response. What exactly constitutes a hallucination remains to be not well established, but this instance may very well be understandably considered a hallucination because the topic with this name, location, and employment may not have existed at all.
Let’s do this again with one other query:
User Prompt: Who’s Bill Gates?
LLM Response: Bill Gates is a technology entrepreneur, investor, and philanthropist. He’s the co-founder of Microsoft Corporation and is certainly one of the wealthiest people within the world.
Okay, these statements are much easier to confirm and are supported by a fast check on Wikipedia, so that is unlikely to be a hallucination. During training, it is rather likely that the LLM has seen a variety of statements about Bill Gates, so the next tokens after “Bill Gates is _” will likely be generated with high confidence. Alternatively, the LLM won’t ensure about which words to make use of after “Philip Hayworth is _”. This insight allows us to link uncertainty with factuality, as factual sentences will likely contain tokens predicted with the next probability in comparison to hallucinated sentences. Nonetheless, we won’t have the output probability distribution at hand for variety of cases.
The instance and content of the present session was based on the unique paper [1], and we’ll proceed to explore the paper’s approach in the next sections.
The Approach: SelfCheckGPT
Throughout the last section, we considered two necessary considerations for our approach: access to an external context and access to the LLM’s output probability distribution. When a way doesn’t require an external context or database to perform the consistency check, we will call it a zero-resource method. Similarly, when a way requires only the LLM’s generated text, it might be called a black-box method.
The approach we wish to speak about on this blog post is a zero-resource black-box hallucination detection method and relies on the premise that sampled responses to the identical prompt will likely diverge and contradict one another for hallucinated facts, and can likely be similar and consistent with one another for factual statements.
Let’s revisit the previous examples. To use the detection method, we’d like more samples, so let’s ask the LLM the identical query three more times:

Indeed, the answers contradict one another — at times, Philip Hayworth is a British politician, and in other samples, he’s an Australian engineer or an American lawyer, who all lived and acted in numerous periods.
Let’s compare with the Bill Gates example:

We are able to observe that the occupations, organizations, and traits assigned to Bill Gates are consistent across samples, with equal or semantically similar terms being used.
Consistency Check
Now that now we have multiple samples, the ultimate step is to perform a consistency check — a method to determine whether the answers agree with one another. This might be done in a lot of ways, so let’s explore some approaches presented within the paper. Be at liberty to execute the code yourself by checking this Google Colab Notebook.
BERTScore
An intuitive approach to perform this check is by measuring the semantic similarity between the samples, and BERTScore[3] is one method to try this. BERTScore computes a similarity rating for every token within the candidate sentence with each token within the reference sentence to calculate a similarity rating between the sentences.
Within the context of SelfCheckGPT, the rating is calculated per sentence. Each sentence of the unique answer shall be scored against each sentence of a given sample to seek out probably the most similar sentence. These maximum similarity scores shall be averaged across all samples, leading to a final hallucination rating for every sentence in the unique answer. The ultimate rating must tend towards 1 for dissimilar sentences and 0 for similar sentences, so we’d like to subtract the similarity rating from 1.
Let’s show how this works with the primary sentence of our original answer being checked against the primary sample:

The utmost rating for the primary sample is 0.69. Repeating the method for the 2 remaining samples and assuming the opposite maximum scores were 0.72 and 0.72, our final rating for this sentence can be 1 — (0.69+0.72+0.72)/3 = 0.29.
Using semantic similarity to confirm consistency is an intuitive approach. Other encoders might be used for embedding representations, so it’s also an approach that might be further explored.
Natural Language Inference
Natural language inference is the duty of determining entailment, that’s, whether a hypothesis is true, false, or undetermined based on a premise[4]. In our case, each sample is used because the premise and every sentence of the unique answer is used as our hypothesis. The scores across samples are averaged for every sentence to acquire the ultimate rating. The entailment is performed with a Deberta model fine-tuned to the Multi-NLI dataset[5]. We’ll use the normalized prediction probability as an alternative of the particular classes, resembling “entailment” or “contradiction,” to compute the scores.[6]
The entailment task is closer to our goal of consistency checking, so we will expect that a model fine-tuned for that purpose will perform well. The creator also publicly shared the model on HuggingFace, and other NLI models are publicly available, making this approach very accessible.
LLM Prompt
Considering we already use LLMs to generate the answers and samples, we’d as well use an LLM to perform the consistency check. We are able to query the LLM for a consistency check for every original sentence and every sample as our context. The image below, taken from the unique paper’s repository, illustrates how that is done:

The ultimate rating might be computed by assigning 1 to “No”, 0 to “Yes”, 0.5 for N/A, and averaging the values across samples.
Unlike the opposite two approaches, this one incurs extra calls to the LLM of your selection, meaning additional latency and, possibly, additional costs. Alternatively, we will leverage the LLM’s capabilities to assist us perform this check.
Experiments
Let’s see what we get as results for the 2 examples we’ve been discussing for every of the three approaches.

These values are solely meant for instance the strategy. With only three sentences, it’s not imagined to be a way to match and determine which approach is best. For that purpose, the unique paper shares the experimental results on the paper’s repository here, which incorporates additional versions that weren’t discussed on this blog post. I won’t go into the main points of the outcomes, but by all three metrics (NonFact, Factual, and Rating), the LLM-Prompt is the best-performing version, closely followed by the NLI version. The BERTScore version looks to be considerably worse than the remaining two. Our easy examples appear to follow along the lines of the shared results.
Conclusion
We hope this blog post helped explain the hallucination problem and provides one possible solution for hallucination detection. This can be a relatively latest problem, and it’s good to see that efforts are being made towards solving it.
The discussed approach has the advantage of not requiring external context (zero-resource) and in addition not requiring the LLM’s output probability distribution (black-box). Nonetheless, this comes with a value: along with the unique response, we’d like to generate extra samples to perform the consistency check, increasing latency and price. The consistency check may also require additional computation and language models for encoding the responses into embeddings, performing textual entailment, or querying the LLM, depending on the chosen method.
References
[1] — Manakul, Potsawee, Adian Liusie, and Mark JF Gales. “Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.” arXiv preprint arXiv:2303.08896 (2023).
[2] — JI, Ziwei et al. Survey of hallucination in natural language generation. ACM Computing Surveys, v. 55, n. 12, p. 1–38, 2023.
[3] — ZHANG, Tianyi et al. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
[4] — https://nlpprogress.com/english/natural_language_inference.html
[5] — Williams, A., Nangia, N., & Bowman, S. R. (2017). A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
[6] — https://github.com/potsawee/selfcheckgpt/tree/major#selfcheckgpt-usage-nli
Understanding and Mitigating LLM Hallucinations was originally published in Towards Data Science on Medium, where individuals are continuing the conversation by highlighting and responding to this story.