ChatGPT, developed by OpenAI, is currently the preferred Large Language Model (LLM) that understands human intent. It generates good-quality content and is known for having human-like conversations. LLMs are trained on an enormous amount of textual data and show extraordinary capabilities in Natural Language Processing (NLP) and Natural Language Understanding (NLU). Using deep learning, LLMs process natural language and excel in language-related tasks.  

LLMs like ChatGPT and PaLM perform extremely well on unseen tasks with the assistance of proper instruction or task definition. They even use Chain-of-Thought (CoT) prompting to enhance their performance on such tasks, which is a prompting method that allows an LLM to clarify its reasoning. CoT prompting provides the model with a series of related prompts to guide its responses.

In a recently released research paper, authors have discussed ChatGPT’s performance and the option to assess its overall ability to perform fine-grained information extraction (IE) tasks. Information extraction (IE) is the strategy of mechanically extracting specific information, equivalent to structured information, from an unstructured or semi-structured data source like a body of text. It extracts heterogeneous structures, using factual knowledge, and targeting diverse information, making it a perfect scenario for evaluating ChatGPT’s capabilities.

Evaluating ChatGPT’s responses requires assessing its ability to attain high performance and measuring its answers’ reliability. To assist users higher understand the general quality of ChatGPT’s responses, the authors of the paper have designed 4 metric dimensions: Performance, Explainability, Calibration, and Faithfulness. Performance refers back to the overall performance of ChatGPT on various IE tasks from quite a few perspectives. Explainability evaluates whether ChatGPT can provide a justified reason for its prediction or not. It provides insights into its decision-making process. Calibration measures the predictive uncertainty of a model and assesses if ChatGPT is overconfident in its prediction. Lastly, Faithfulness determines whether the reasons provided by ChatGPT are truthful to the input or in the event that they are false. 

The authors have conducted their experiments and evaluation based on 14 datasets belonging to 7 fine-grained IE tasks, a few of which include named entity recognition (NER), relation extraction (RE), and event extraction (EE). The outcomes show that ChatGPT’s performance within the Standard-IE setting is poor, so it struggles with tasks requiring structured information extraction. Then again, it exhibits excellent performance within the OpenIE setting, which involves extracting information from unstructured text. These results were evidenced by human evaluation, where human evaluators rated ChatGPT’s responses as being high-quality and appropriate.

The authors have shared how ChatGPT provides high-quality and trustworthy explanations for its decisions, but its overconfident nature leads to low calibration, i.e., its predicted probabilities don’t match actual probabilities. ChatGPT portrays a high level of Faithfulness to the unique text most often and is thus faithful to the meaning and intent of the unique text. 

In conclusion, this research provides a worthwhile framework for evaluating ChatGPT and similar LLMs, enabling users to higher understand their responses’ overall quality. A Study of ChatGPT’s Information Extraction Abilities: Assessing its Performance, Explainability, Calibration, and Faithfulness

Try the Paper. Don’t forget to affix our 20k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more. If you’ve gotten any questions regarding the above article or if we missed anything, be at liberty to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club