Mathematical reasoning is significant for problem-solving and decision-making, particularly in large language models (LLMs). Evaluating LLMs’ mathematical reasoning normally focuses on the outcome somewhat than the reasoning process intricacies. Current methodologies, just like the OpenLLM leaderboard, primarily use overall accuracy, potentially overlooking logical errors or inefficient steps. Enhanced evaluation approaches are needed to uncover underlying issues and improve LLMs’ reasoning.
Existing approaches typically evaluate mathematical reasoning in LLMs by comparing final answers with ground truth and computing overall accuracy. Nonetheless, some methods assess reasoning quality by comparing generated solution steps with reference ones. Despite datasets providing ground truth, diverse reasoning paths to the identical answer challenge reliance on any single reference. Prompting-based methods directly ask LLMs, often GPT-4, to guage generated solutions, but their high computational cost and transparency issues hinder the practicality of iterative model development.
Researchers from Shanghai Jiao Tong University, Shanghai Artificial Intelligence Laboratory, Yale University, Carnegie Mellon University, and Generative AI Research Lab (GAIR) introduced REASONEVAL, a brand new approach to evaluating reasoning quality beyond final-answer accuracy. It utilizes validity and redundancy metrics to characterize reasoning steps’ quality, which is routinely assessed by accompanying LLMs. REASONEVAL relies on base models with robust mathematical knowledge, trained on high-quality labeled data, to instantiate its evaluation framework.
REASONEVAL focuses on multi-step reasoning tasks, assessing the standard of reasoning beyond final-answer accuracy. It evaluates each reasoning step for validity and redundancy, categorizing them into positive, neutral, or negative labels. Step-level scores are computed based on validity and redundancy after which aggregated to generate solution-level scores. The strategy utilizes various LLMs with different base models, sizes, and training strategies. Training data is sourced from PRM800K, a dataset of labeled step-by-step solutions collected by human annotators.
REASONEVAL achieves state-of-the-art performance on human-labeled datasets and might accurately detect different errors generated by perturbation. It reveals that enhanced final-answer accuracy doesn’t consistently improve the standard of reasoning steps for complex mathematical problems. The strategy’s assessment also aids in data selection. Observations highlight significant decreases in validity scores for logical and calculation errors, while redundancy scores remain stable. REASONEVAL distinguishes between errors affecting validity and people introducing redundancy.
In conclusion, the research introduces REASONEVAL, an efficient metric for assessing reasoning step quality based on correctness and efficiency. Experimentation confirms its ability to discover diverse errors and competitive performance in comparison with existing methods. REASONEVAL exposes inconsistencies between final-answer accuracy and reasoning step quality while also proving effective in data selection for training.
Take a look at the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.
In case you like our work, you’ll love our newsletter..
Don’t Forget to affix our 40k+ ML SubReddit
Asjad is an intern consultant at Marktechpost. He’s persuing B.Tech in mechanical engineering on the Indian Institute of Technology, Kharagpur. Asjad is a Machine learning and deep learning enthusiast who’s at all times researching the applications of machine learning in healthcare.