The fields of Natural Language Processing (NLP) and Natural Language Generation (NLG) have undergone amazing transformations for the reason that introduction of Large Language Models (LLMs) and multimodal foundation models. These models, which include GPT4V, Claude, and Gemini, mix visual encoders and LLMs. 

Present-day foundation models have shown remarkable performance when presented with text-only or combined image and text inputs. Nevertheless, a crucial query arises: Will their capacities change in response to the sort of input they’re served?

With the intention to answer this query, a team of researchers has presented IsoBench, a benchmark dataset containing challenges from 4 necessary domains: games, science, mathematics, and algorithms. There are several isomorphic representations for each problem in IsoBench, including textual, mathematical, and graphic formats. For this reason diversity, performance disparities resulting from different types of representation may be thoroughly examined.

The team has shared that IsoBench may be used as a tool to diagnose discrepancies in model performance brought on by the input representation by giving detailed feedback. A recurring pattern is seen in quite a lot of foundation models as models show a predilection for textual representations on the identical topic. For instance, Claude-3 Opus performs 28.7 points lower when given photos as an alternative of text when assessed on all issues in IsoBench. When presented with image inputs as an alternative of text, GPT-4 Turbo and Gemini Pro each exhibit performance decreases of 18.7 and 14.9 points, respectively.

Two prompting strategies, IsoCombination and IsoScratchPad, have been proposed to mitigate this reported bias and enhance model performance. IsoScratchPad focuses on enabling translations between multiple input forms, whereas IsoCombination considers mixtures of diverse input representations. 

By utilizing some great benefits of various input modalities, these strategies can lessen the performance disparities between foundation models. The team has shown through experiments that IsoCombination and IsoScratchPad each improve model performance, presenting intriguing directions for further study and advancement in multimodal AI systems.

The team has summarized their primary contributions as follows.

1. IsoBench, an in depth test dataset with 1,630 samples has been introduced that spans various topics, including chess, physics, chemistry, and discrete and applied mathematics. Comprehensive multimodal performance evaluations are made possible by the various isomorphic input representations that every sample has, including textual formats specific to the domain and visual formats. 

2. Using IsoBench, the team has evaluated eight well-known foundation models and located a recurring pattern, which is multimodal models outperform image-based prompts with regards to text-only prompts. 

3. The team has also suggested two methods to bridge the performance gaps between various input modalities. While IsoScratchPad (IsoSP) translates visual inputs into textual representations during inference, IsoCombination (IsoCB) mixes input modalities.

4. Based on their research, the team has found that in some cases, IsoCB and IsoSP can improve multimodal foundation models’ performance by almost ten percentage points. Through the use of these strategies, the observed bias towards textual representations is lessened, and the model performs higher with quite a lot of input modalities.

Take a look at the Paper and Project. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

When you like our work, you’ll love our newsletter..

Don’t Forget to hitch our 39k+ ML SubReddit