The fields of Artificial Intelligence and Machine Learning are solely dependent upon data. Everyone seems to be deluged with data from different sources like social media, healthcare, finance, etc., and this data is of great use to applications involving Natural Language Processing. But even with a lot data, readily usable data is scarce for training an NLP model for a specific task. Finding high-quality data with usefulness and good-quality filters is a difficult task. Specifically talking about developing NLP models for various languages, the dearth of knowledge for many languages comes as a limitation that hinders progress in NLP for under-represented languages (ULs). 

The emerging tasks like news summarization, sentiment evaluation, query answering, or the event of a virtual assistant all heavily depend on data availability in high-resource languages. These tasks are dependent upon technologies like language identification, automatic speech recognition (ASR), or optical character recognition (OCR), that are mostly unavailable for under-represented languages, to beat which it will be significant to construct datasets and evaluate models on tasks that might be useful for UL speakers. 

Recently, a team of researchers from GoogleAI has proposed a benchmark called XTREME-UP (Under-Represented and User-Centric with Paucal Data) that evaluates multilingual models on user-centric tasks in a few-shot learning setting. It primarily focuses on activities that technology users often perform of their day-to-day lives, corresponding to information access and input/output activities that enable other technologies. The three predominant features that distinguish XTREME-UP are – its use of scarce data, its user-centric design, and its concentrate on under-represented languages.

With XTREME-UP, the researchers have introduced a standardized multilingual in-language fine-tuning setting rather than the traditional cross-lingual zero-shot option. This method considers the quantity of knowledge that may be generated or annotated in an 8-hour period for a specific language, thus aiming to present the ULs a more useful evaluation setup. 

XTREME-UP assesses the performance of language models across 88 under-represented languages in 9 significant user-centric technologies, a few of which include Automatic Speech Recognition (ASR), Optical Character Recognition (OCR), Machine Translation (MT), and knowledge access tasks which have general utility. The researchers have developed latest datasets specifically for operations like OCR, autocomplete, semantic parsing, and transliteration with a view to evaluate the capabilities of the language models. They’ve also improved and polished the currently existing datasets for other tasks in the identical benchmark.

XTREME-UP has one in all its key abilities to evaluate various modeling situations, including each text-only and multi-modal scenarios with visual, audio, and text inputs. It also offers methods for supervised parameter adjustment and in-context learning, allowing for a radical assessment of varied modeling approaches. The tasks in XTREME-UP involve enabling access to language technology, enabling information access as part of a bigger system corresponding to query answering, information extraction, and virtual assistants, followed by making information accessible within the speaker’s language.

Consequently, XTREME-UP is a terrific benchmark that addresses the information scarcity challenge in highly multilingual NLP systems. It’s a standardized evaluation framework for under-represented language and seems really useful for future NLP research and developments.

Try the Paper and Github. Don’t forget to affix our 21k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more. If you have got any questions regarding the above article or if we missed anything, be at liberty to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club