Within the digital era of emerging technologies, LLMs have emerged as a robust tool revolutionizing many elements of human society and culture, reshaping how we interact with computers. Yet, there’s a pivotal challenge that should be solved. The constraints of  LLMs are evident, revealing a niche in the lack to know the contexts and nuances of a conversation and rely upon the standard and specificity of the prompt. One major limitation is that they lack the depth of real communication, missing all of the paralinguistic information.

Project Rumi from Microsoft goals to boost the capabilities of LLMs by addressing limitations in understanding nonverbal cues and contextual nuances. It incorporates paralinguistic input into prompt-based interactions with LLMs to enhance the standard of communication. The researchers have used audio and video models to detect real-time non-verbal cues from data streams. Two separate models are used for paralinguistic information from the user’s audio, the primary prosody tone and inflection of audio and the opposite from the semantics of the speech. They’ve used vision transformers for encoding the frames and identifying facial expressions from video. A downstream service incorporates the paralinguistic information into the text-based prompt. This multimodal approach goals to boost user sentiment and intent understanding, thus elevating human-AI interaction to a brand new level.

On this research, researchers have only briefly explored the role that paralinguistic provides in communicating critical details about user’s intentions. In the longer term, they plan to model to make the model higher and more efficient. Additionally they wish to add more details like  HRV (heart rate variability) derived from standard video and cognitive and ambient sensing. That is all a part of an even bigger effort so as to add unspoken meaning and intention in the subsequent wave of interactions with AI.

Take a look at the Project Page. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 27k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more.