The AI community is now significantly impacted by large language models, and the introduction of ChatGPT and GPT-4 has advanced natural language processing. Due to vast web-text data and robust architecture, LLMs can read, write, and converse like humans. Despite the successful applications in text processing and generation, the success of audio modality, music, sound, and talking head) is restricted, despite the fact that it is extremely advantageous because: 1) In real-world scenarios, humans communicate using spoken language throughout every day conversations, they usually use spoken assistant to make life more convenient; 2) Processing audio modality information is required to attain artificial generation success. 

The crucial step for LLMs towards more sophisticated AI systems is knowing and producing voice, music, sound, and talking heads. Despite some great benefits of audio modality, it remains to be difficult to coach LLMs that support audio processing due to the following problems: 1) Data: Only a few sources offer real-world spoken conversations, and obtaining human-labeled speech data is an expensive and time-consuming operation. Moreover, there’s a necessity for multilingual conversational speech data in comparison with the vast corpora of web-text data, and the quantity of knowledge is restricted. 2) Computational resources: Training multi-modal LLMs from scratch is computationally demanding and time-consuming.

Researchers from Zhejiang University, Peking University, Carnegie Mellon University, and the Remin University of China present “AudioGPT” on this work, a system made to be excellent in comprehending and producing audio modality in spoken dialogues. Particularly:

1. They use a wide range of audio foundation models to process complex audio information as an alternative of coaching multi-modal LLMs from scratch.
2. They connect LLMs with input/output interfaces for speech conversations fairly than training a spoken language model.
3. They use LLMs because the general-purpose interface that allows AudioGPT to unravel quite a few audio understanding and generation tasks.

It might be useless to start training from scratch since audio foundation models can already comprehend and produce speech, music, sound, and talking heads. 

Using input/output interfaces, ChatGPT, and spoken language, LLMs can communicate more effectively by converting speech to text. ChatGPT uses the conversation engine and prompt manager to find out a user’s intent when processing audio data. The AudioGPT process could also be separated into 4 parts, as shown in Figure 1: 

• Transformation of modality: Using input/output interfaces, ChatGPT, and spoken language LLMs can communicate more effectively by converting speech to text.

• Evaluation of tasks: ChatGPT uses the conversation engine and prompt manager to find out a user’s intent when processing audio data.

• Task of a model: ChatGPT allocates the audio foundation models for comprehension and generation after receiving the structured arguments for prosody, timbre, and language control.

• Response Design: Generating and providing consumers with a final answer following audio foundation model execution.

Evaluating the effectiveness of multi-modal LLMs in comprehending human intention and orchestrating the collaboration of varied foundation models is becoming an increasingly popular research issue. Results from experiments show that AudioGPT can process complex audio data in multi-round dialogue for various AI applications, including creating and comprehending speech, music, sound, and talking heads. They describe the design concepts and evaluation procedure for AudioGPT’s consistency, capability, and robustness on this study. 

They suggest AudioGPT, which provides ChatGPT with audio foundation models for stylish audio jobs. 

That is one in every of the paper’s major contributions. A modalities transformation interface is coupled to ChatGPT as a general-purpose interface to enable spoken communication. They describe the design concepts and evaluation procedure for multi-modal LLMs and assess the consistency, capability, and robustness of AudioGPT. AudioGPT effectively understands and produces audio with quite a few rounds of debate, enabling people to provide wealthy and varied audio material with previously unheard-of simplicity. The code has been open-sourced on GitHub.

Take a look at the Paper and Github Link. Don’t forget to hitch our 20k+ ML SubReddit, Discord Channel, and Email Newsletter, where we share the most recent AI research news, cool AI projects, and more. If you might have any questions regarding the above article or if we missed anything, be at liberty to email us at Asif@marktechpost.com

🚀 Check Out 100’s AI Tools in AI Tools Club