The need for large-scale music datasets with natural language captions is an issue for text-to-music production, which this research addresses. Although closed-source captioned datasets can be found, their scarcity prevents text-to-music creation research from progressing. To tackle this, the researchers suggest the Music Understanding LLaMA (MU-LLaMA) model, intended for captioning and music query answering. It does this by utilizing an approach to create many music question-answer pairings from audio captioning datasets which can be already available.

Text-to-music creation techniques now in use have limits, and datasets are ceaselessly closed-source due to license constraints. Constructing on Meta’s LLaMA model and utilizing the Music Understanding Encoder-Decoder architecture, a research team from ARC Lab, Tencent PCG and National University of Singapore present MU-LLaMA. Specifically, the study describes how the MERT model is used because the music encoder, enabling the model to grasp music and reply to queries. By mechanically creating subtitles for numerous music files from public resources, this novel method seeks to shut the gap.

The methodology of MU-LLaMA is predicated on a well-designed architecture, which begins with a frozen MERT encoder that produces embeddings of musical features. After that, these embeddings are processed by a thick neural network with three sub-blocks and a 1D convolutional layer. The linear layer, SiLU activation function, and normalization components are all included in each sub-block and are connected via skip connections. The last (L-1) layers of the LLaMA model use the resulting embedding, which supplies crucial music context information for the question-answering procedure. The music understanding adapter is tweaked during training, however the MERT encoder and LLaMA’s Transformer layers are frozen. With this method, MU-LLaMA can produce captions and reply to queries based on the context of music.

BLEU, METEOR, ROUGE-L, and BERT-Rating are the predominant text generation measures used to evaluate MU-LLaMA’s performance. Two primary subtasks are used to check the model: music query answering and music captioning. Comparisons are made with existing large language model (LLM) based models for addressing music questions, specifically the LTU model and the LLaMA Adapter with ImageBind encoder. In every metric, MU-LLaMA performs higher than comparable models, demonstrating its ability to reply accurately and contextually to questions on music. MU-LLaMA has competition from Whisper Audio Captioning (WAC), MusCaps, LTU, and LP-MusicCaps in music captioning. The outcomes highlight MU-LLaMA’s capability to provide high-quality captions for music files by demonstrating its superiority in BLEU, METEOR, and ROUGE-L criteria.

In conclusion, MU-LLaMA shows promise to handle text-to-music generating issues while demonstrating improvements in music query responding and captioning. The suggested process for producing quite a few music question-answer pairs from existing datasets contributes substantially to the topic. The incontrovertible fact that MU-LLaMA performs higher than existing models indicates that it has the potential to vary the text-to-music generating environment by providing a reliable and adaptable method.

Take a look at the Paper and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our 35k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

In case you like our work, you’ll love our newsletter..