Natural language processing (NLP) has paid much attention to large-scale Transformers. These models, trained on large datasets, have demonstrated amazing emergent abilities in various downstream applications. Notably, comparable pre-training methods have been successfully utilized in voice processing. A promising path to creating universal speech models that may handle many speech tasks inside a single model is large-scale supervised learning. A set of multilingual, multitask models called OpenAI Whisper [15] was developed using 680k hours of labeled voice data that was rigorously chosen from various online sources.

The entire process for model constructing (from data preparation to training) remains to be unavailable to most people despite the publication of pre-trained Whisper models and inference code, which has been a usual situation for giant language models (LLMs). This restriction raises several issues:

1. Because users are unaware of the particular training data, using pre-trained models on latest benchmarks carries the danger of information leakage.
2. Users lack access to the training dynamics. So, researchers have difficulty understanding the underlying mechanisms and illuminating strategies for improving the model’s performance. 
3. Coping with problems regarding robustness, fairness, bias, and toxicity, all of which generally arise because of the information and training process, is made far tougher by the dearth of access to the complete model development pipeline.

By pushing for the publication of comprehensive training pipelines, there has recently been a determined movement to advertise open science in the sphere of LLM research. This inspired the research team from Carnegie Mellon University, Shanghai Jiao Tong University, and Honda Research Institute to create the Open Whisper-style Speech Model (OWSM)2, which uses an open-source toolbox and publicly available data to copy whisper-style training. To handle crucial tasks, including language identification (LID), multilingual automated speech recognition (ASR), and utterance-level segmentation, OWSM adopts the Whisper framework. 

Notably, OWSM also displays several technical innovations. As a substitute of just any-to-English translation, it handles any-to-any speech translation. OWSM also uses a wide range of tactics to enhance efficiency. The whole pipeline, including data preparation, training, inference, and scoring, will likely be covered by reproducible recipes. The team also plans to make pre-trained models and training logs available, letting researchers dig into the mechanics of the training procedure and acquire necessary knowledge for his or her research. 

While OWSM performs similarly to Whisper and even higher on some metrics, its goal will not be to have interaction in a protracted arms race with Whisper. The team’s largest dataset only makes up about 25% of the training set utilized by Whisper, and they can’t execute quite a few trial runs due to resource constraints. 

In the longer term, the team plans to explore the next directions:

1. The present OWSM still lags behind Whisper in lots of benchmarks. The researchers imagine that using more sophisticated encoder or decoder architectures, gathering more varied ASR and ST data from open sources, and incorporating self-supervised speech representations much like Google USM might help.
2. In addition they intend so as to add other speech-processing tasks to the multitask framework, resembling spoken language comprehension and speech synthesis based on discrete representations, to create “universal speech models.”

Try the Paper and Code. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 31k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.

In the event you like our work, you’ll love our newsletter..