The paper introduces VITS2, a single-stage text-to-speech model that synthesizes more natural speech by improving various facets of previous models. The model addresses issues like intermittent unnaturalness, computational efficiency, and dependence on phoneme conversion. The proposed methods enhance naturalness, speech characteristic similarity in multi-speaker models, and training and inference efficiency.
The strong dependence on phoneme conversion in previous works is significantly reduced, allowing for a completely end-to-end single-stage approach.
Previous Methods:
Two-Stage Pipeline Systems: These systems divided the technique of generating waveforms from input texts into two cascaded stages. The primary stage produced intermediate speech representations like mel-spectrograms or linguistic features from the input texts. The second stage then generated raw waveforms based on those intermediate representations. These systems had limitations resembling error propagation from the primary stage to the second, reliance on human-defined features like mel-spectrogram, and the computation required to generate intermediate features.
Single-Stage Models: Recent studies have actively explored single-stage models that directly generate waveforms from input texts. These models haven’t only outperformed the two-stage systems but additionally demonstrated the power to generate high-quality speech nearly indistinguishable from human speech.
by J. Kim, J. Kong, and J. Son was a major prior work in the sector of single-stage text-to-speech synthesis. This previous single-stage approach achieved great success but had several problems, including intermittent unnaturalness, low efficiency of the duration predictor, complex input format, insufficient speaker similarity in multi-speaker models, slow training, and robust dependence on phoneme conversion.
The present paper’s foremost contribution is to handle the problems present in the previous single-stage model, particularly the one mentioned within the above successful model, and introduce improvements to realize higher quality and efficiency in text-to-speech synthesis.
Deep neural network-based text-to-speech has seen significant advancements. The challenge lies in converting discontinuous text into continuous waveforms, ensuring high-quality speech audio. Previous solutions divided the method into two stages: producing intermediate speech representations from texts after which generating raw waveforms based on those representations. Single-stage models have been actively studied and have outperformed two-stage systems. The paper goals to handle issues present in previous single-stage models.
The paper describes improvements in 4 areas: duration prediction, augmented variational autoencoder with normalizing flows, alignment search, and speaker-conditioned text encoder. A stochastic duration predictor is proposed, trained through adversarial learning. The Monotonic Alignment Search (MAS) is used for alignment, with modifications for quality improvement. The model introduces a transformer block into the normalizing flows for capturing long-term dependencies. A speaker-conditioned text encoder is designed to higher mimic the varied speech characteristics of every speaker.
Experiments were conducted on the LJ Speech dataset and the VCTK dataset. The study used each phoneme sequences and normalized texts as model inputs. Networks were trained using the AdamW optimizer, and the training was conducted on NVIDIA V100 GPUs.Crowdsourced mean opinion rating (MOS) tests were conducted to guage the naturalness of the synthesized speech. The proposed method showed significant improvement in the standard of synthesized speech in comparison with previous models. Ablation studies were conducted to confirm the validity of the proposed methods.
Finally, the authors demonstrated the validity of their proposed methods through experiments, quality evaluation, and computation speed measurement but conveyed that various problems still exist in the sector of speech synthesis that have to be addressed, and hope that their work is usually a basis for future research.
Take a look at the Paper and Github. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to hitch our 29k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the newest AI research news, cool AI projects, and more.
Should you like our work, you’ll love our newsletter..
I’m Mahitha Sannala, a Computer Science Master’s student on the University of California, Riverside. I hold a Bachelor’s degree in Computer Science and Engineering from the Indian Institute of Technology, Palakkad. My foremost areas of interest lie in Artificial Intelligence and Machine learning. I’m particularly captivated with working with medical data and to derive invaluable insights from them . As a dedicated learner, I’m desirous to stay updated with the newest advancements within the fields of AI and ML.