# FastSpeech2Conformer ## Overview The FastSpeech2Conformer model was proposed with the paper [Recent Developments On Espnet Toolkit Boosted By Conformer](https://arxiv.org/abs/2010.13956) by Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi, Jing Shi, Shinji Watanabe, Kun Wei, Wangyou Zhang, and Yuekai Zhang. The abstract from the original FastSpeech2 paper is the following: *Non-autoregressive text to speech (TTS) models such as FastSpeech (Ren et al., 2019) can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.* This model was contributed by [Connor Henderson](https://huggingface.co/connor-henderson). The original code can be found [here](https://github.com/espnet/espnet/blob/master/espnet2/tts/fastspeech2/fastspeech2.py). ## 🤗 Model Architecture FastSpeech2's general structure with a Mel-spectrogram decoder was implemented, and the traditional transformer blocks were replaced with conformer blocks as done in the ESPnet library. #### FastSpeech2 Model Architecture ![FastSpeech2 Model Architecture](https://www.microsoft.com/en-us/research/uploads/prod/2021/04/fastspeech2-1.png) #### Conformer Blocks ![Conformer Blocks](https://www.researchgate.net/profile/Hirofumi-Inaguma-2/publication/344911155/figure/fig2/AS:951455406108673@1603856054097/An-overview-of-Conformer-block.png) #### Convolution Module ![Convolution Module](https://d3i71xaburhd42.cloudfront.net/8809d0732f6147d4ad9218c8f9b20227c837a746/2-Figure1-1.png) ## 🤗 Transformers Usage You can run FastSpeech2Conformer locally with the 🤗 Transformers library. 1. First install the 🤗 [Transformers library](https://github.com/huggingface/transformers), g2p-en: ```bash pip install --upgrade pip pip install --upgrade transformers g2p-en ``` 2. Run inference via the Transformers modelling code with the model and hifigan separately ```python from transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerModel, FastSpeech2ConformerHifiGan import soundfile as sf tokenizer = FastSpeech2ConformerTokenizer.from_pretrained("espnet/fastspeech2_conformer") inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt") input_ids = inputs["input_ids"] model = FastSpeech2ConformerModel.from_pretrained("espnet/fastspeech2_conformer") output_dict = model(input_ids, return_dict=True) spectrogram = output_dict["spectrogram"] hifigan = FastSpeech2ConformerHifiGan.from_pretrained("espnet/fastspeech2_conformer_hifigan") waveform = hifigan(spectrogram) sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050) ``` 3. Run inference via the Transformers modelling code with the model and hifigan combined ```python from transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerWithHifiGan import soundfile as sf tokenizer = FastSpeech2ConformerTokenizer.from_pretrained("espnet/fastspeech2_conformer") inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt") input_ids = inputs["input_ids"] model = FastSpeech2ConformerWithHifiGan.from_pretrained("espnet/fastspeech2_conformer_with_hifigan") output_dict = model(input_ids, return_dict=True) waveform = output_dict["waveform"] sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050) ``` 4. Run inference with a pipeline and specify which vocoder to use ```python from transformers import pipeline, FastSpeech2ConformerHifiGan import soundfile as sf vocoder = FastSpeech2ConformerHifiGan.from_pretrained("espnet/fastspeech2_conformer_hifigan") synthesiser = pipeline(model="espnet/fastspeech2_conformer", vocoder=vocoder) speech = synthesiser("Hello, my dog is cooler than you!") sf.write("speech.wav", speech["audio"].squeeze(), samplerate=speech["sampling_rate"]) ``` ## FastSpeech2ConformerConfig [[autodoc]] FastSpeech2ConformerConfig ## FastSpeech2ConformerHifiGanConfig [[autodoc]] FastSpeech2ConformerHifiGanConfig ## FastSpeech2ConformerWithHifiGanConfig [[autodoc]] FastSpeech2ConformerWithHifiGanConfig ## FastSpeech2ConformerTokenizer [[autodoc]] FastSpeech2ConformerTokenizer - __call__ - save_vocabulary - decode - batch_decode ## FastSpeech2ConformerModel [[autodoc]] FastSpeech2ConformerModel - forward ## FastSpeech2ConformerHifiGan [[autodoc]] FastSpeech2ConformerHifiGan - forward ## FastSpeech2ConformerWithHifiGan [[autodoc]] FastSpeech2ConformerWithHifiGan - forward