mirror of https://github.com/huggingface/transformers.git synced 2025-07-04 13:20:12 +06:00

Vaibhav Srivastav 4a6024921f

doc: add info about wav2vec2 bert in older wav2vec2 models. (#31120 )

* doc: add info about wav2vec2 bert in older wav2vec2 models.

* apply suggestions from review.

* forward contrib credits from review

---------

Co-authored-by: Sanchit Gandhi <sanchit-gandhi@users.noreply.github.com>

2024-06-05 11:56:11 +01:00

3.3 KiB

Raw Blame History

Wav2Vec2-Conformer

Overview

The Wav2Vec2-Conformer was added to an updated version of fairseq S2T: Fast Speech-to-Text Modeling with fairseq by Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Sravya Popuri, Dmytro Okhonko, Juan Pino.

The official results of the model can be found in Table 3 and Table 4 of the paper.

The Wav2Vec2-Conformer weights were released by the Meta AI team within the Fairseq library.

This model was contributed by patrickvonplaten. The original code can be found here.

Note: Meta (FAIR) released a new version of Wav2Vec2-BERT 2.0 - it's pretrained on 4.5M hours of audio. We especially recommend using it for fine-tuning tasks, e.g. as per this guide.

Usage tips

Wav2Vec2-Conformer follows the same architecture as Wav2Vec2, but replaces the Attention-block with a Conformer-block as introduced in Conformer: Convolution-augmented Transformer for Speech Recognition.
For the same number of layers, Wav2Vec2-Conformer requires more parameters than Wav2Vec2, but also yields an improved word error rate.
Wav2Vec2-Conformer uses the same tokenizer and feature extractor as Wav2Vec2.
Wav2Vec2-Conformer can use either no relative position embeddings, Transformer-XL-like position embeddings, or rotary position embeddings by setting the correct config.position_embeddings_type.

Resources