mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-05 05:40:05 +06:00
eb849f6604
* Rename index.mdx to index.md * With saved modifs * Address review comment * Treat all files * .mdx -> .md * Remove special char * Update utils/tests_fetcher.py Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr> --------- Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
80 lines
3.3 KiB
Markdown
80 lines
3.3 KiB
Markdown
<!--Copyright 2021 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|
rendered properly in your Markdown viewer.
|
|
|
|
-->
|
|
|
|
# Hubert
|
|
|
|
## Overview
|
|
|
|
Hubert was proposed in [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://arxiv.org/abs/2106.07447) by Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan
|
|
Salakhutdinov, Abdelrahman Mohamed.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*Self-supervised approaches for speech representation learning are challenged by three unique problems: (1) there are
|
|
multiple sound units in each input utterance, (2) there is no lexicon of input sound units during the pre-training
|
|
phase, and (3) sound units have variable lengths with no explicit segmentation. To deal with these three problems, we
|
|
propose the Hidden-Unit BERT (HuBERT) approach for self-supervised speech representation learning, which utilizes an
|
|
offline clustering step to provide aligned target labels for a BERT-like prediction loss. A key ingredient of our
|
|
approach is applying the prediction loss over the masked regions only, which forces the model to learn a combined
|
|
acoustic and language model over the continuous inputs. HuBERT relies primarily on the consistency of the unsupervised
|
|
clustering step rather than the intrinsic quality of the assigned cluster labels. Starting with a simple k-means
|
|
teacher of 100 clusters, and using two iterations of clustering, the HuBERT model either matches or improves upon the
|
|
state-of-the-art wav2vec 2.0 performance on the Librispeech (960h) and Libri-light (60,000h) benchmarks with 10min, 1h,
|
|
10h, 100h, and 960h fine-tuning subsets. Using a 1B parameter model, HuBERT shows up to 19% and 13% relative WER
|
|
reduction on the more challenging dev-other and test-other evaluation subsets.*
|
|
|
|
Tips:
|
|
|
|
- Hubert is a speech model that accepts a float array corresponding to the raw waveform of the speech signal.
|
|
- Hubert model was fine-tuned using connectionist temporal classification (CTC) so the model output has to be decoded
|
|
using [`Wav2Vec2CTCTokenizer`].
|
|
|
|
This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten).
|
|
|
|
## Documentation resources
|
|
|
|
- [Audio classification task guide](../tasks/audio_classification)
|
|
- [Automatic speech recognition task guide](../tasks/asr)
|
|
|
|
## HubertConfig
|
|
|
|
[[autodoc]] HubertConfig
|
|
|
|
## HubertModel
|
|
|
|
[[autodoc]] HubertModel
|
|
- forward
|
|
|
|
## HubertForCTC
|
|
|
|
[[autodoc]] HubertForCTC
|
|
- forward
|
|
|
|
## HubertForSequenceClassification
|
|
|
|
[[autodoc]] HubertForSequenceClassification
|
|
- forward
|
|
|
|
## TFHubertModel
|
|
|
|
[[autodoc]] TFHubertModel
|
|
- call
|
|
|
|
## TFHubertForCTC
|
|
|
|
[[autodoc]] TFHubertForCTC
|
|
- call
|