mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-06 14:20:04 +06:00
692e61e91a
* first draft adding Flax-t5-encoder and Flax-mt5-encoder * imports * after make fixup * flax t5 encoder test * black on test * make fix-copies * clean * all_model_classes -> tuple * clean test * is_encoder_decoder=False in t5-enc tester * remove file docstring before FlaxT5Encoder * black * isort * commit suggestions on src/transformers/models/t5/modeling_flax_t5.py Co-authored-by: Suraj Patil <surajp815@gmail.com> * commit suggestions on src/transformers/models/t5/modeling_flax_t5.py Co-authored-by: Suraj Patil <surajp815@gmail.com> * Apply suggestions from code review Co-authored-by: Suraj Patil <surajp815@gmail.com> * remove _get_encoder_module * self.decoder_seq_length -> self.encoder_seq_length as t5-enc does not have decoder * bugfix - self.module_class is class itself, not instance; * docs for mt5 and t5 * call -> __call__ in t5 doc * FlaxMT5EncoderModel to TYPE_HINT * run doc-builder to allow change the files Co-authored-by: Suraj Patil <surajp815@gmail.com>
103 lines
3.2 KiB
Plaintext
103 lines
3.2 KiB
Plaintext
<!--Copyright 2020 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
-->
|
|
|
|
# mT5
|
|
|
|
## Overview
|
|
|
|
The mT5 model was presented in [mT5: A massively multilingual pre-trained text-to-text transformer](https://arxiv.org/abs/2010.11934) by Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya
|
|
Siddhant, Aditya Barua, Colin Raffel.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*The recent "Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain
|
|
state-of-the-art results on a wide variety of English-language NLP tasks. In this paper, we introduce mT5, a
|
|
multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages. We detail
|
|
the design and modified training of mT5 and demonstrate its state-of-the-art performance on many multilingual
|
|
benchmarks. We also describe a simple technique to prevent "accidental translation" in the zero-shot setting, where a
|
|
generative model chooses to (partially) translate its prediction into the wrong language. All of the code and model
|
|
checkpoints used in this work are publicly available.*
|
|
|
|
Note: mT5 was only pre-trained on [mC4](https://huggingface.co/datasets/mc4) excluding any supervised training.
|
|
Therefore, this model has to be fine-tuned before it is useable on a downstream task, unlike the original T5 model.
|
|
Since mT5 was pre-trained unsupervisedly, there's no real advantage to using a task prefix during single-task
|
|
fine-tuning. If you are doing multi-task fine-tuning, you should use a prefix.
|
|
|
|
Google has released the following variants:
|
|
|
|
- [google/mt5-small](https://huggingface.co/google/mt5-small)
|
|
|
|
- [google/mt5-base](https://huggingface.co/google/mt5-base)
|
|
|
|
- [google/mt5-large](https://huggingface.co/google/mt5-large)
|
|
|
|
- [google/mt5-xl](https://huggingface.co/google/mt5-xl)
|
|
|
|
- [google/mt5-xxl](https://huggingface.co/google/mt5-xxl).
|
|
|
|
This model was contributed by [patrickvonplaten](https://huggingface.co/patrickvonplaten). The original code can be
|
|
found [here](https://github.com/google-research/multilingual-t5).
|
|
|
|
## MT5Config
|
|
|
|
[[autodoc]] MT5Config
|
|
|
|
## MT5Tokenizer
|
|
|
|
[[autodoc]] MT5Tokenizer
|
|
|
|
See [`T5Tokenizer`] for all details.
|
|
|
|
|
|
## MT5TokenizerFast
|
|
|
|
[[autodoc]] MT5TokenizerFast
|
|
|
|
See [`T5TokenizerFast`] for all details.
|
|
|
|
|
|
## MT5Model
|
|
|
|
[[autodoc]] MT5Model
|
|
|
|
## MT5ForConditionalGeneration
|
|
|
|
[[autodoc]] MT5ForConditionalGeneration
|
|
|
|
## MT5EncoderModel
|
|
|
|
[[autodoc]] MT5EncoderModel
|
|
|
|
## TFMT5Model
|
|
|
|
[[autodoc]] TFMT5Model
|
|
|
|
## TFMT5ForConditionalGeneration
|
|
|
|
[[autodoc]] TFMT5ForConditionalGeneration
|
|
|
|
## TFMT5EncoderModel
|
|
|
|
[[autodoc]] TFMT5EncoderModel
|
|
|
|
## FlaxMT5Model
|
|
|
|
[[autodoc]] FlaxMT5Model
|
|
|
|
## FlaxMT5ForConditionalGeneration
|
|
|
|
[[autodoc]] FlaxMT5ForConditionalGeneration
|
|
|
|
## FlaxMT5EncoderModel
|
|
|
|
[[autodoc]] FlaxMT5EncoderModel
|