diff --git a/README.md b/README.md index 1a898a9f076..ac2e588de43 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,7 @@ Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 1. **[LXMERT](https://huggingface.co/transformers/model_doc/lxmert.html)** (from UNC Chapel Hill) released with the paper [LXMERT: Learning Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answering](https://arxiv.org/abs/1908.07490) by Hao Tan and Mohit Bansal. 1. **[MarianMT](https://huggingface.co/transformers/model_doc/marian.html)** Machine translation models trained using [OPUS](http://opus.nlpl.eu/) data by Jörg Tiedemann. The [Marian Framework](https://marian-nmt.github.io/) is being developed by the Microsoft Translator Team. 1. **[MBart](https://huggingface.co/transformers/model_doc/mbart.html)** (from Facebook) released with the paper [Multilingual Denoising Pre-training for Neural Machine Translation](https://arxiv.org/abs/2001.08210) by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer. +1. **[MT5](https://huggingface.co/transformers/model_doc/mt5.html)** (from Google AI) released with the paper [mT5: A massively multilingual pre-trained text-to-text transformer](https://arxiv.org/abs/2010.11934) by Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel. 1. **[Pegasus](https://huggingface.co/transformers/model_doc/pegasus.html)** (from Google) released with the paper [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://arxiv.org/abs/1912.08777)> by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu. 1. **[ProphetNet](https://huggingface.co/transformers/model_doc/prophetnet.html)** (from Microsoft Research) released with the paper [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou. 1. **[Reformer](https://huggingface.co/transformers/model_doc/reformer.html)** (from Google Research) released with the paper [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya. diff --git a/docs/source/index.rst b/docs/source/index.rst index dfd4164206b..1c70c98584a 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -126,41 +126,44 @@ conversion utilities for the following models: 21. :doc:`MBart ` (from Facebook) released with the paper `Multilingual Denoising Pre-training for Neural Machine Translation `__ by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer. -22. :doc:`Pegasus ` (from Google) released with the paper `PEGASUS: Pre-training with Extracted +22. :doc:`MT5 ` (from Google AI) released with the paper `mT5: A massively multilingual pre-trained + text-to-text transformer `__ by Linting Xue, Noah Constant, Adam Roberts, Mihir + Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, Colin Raffel. +23. :doc:`Pegasus ` (from Google) released with the paper `PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization `__> by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu. -23. :doc:`ProphetNet ` (from Microsoft Research) released with the paper `ProphetNet: Predicting +24. :doc:`ProphetNet ` (from Microsoft Research) released with the paper `ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training `__ by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou. -24. :doc:`Reformer ` (from Google Research) released with the paper `Reformer: The Efficient +25. :doc:`Reformer ` (from Google Research) released with the paper `Reformer: The Efficient Transformer `__ by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya. -25. :doc:`RoBERTa ` (from Facebook), released together with the paper a `Robustly Optimized BERT +26. :doc:`RoBERTa ` (from Facebook), released together with the paper a `Robustly Optimized BERT Pretraining Approach `__ by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. ultilingual BERT into `DistilmBERT `__ and a German version of DistilBERT. -26. :doc:`SqueezeBert ` released with the paper `SqueezeBERT: What can computer vision teach NLP +27. :doc:`SqueezeBert ` released with the paper `SqueezeBERT: What can computer vision teach NLP about efficient neural networks? `__ by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, and Kurt W. Keutzer. -27. :doc:`T5 ` (from Google AI) released with the paper `Exploring the Limits of Transfer Learning with a +28. :doc:`T5 ` (from Google AI) released with the paper `Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer `__ by Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu. -28. :doc:`Transformer-XL ` (from Google/CMU) released with the paper `Transformer-XL: +29. :doc:`Transformer-XL ` (from Google/CMU) released with the paper `Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context `__ by Zihang Dai*, Zhilin Yang*, Yiming Yang, Jaime Carbonell, Quoc V. Le, Ruslan Salakhutdinov. -29. :doc:`XLM ` (from Facebook) released together with the paper `Cross-lingual Language Model +30. :doc:`XLM ` (from Facebook) released together with the paper `Cross-lingual Language Model Pretraining `__ by Guillaume Lample and Alexis Conneau. -30. :doc:`XLM-ProphetNet ` (from Microsoft Research) released with the paper `ProphetNet: +31. :doc:`XLM-ProphetNet ` (from Microsoft Research) released with the paper `ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training `__ by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou. -31. :doc:`XLM-RoBERTa ` (from Facebook AI), released together with the paper `Unsupervised +32. :doc:`XLM-RoBERTa ` (from Facebook AI), released together with the paper `Unsupervised Cross-lingual Representation Learning at Scale `__ by Alexis Conneau*, Kartikay Khandelwal*, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov. -32. :doc:`XLNet ` (from Google/CMU) released with the paper `​XLNet: Generalized Autoregressive +33. :doc:`XLNet ` (from Google/CMU) released with the paper `​XLNet: Generalized Autoregressive Pretraining for Language Understanding `__ by Zhilin Yang*, Zihang Dai*, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le. -33. `Other community models `__, contributed by the `community +34. `Other community models `__, contributed by the `community `__. .. toctree:: diff --git a/docs/source/model_summary.rst b/docs/source/model_summary.rst index 1b6a86b7a67..ea36587d108 100644 --- a/docs/source/model_summary.rst +++ b/docs/source/model_summary.rst @@ -560,6 +560,7 @@ A framework for translation models, using the same models as BART The library provides a version of this model for conditional generation. + T5 ----------------------------------------------------------------------------------------------------------------------- @@ -592,6 +593,28 @@ For instance, if we have the sentence “My dog is very cute .”, and we decide The library provides a version of this model for conditional generation. + +MT5 +----------------------------------------------------------------------------------------------------------------------- + +.. raw:: html + + + Models + + + Doc + + +`mT5: A massively multilingual pre-trained text-to-text transformer `_, Linting Xue +et al. + +The model architecture is same as T5. mT5's pre-training objective includes T5's self-supervised training, but not T5's +supervised training. mT5 is trained on 101 languages. + +The library provides a version of this model for conditional generation. + + MBart -----------------------------------------------------------------------------------------------------------------------