mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-18 03:58:25 +06:00
cf3cf304ca
* Replace mecab-python3 with fugashi This replaces mecab-python3 with fugashi for Japanese tokenization. I am the maintainer of both projects. Both projects are MeCab wrappers, so the underlying C++ code is the same. fugashi is the newer wrapper and doesn't use SWIG, so for basic use of the MeCab API it's easier to use. This code insures the use of a version of ipadic installed via pip, which should make versioning and tracking down issues easier. fugashi has wheels for Windows, OSX, and Linux, which will help with issues with installing old versions of mecab-python3 on Windows. Compared to mecab-python3, because fugashi doesn't use SWIG, it doesn't require a C++ runtime to be installed on Windows. In adding this change I removed some code dealing with `cursor`, `token_start`, and `token_end` variables. These variables didn't seem to be used for anything, it is unclear to me why they were there. I ran the tests and they passed, though I couldn't figure out how to run the slow tests (`--runslow` gave an error) and didn't try testing with Tensorflow. * Style fix * Remove unused variable Forgot to delete this... * Adapt doc with install instructions * Fix typo Co-authored-by: sgugger <sylvain.gugger@gmail.com> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
362 lines
76 KiB
ReStructuredText
362 lines
76 KiB
ReStructuredText
Pretrained models
|
|
================================================
|
|
|
|
Here is the full list of the currently provided pretrained models together with a short presentation of each model.
|
|
|
|
For a list that includes community-uploaded models, refer to `https://huggingface.co/models <https://huggingface.co/models>`__.
|
|
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| Architecture | Shortcut name | Details of the model |
|
|
+===================+============================================================+=======================================================================================================================================+
|
|
| BERT | ``bert-base-uncased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on lower-cased English text. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-large-uncased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. |
|
|
| | | | Trained on lower-cased English text. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-base-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on cased English text. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-large-cased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. |
|
|
| | | | Trained on cased English text. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-base-multilingual-uncased`` | | (Original, not recommended) 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on lower-cased text in the top 102 languages with the largest Wikipedias |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/bert/blob/master/multilingual.md>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-base-multilingual-cased`` | | (New, **recommended**) 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on cased text in the top 104 languages with the largest Wikipedias |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/bert/blob/master/multilingual.md>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-base-chinese`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on cased Chinese Simplified and Traditional text. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-base-german-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on cased German text by Deepset.ai |
|
|
| | | |
|
|
| | | (see `details on deepset.ai website <https://deepset.ai/german-bert>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-large-uncased-whole-word-masking`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. |
|
|
| | | | Trained on lower-cased English text using Whole-Word-Masking |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/bert/#bert>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-large-cased-whole-word-masking`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. |
|
|
| | | | Trained on cased English text using Whole-Word-Masking |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/bert/#bert>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-large-uncased-whole-word-masking-finetuned-squad`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. |
|
|
| | | | The ``bert-large-uncased-whole-word-masking`` model fine-tuned on SQuAD |
|
|
| | | |
|
|
| | | (see details of fine-tuning in the `example section <https://github.com/huggingface/transformers/tree/master/examples>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-large-cased-whole-word-masking-finetuned-squad`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters |
|
|
| | | | The ``bert-large-cased-whole-word-masking`` model fine-tuned on SQuAD |
|
|
| | | |
|
|
| | | (see `details of fine-tuning in the example section <https://huggingface.co/transformers/examples.html>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-base-cased-finetuned-mrpc`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | The ``bert-base-cased`` model fine-tuned on MRPC |
|
|
| | | |
|
|
| | | (see `details of fine-tuning in the example section <https://huggingface.co/transformers/examples.html>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-base-german-dbmdz-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on cased German text by DBMDZ |
|
|
| | | |
|
|
| | | (see `details on dbmdz repository <https://github.com/dbmdz/german-bert>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``bert-base-german-dbmdz-uncased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on uncased German text by DBMDZ |
|
|
| | | |
|
|
| | | (see `details on dbmdz repository <https://github.com/dbmdz/german-bert>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``cl-tohoku/bert-base-japanese`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on Japanese text. Text is tokenized with MeCab and WordPiece and this requires some extra dependencies, |
|
|
| | | | `fugashi <https://github.com/polm/fugashi>`__ which is a wrapper around `MeCab <https://taku910.github.io/mecab/>`__. |
|
|
| | | | Use ``pip install transformers["ja"]`` (or ``pip install -e .["ja"]`` if you install from source) to install them. |
|
|
| | | |
|
|
| | | (see `details on cl-tohoku repository <https://github.com/cl-tohoku/bert-japanese>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``cl-tohoku/bert-base-japanese-whole-word-masking`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on Japanese text. Text is tokenized with MeCab and WordPiece and this requires some extra dependencies, |
|
|
| | | | `fugashi <https://github.com/polm/fugashi>`__ which is a wrapper around `MeCab <https://taku910.github.io/mecab/>`__. |
|
|
| | | | Use ``pip install transformers["ja"]`` (or ``pip install -e .["ja"]`` if you install from source) to install them. |
|
|
| | | |
|
|
| | | (see `details on cl-tohoku repository <https://github.com/cl-tohoku/bert-japanese>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``cl-tohoku/bert-base-japanese-char`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on Japanese text. Text is tokenized into characters. |
|
|
| | | |
|
|
| | | (see `details on cl-tohoku repository <https://github.com/cl-tohoku/bert-japanese>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``cl-tohoku/bert-base-japanese-char-whole-word-masking`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on Japanese text using Whole-Word-Masking. Text is tokenized into characters. |
|
|
| | | |
|
|
| | | (see `details on cl-tohoku repository <https://github.com/cl-tohoku/bert-japanese>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``TurkuNLP/bert-base-finnish-cased-v1`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on cased Finnish text. |
|
|
| | | |
|
|
| | | (see `details on turkunlp.org <http://turkunlp.org/FinBERT/>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``TurkuNLP/bert-base-finnish-uncased-v1`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on uncased Finnish text. |
|
|
| | | |
|
|
| | | (see `details on turkunlp.org <http://turkunlp.org/FinBERT/>`__). |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``wietsedv/bert-base-dutch-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | Trained on cased Dutch text. |
|
|
| | | |
|
|
| | | (see `details on wietsedv repository <https://github.com/wietsedv/bertje/>`__). |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| GPT | ``openai-gpt`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | OpenAI GPT English model |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| GPT-2 | ``gpt2`` | | 12-layer, 768-hidden, 12-heads, 117M parameters. |
|
|
| | | | OpenAI GPT-2 English model |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``gpt2-medium`` | | 24-layer, 1024-hidden, 16-heads, 345M parameters. |
|
|
| | | | OpenAI's Medium-sized GPT-2 English model |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``gpt2-large`` | | 36-layer, 1280-hidden, 20-heads, 774M parameters. |
|
|
| | | | OpenAI's Large-sized GPT-2 English model |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``gpt2-xl`` | | 48-layer, 1600-hidden, 25-heads, 1558M parameters. |
|
|
| | | | OpenAI's XL-sized GPT-2 English model |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| Transformer-XL | ``transfo-xl-wt103`` | | 18-layer, 1024-hidden, 16-heads, 257M parameters. |
|
|
| | | | English model trained on wikitext-103 |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| XLNet | ``xlnet-base-cased`` | | 12-layer, 768-hidden, 12-heads, 110M parameters. |
|
|
| | | | XLNet English model |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlnet-large-cased`` | | 24-layer, 1024-hidden, 16-heads, 340M parameters. |
|
|
| | | | XLNet Large English model |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| XLM | ``xlm-mlm-en-2048`` | | 12-layer, 2048-hidden, 16-heads |
|
|
| | | | XLM English model |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-mlm-ende-1024`` | | 6-layer, 1024-hidden, 8-heads |
|
|
| | | | XLM English-German model trained on the concatenation of English and German wikipedia |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-mlm-enfr-1024`` | | 6-layer, 1024-hidden, 8-heads |
|
|
| | | | XLM English-French model trained on the concatenation of English and French wikipedia |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-mlm-enro-1024`` | | 6-layer, 1024-hidden, 8-heads |
|
|
| | | | XLM English-Romanian Multi-language model |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-mlm-xnli15-1024`` | | 12-layer, 1024-hidden, 8-heads |
|
|
| | | | XLM Model pre-trained with MLM on the `15 XNLI languages <https://github.com/facebookresearch/XNLI>`__. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-mlm-tlm-xnli15-1024`` | | 12-layer, 1024-hidden, 8-heads |
|
|
| | | | XLM Model pre-trained with MLM + TLM on the `15 XNLI languages <https://github.com/facebookresearch/XNLI>`__. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-clm-enfr-1024`` | | 6-layer, 1024-hidden, 8-heads |
|
|
| | | | XLM English-French model trained with CLM (Causal Language Modeling) on the concatenation of English and French wikipedia |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-clm-ende-1024`` | | 6-layer, 1024-hidden, 8-heads |
|
|
| | | | XLM English-German model trained with CLM (Causal Language Modeling) on the concatenation of English and German wikipedia |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-mlm-17-1280`` | | 16-layer, 1280-hidden, 16-heads |
|
|
| | | | XLM model trained with MLM (Masked Language Modeling) on 17 languages. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-mlm-100-1280`` | | 16-layer, 1280-hidden, 16-heads |
|
|
| | | | XLM model trained with MLM (Masked Language Modeling) on 100 languages. |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| RoBERTa | ``roberta-base`` | | 12-layer, 768-hidden, 12-heads, 125M parameters |
|
|
| | | | RoBERTa using the BERT-base architecture |
|
|
| | | |
|
|
| | | (see `details <https://github.com/pytorch/fairseq/tree/master/examples/roberta>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``roberta-large`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters |
|
|
| | | | RoBERTa using the BERT-large architecture |
|
|
| | | |
|
|
| | | (see `details <https://github.com/pytorch/fairseq/tree/master/examples/roberta>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``roberta-large-mnli`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters |
|
|
| | | | ``roberta-large`` fine-tuned on `MNLI <http://www.nyu.edu/projects/bowman/multinli/>`__. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/pytorch/fairseq/tree/master/examples/roberta>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``distilroberta-base`` | | 6-layer, 768-hidden, 12-heads, 82M parameters |
|
|
| | | | The DistilRoBERTa model distilled from the RoBERTa model `roberta-base` checkpoint. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``roberta-base-openai-detector`` | | 12-layer, 768-hidden, 12-heads, 125M parameters |
|
|
| | | | ``roberta-base`` fine-tuned by OpenAI on the outputs of the 1.5B-parameter GPT-2 model. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/openai/gpt-2-output-dataset/tree/master/detector>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``roberta-large-openai-detector`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters |
|
|
| | | | ``roberta-large`` fine-tuned by OpenAI on the outputs of the 1.5B-parameter GPT-2 model. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/openai/gpt-2-output-dataset/tree/master/detector>`__) |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| DistilBERT | ``distilbert-base-uncased`` | | 6-layer, 768-hidden, 12-heads, 66M parameters |
|
|
| | | | The DistilBERT model distilled from the BERT model `bert-base-uncased` checkpoint |
|
|
| | | |
|
|
| | | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``distilbert-base-uncased-distilled-squad`` | | 6-layer, 768-hidden, 12-heads, 66M parameters |
|
|
| | | | The DistilBERT model distilled from the BERT model `bert-base-uncased` checkpoint, with an additional linear layer. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``distilbert-base-cased`` | | 6-layer, 768-hidden, 12-heads, 65M parameters |
|
|
| | | | The DistilBERT model distilled from the BERT model `bert-base-cased` checkpoint |
|
|
| | | |
|
|
| | | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``distilbert-base-cased-distilled-squad`` | | 6-layer, 768-hidden, 12-heads, 65M parameters |
|
|
| | | | The DistilBERT model distilled from the BERT model `bert-base-cased` checkpoint, with an additional question answering layer. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``distilgpt2`` | | 6-layer, 768-hidden, 12-heads, 82M parameters |
|
|
| | | | The DistilGPT2 model distilled from the GPT2 model `gpt2` checkpoint. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``distilbert-base-german-cased`` | | 6-layer, 768-hidden, 12-heads, 66M parameters |
|
|
| | | | The German DistilBERT model distilled from the German DBMDZ BERT model `bert-base-german-dbmdz-cased` checkpoint. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``distilbert-base-multilingual-cased`` | | 6-layer, 768-hidden, 12-heads, 134M parameters |
|
|
| | | | The multilingual DistilBERT model distilled from the Multilingual BERT model `bert-base-multilingual-cased` checkpoint. |
|
|
| | | |
|
|
| | | (see `details <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__) |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| CTRL | ``ctrl`` | | 48-layer, 1280-hidden, 16-heads, 1.6B parameters |
|
|
| | | | Salesforce's Large-sized CTRL English model |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| CamemBERT | ``camembert-base`` | | 12-layer, 768-hidden, 12-heads, 110M parameters |
|
|
| | | | CamemBERT using the BERT-base architecture |
|
|
| | | |
|
|
| | | (see `details <https://github.com/pytorch/fairseq/tree/master/examples/camembert>`__) |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| ALBERT | ``albert-base-v1`` | | 12 repeating layers, 128 embedding, 768-hidden, 12-heads, 11M parameters |
|
|
| | | | ALBERT base model |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/ALBERT>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``albert-large-v1`` | | 24 repeating layers, 128 embedding, 1024-hidden, 16-heads, 17M parameters |
|
|
| | | | ALBERT large model |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/ALBERT>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``albert-xlarge-v1`` | | 24 repeating layers, 128 embedding, 2048-hidden, 16-heads, 58M parameters |
|
|
| | | | ALBERT xlarge model |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/ALBERT>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``albert-xxlarge-v1`` | | 12 repeating layer, 128 embedding, 4096-hidden, 64-heads, 223M parameters |
|
|
| | | | ALBERT xxlarge model |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/ALBERT>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``albert-base-v2`` | | 12 repeating layers, 128 embedding, 768-hidden, 12-heads, 11M parameters |
|
|
| | | | ALBERT base model with no dropout, additional training data and longer training |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/ALBERT>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``albert-large-v2`` | | 24 repeating layers, 128 embedding, 1024-hidden, 16-heads, 17M parameters |
|
|
| | | | ALBERT large model with no dropout, additional training data and longer training |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/ALBERT>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``albert-xlarge-v2`` | | 24 repeating layers, 128 embedding, 2048-hidden, 16-heads, 58M parameters |
|
|
| | | | ALBERT xlarge model with no dropout, additional training data and longer training |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/ALBERT>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``albert-xxlarge-v2`` | | 12 repeating layer, 128 embedding, 4096-hidden, 64-heads, 223M parameters |
|
|
| | | | ALBERT xxlarge model with no dropout, additional training data and longer training |
|
|
| | | |
|
|
| | | (see `details <https://github.com/google-research/ALBERT>`__) |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| T5 | ``t5-small`` | | ~60M parameters with 6-layers, 512-hidden-state, 2048 feed-forward hidden-state, 8-heads, |
|
|
| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``t5-base`` | | ~220M parameters with 12-layers, 768-hidden-state, 3072 feed-forward hidden-state, 12-heads, |
|
|
| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``t5-large`` | | ~770M parameters with 24-layers, 1024-hidden-state, 4096 feed-forward hidden-state, 16-heads, |
|
|
| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``t5-3B`` | | ~2.8B parameters with 24-layers, 1024-hidden-state, 16384 feed-forward hidden-state, 32-heads, |
|
|
| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``t5-11B`` | | ~11B parameters with 24-layers, 1024-hidden-state, 65536 feed-forward hidden-state, 128-heads, |
|
|
| | | | Trained on English text: the Colossal Clean Crawled Corpus (C4) |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| XLM-RoBERTa | ``xlm-roberta-base`` | | ~125M parameters with 12-layers, 768-hidden-state, 3072 feed-forward hidden-state, 8-heads, |
|
|
| | | | Trained on on 2.5 TB of newly created clean CommonCrawl data in 100 languages |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``xlm-roberta-large`` | | ~355M parameters with 24-layers, 1027-hidden-state, 4096 feed-forward hidden-state, 16-heads, |
|
|
| | | | Trained on 2.5 TB of newly created clean CommonCrawl data in 100 languages |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| FlauBERT | ``flaubert/flaubert_small_cased`` | | 6-layer, 512-hidden, 8-heads, 54M parameters |
|
|
| | | | FlauBERT small architecture |
|
|
| | | |
|
|
| | | (see `details <https://github.com/getalp/Flaubert>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``flaubert/flaubert_base_uncased`` | | 12-layer, 768-hidden, 12-heads, 137M parameters |
|
|
| | | | FlauBERT base architecture with uncased vocabulary |
|
|
| | | |
|
|
| | | (see `details <https://github.com/getalp/Flaubert>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``flaubert/flaubert_base_cased`` | | 12-layer, 768-hidden, 12-heads, 138M parameters |
|
|
| | | | FlauBERT base architecture with cased vocabulary |
|
|
| | | |
|
|
| | | (see `details <https://github.com/getalp/Flaubert>`__) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``flaubert/flaubert_large_cased`` | | 24-layer, 1024-hidden, 16-heads, 373M parameters |
|
|
| | | | FlauBERT large architecture |
|
|
| | | |
|
|
| | | (see `details <https://github.com/getalp/Flaubert>`__) |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| Bart | ``facebook/bart-large`` | | 24-layer, 1024-hidden, 16-heads, 406M parameters |
|
|
| | | |
|
|
| | | (see `details <https://github.com/pytorch/fairseq/tree/master/examples/bart>`_) |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``facebook/bart-base`` | | 12-layer, 768-hidden, 16-heads, 139M parameters |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``facebook/bart-large-mnli`` | | Adds a 2 layer classification head with 1 million parameters |
|
|
| | | | bart-large base architecture with a classification head, finetuned on MNLI |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``facebook/bart-large-cnn`` | | 12-layer, 1024-hidden, 16-heads, 406M parameters (same as base) |
|
|
| | | | bart-large base architecture finetuned on cnn summarization task |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``facebook/mbart-large-en-ro`` | | 12-layer, 1024-hidden, 16-heads, 880M parameters |
|
|
| | | | bart-large architecture pretrained on cc25 multilingual data , finetuned on WMT english romanian translation. |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| DialoGPT | ``DialoGPT-small`` | | 12-layer, 768-hidden, 12-heads, 124M parameters |
|
|
| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``DialoGPT-medium`` | | 24-layer, 1024-hidden, 16-heads, 355M parameters |
|
|
| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``DialoGPT-large`` | | 36-layer, 1280-hidden, 20-heads, 774M parameters |
|
|
| | | | Trained on English text: 147M conversation-like exchanges extracted from Reddit. |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| Reformer | ``reformer-enwik8`` | | 12-layer, 1024-hidden, 8-heads, 149M parameters |
|
|
| | | | Trained on English Wikipedia data - enwik8. |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``reformer-crime-and-punishment`` | | 6-layer, 256-hidden, 2-heads, 3M parameters |
|
|
| | | | Trained on English text: Crime and Punishment novel by Fyodor Dostoyevsky. |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| MarianMT | ``Helsinki-NLP/opus-mt-{src}-{tgt}`` | | 12-layer, 512-hidden, 8-heads, ~74M parameter Machine translation models. Parameter counts vary depending on vocab size. |
|
|
| | | | (see `model list <https://huggingface.co/Helsinki-NLP>`_) |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| Longformer | ``allenai/longformer-base-4096`` | | 12-layer, 768-hidden, 12-heads, ~149M parameters |
|
|
| | | | Starting from RoBERTa-base checkpoint, trained on documents of max length 4,096 |
|
|
| +------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
|
| | ``allenai/longformer-large-4096`` | | 24-layer, 1024-hidden, 16-heads, ~435M parameters |
|
|
| | | | Starting from RoBERTa-large checkpoint, trained on documents of max length 4,096 |
|
|
+-------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|