mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-06 14:20:04 +06:00
9aeacb58ba
* [WIP] SP tokenizers * fixing tests for T5 * WIP tokenizers * serialization * update T5 * WIP T5 tokenization * slow to fast conversion script * Refactoring to move tokenzier implementations inside transformers * Adding gpt - refactoring - quality * WIP adding several tokenizers to the fast world * WIP Roberta - moving implementations * update to dev4 switch file loading to in-memory loading * Updating and fixing * advancing on the tokenizers - updating do_lower_case * style and quality * moving forward with tokenizers conversion and tests * MBart, T5 * dumping the fast version of transformer XL * Adding to autotokenizers + style/quality * update init and space_between_special_tokens * style and quality * bump up tokenizers version * add protobuf * fix pickle Bert JP with Mecab * fix newly added tokenizers * style and quality * fix bert japanese * fix funnel * limite tokenizer warning to one occurence * clean up file * fix new tokenizers * fast tokenizers deep tests * WIP adding all the special fast tests on the new fast tokenizers * quick fix * adding more fast tokenizers in the fast tests * all tokenizers in fast version tested * Adding BertGenerationFast * bump up setup.py for CI * remove BertGenerationFast (too early) * bump up tokenizers version * Clean old docstrings * Typo * Update following Lysandre comments Co-authored-by: Sylvain Gugger <sylvain.gugger@gmail.com> |
||
|---|---|---|
| .. | ||
| albert.rst | ||
| auto.rst | ||
| bart.rst | ||
| bert.rst | ||
| bertgeneration.rst | ||
| blenderbot.rst | ||
| camembert.rst | ||
| ctrl.rst | ||
| deberta.rst | ||
| dialogpt.rst | ||
| distilbert.rst | ||
| dpr.rst | ||
| electra.rst | ||
| encoderdecoder.rst | ||
| flaubert.rst | ||
| fsmt.rst | ||
| funnel.rst | ||
| gpt.rst | ||
| gpt2.rst | ||
| layoutlm.rst | ||
| longformer.rst | ||
| lxmert.rst | ||
| marian.rst | ||
| mbart.rst | ||
| mobilebert.rst | ||
| pegasus.rst | ||
| rag.rst | ||
| reformer.rst | ||
| retribert.rst | ||
| roberta.rst | ||
| squeezebert.rst | ||
| t5.rst | ||
| transformerxl.rst | ||
| xlm.rst | ||
| xlmroberta.rst | ||
| xlnet.rst | ||