mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-18 03:58:25 +06:00
827d6d6ef0
* First pass on utility classes and python tokenizers * finishing cleanup pass * style and quality * Fix tests * Updating following @mfuntowicz comment * style and quality * Fix Roberta * fix batch_size/seq_length inBatchEncoding * add alignement methods + tests * Fix OpenAI and Transfo-XL tokenizers * adding trim_offsets=True default for GPT2 et RoBERTa * style and quality * fix tests * add_prefix_space in roberta * bump up tokenizers to rc7 * style * unfortunately tensorfow does like these - removing shape/seq_len for now * Update src/transformers/tokenization_utils.py Co-Authored-By: Stefan Schweter <stefan@schweter.it> * Adding doc and docstrings * making flake8 happy Co-authored-by: Stefan Schweter <stefan@schweter.it> |
||
|---|---|---|
| .. | ||
| albert.rst | ||
| auto.rst | ||
| bart.rst | ||
| bert.rst | ||
| camembert.rst | ||
| ctrl.rst | ||
| dialogpt.rst | ||
| distilbert.rst | ||
| electra.rst | ||
| flaubert.rst | ||
| gpt.rst | ||
| gpt2.rst | ||
| roberta.rst | ||
| t5.rst | ||
| transformerxl.rst | ||
| xlm.rst | ||
| xlmroberta.rst | ||
| xlnet.rst | ||