mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-20 13:08:21 +06:00
827d6d6ef0
* First pass on utility classes and python tokenizers * finishing cleanup pass * style and quality * Fix tests * Updating following @mfuntowicz comment * style and quality * Fix Roberta * fix batch_size/seq_length inBatchEncoding * add alignement methods + tests * Fix OpenAI and Transfo-XL tokenizers * adding trim_offsets=True default for GPT2 et RoBERTa * style and quality * fix tests * add_prefix_space in roberta * bump up tokenizers to rc7 * style * unfortunately tensorfow does like these - removing shape/seq_len for now * Update src/transformers/tokenization_utils.py Co-Authored-By: Stefan Schweter <stefan@schweter.it> * Adding doc and docstrings * making flake8 happy Co-authored-by: Stefan Schweter <stefan@schweter.it> |
||
|---|---|---|
| .. | ||
| _static | ||
| imgs | ||
| main_classes | ||
| model_doc | ||
| benchmarks.md | ||
| bertology.rst | ||
| conf.py | ||
| converting_tensorflow_models.rst | ||
| examples.md | ||
| favicon.ico | ||
| glossary.rst | ||
| index.rst | ||
| installation.md | ||
| migration.md | ||
| model_sharing.md | ||
| multilingual.rst | ||
| notebooks.md | ||
| pretrained_models.rst | ||
| quickstart.md | ||
| serialization.rst | ||
| torchscript.rst | ||
| usage.rst | ||