mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-24 06:48:58 +06:00
827d6d6ef0
* First pass on utility classes and python tokenizers * finishing cleanup pass * style and quality * Fix tests * Updating following @mfuntowicz comment * style and quality * Fix Roberta * fix batch_size/seq_length inBatchEncoding * add alignement methods + tests * Fix OpenAI and Transfo-XL tokenizers * adding trim_offsets=True default for GPT2 et RoBERTa * style and quality * fix tests * add_prefix_space in roberta * bump up tokenizers to rc7 * style * unfortunately tensorfow does like these - removing shape/seq_len for now * Update src/transformers/tokenization_utils.py Co-Authored-By: Stefan Schweter <stefan@schweter.it> * Adding doc and docstrings * making flake8 happy Co-authored-by: Stefan Schweter <stefan@schweter.it> |
||
|---|---|---|
| .. | ||
| configuration.rst | ||
| model.rst | ||
| optimizer_schedules.rst | ||
| pipelines.rst | ||
| processors.rst | ||
| tokenizer.rst | ||