Thomas Wolf 827d6d6ef0 Cleanup fast tokenizers integration (#3706) ... * First pass on utility classes and python tokenizers * finishing cleanup pass * style and quality * Fix tests * Updating following @mfuntowicz comment * style and quality * Fix Roberta * fix batch_size/seq_length inBatchEncoding * add alignement methods + tests * Fix OpenAI and Transfo-XL tokenizers * adding trim_offsets=True default for GPT2 et RoBERTa * style and quality * fix tests * add_prefix_space in roberta * bump up tokenizers to rc7 * style * unfortunately tensorfow does like these - removing shape/seq_len for now * Update src/transformers/tokenization_utils.py Co-Authored-By: Stefan Schweter <stefan@schweter.it> * Adding doc and docstrings * making flake8 happy Co-authored-by: Stefan Schweter <stefan@schweter.it>		2020-04-18 13:43:57 +02:00
..
adding_a_new_example_script	[examples] Use AutoModels in more examples	2020-03-23 20:11:14 -04:00
adding_a_new_model	Cleanup fast tokenizers integration (#3706)	2020-04-18 13:43:57 +02:00