riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity
827d6d6ef0
transformers/tests
History
Thomas Wolf 827d6d6ef0
Cleanup fast tokenizers integration (#3706) 
* First pass on utility classes and python tokenizers

* finishing cleanup pass

* style and quality

* Fix tests

* Updating following @mfuntowicz comment

* style and quality

* Fix Roberta

* fix batch_size/seq_length inBatchEncoding

* add alignement methods + tests

* Fix OpenAI and Transfo-XL tokenizers

* adding trim_offsets=True default for GPT2 et RoBERTa

* style and quality

* fix tests

* add_prefix_space in roberta

* bump up tokenizers to rc7

* style

* unfortunately tensorfow does like these - removing shape/seq_len for now

* Update src/transformers/tokenization_utils.py

Co-Authored-By: Stefan Schweter <stefan@schweter.it>

* Adding doc and docstrings

* making flake8 happy

Co-authored-by: Stefan Schweter <stefan@schweter.it>
2020-04-18 13:43:57 +02:00
..
fixtures AutoConfig + other Auto classes honor model_type 2020-01-11 02:46:17 +00:00
__init__.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_activations.py get_activation('relu') provides a simple mapping from strings i… (#2807) 2020-02-13 08:28:33 -05:00
test_configuration_auto.py Map configs to models and tokenizers 2020-01-13 23:11:44 +00:00
test_configuration_common.py Pass kwargs to configuration (#3147) 2020-03-05 17:16:57 -05:00
test_doc_samples.py make style 2020-03-04 20:18:07 -05:00
test_hf_api.py [model upload] Support for organizations 2020-03-09 17:33:57 -04:00
test_hf_argparser.py [examples] Generate argparsers from type hints on dataclasses (#3669) 2020-04-10 12:21:58 -04:00
test_model_card.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_modeling_albert.py Added test for AlbertForTokenClassification 2020-02-27 12:24:20 +01:00
test_modeling_auto.py Add missing token classification for XLM (#3277) 2020-03-26 10:22:13 -04:00
test_modeling_bart.py Multilingual BART - (#3602) 2020-04-10 11:25:39 -04:00
test_modeling_bert.py BERT decoder: Fix causal mask dtype. 2020-02-11 15:19:22 -05:00
test_modeling_camembert.py Add camembert integration tests (#3375) 2020-03-24 10:18:37 +01:00
test_modeling_common.py [Config, Caching] Remove output_past everywhere and replace by use_cache argument (#3734) 2020-04-14 14:40:28 -04:00
test_modeling_ctrl.py test ctrl 2020-03-09 13:58:01 +00:00
test_modeling_distilbert.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_modeling_electra.py ELECTRA (#3257) 2020-04-03 14:10:54 -04:00
test_modeling_flaubert.py Correct slow test 2020-02-04 18:05:35 -05:00
test_modeling_gpt2.py Clean special token init in modeling_....py (#3264) 2020-03-20 21:41:04 +01:00
test_modeling_openai.py updated all tests 2020-03-08 15:29:10 +01:00
test_modeling_roberta.py [ci] Re-run integration ground truth from fairseq 2020-03-03 15:31:40 -05:00
test_modeling_t5.py Higher tolerance for past testing in T5 (#3843) 2020-04-17 11:25:14 -04:00
test_modeling_tf_albert.py Question Answering support for Albert and Roberta in TF (#3812) 2020-04-17 10:45:30 -04:00
test_modeling_tf_auto.py Add AutoModelForPreTraining 2020-01-27 14:27:07 -05:00
test_modeling_tf_bert.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_modeling_tf_camembert.py Add camembert integration tests (#3375) 2020-03-24 10:18:37 +01:00
test_modeling_tf_common.py [Config, Caching] Remove output_past everywhere and replace by use_cache argument (#3734) 2020-04-14 14:40:28 -04:00
test_modeling_tf_ctrl.py test ctrl 2020-03-09 13:58:01 +00:00
test_modeling_tf_distilbert.py Definitive HeisenDistilBug fix 2020-01-27 12:09:58 -05:00
test_modeling_tf_electra.py ELECTRA (#3257) 2020-04-03 14:10:54 -04:00
test_modeling_tf_gpt2.py [TFT5, Cache] Add cache to TFT5 (#3772) 2020-04-16 16:14:52 +02:00
test_modeling_tf_openai_gpt.py fixed all tests, still need to check ctrl tf and pt and xlm tf 2020-03-08 21:45:55 +01:00
test_modeling_tf_roberta.py Question Answering support for Albert and Roberta in TF (#3812) 2020-04-17 10:45:30 -04:00
test_modeling_tf_t5.py Higher tolerance for past testing in TF T5 (#3844) 2020-04-17 11:26:16 -04:00
test_modeling_tf_transfo_xl.py Clean special token init in modeling_....py (#3264) 2020-03-20 21:41:04 +01:00
test_modeling_tf_xlm.py fix typo in test xlm tf 2020-03-09 11:34:31 +01:00
test_modeling_tf_xlnet.py fixed all tests, still need to check ctrl tf and pt and xlm tf 2020-03-08 21:45:55 +01:00
test_modeling_transfo_xl.py Shift labels internally within TransfoXLLMHeadModel when called with labels (#3716) 2020-04-13 18:11:23 +02:00
test_modeling_xlm_roberta.py [ci] Re-run integration ground truth from fairseq 2020-03-03 15:31:40 -05:00
test_modeling_xlm.py Add missing token classification for XLM (#3277) 2020-03-26 10:22:13 -04:00
test_modeling_xlnet.py fix xlnet & transfotests 2020-03-08 16:25:03 +01:00
test_optimization_tf.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_optimization.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_pipelines.py clean pipelines (#3795) 2020-04-16 10:21:34 -04:00
test_tokenization_albert.py 💄 super 2020-01-15 18:33:50 -05:00
test_tokenization_auto.py Test correct tokenizers after default switch (#3003) 2020-02-24 18:45:53 -05:00
test_tokenization_bert_japanese.py BertJapaneseTokenizer accept options for mecab (#3566) 2020-04-03 11:12:19 -04:00
test_tokenization_bert.py Tokenizers v3.0.0 (#3185) 2020-04-07 00:29:15 +02:00
test_tokenization_common.py Update quotes 2020-04-09 09:09:00 -04:00
test_tokenization_ctrl.py 💄 super 2020-01-15 18:33:50 -05:00
test_tokenization_distilbert.py Updating the TensorFlow models to work as expected with tokenizers v3.0.0 (#3684) 2020-04-08 16:22:44 -04:00
test_tokenization_fast.py Cleanup fast tokenizers integration (#3706) 2020-04-18 13:43:57 +02:00
test_tokenization_gpt2.py Tokenizers v3.0.0 (#3185) 2020-04-07 00:29:15 +02:00
test_tokenization_openai.py 💄 super 2020-01-15 18:33:50 -05:00
test_tokenization_roberta.py Preserve spaces in GPT-2 tokenizers (#2778) 2020-02-13 13:29:43 -05:00
test_tokenization_t5.py 💄 super 2020-01-15 18:33:50 -05:00
test_tokenization_transfo_xl.py 💄 super 2020-01-15 18:33:50 -05:00
test_tokenization_utils.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_tokenization_xlm_roberta.py XLM-R Tokenizer now passes common tests + Integration tests (#3198) 2020-03-18 09:52:49 -04:00
test_tokenization_xlm.py XLM tokenizer should encode with bos token (#3791) 2020-04-17 11:28:55 -04:00
test_tokenization_xlnet.py 💄 super 2020-01-15 18:33:50 -05:00
utils.py TF GPU CI (#3085) 2020-03-02 15:45:25 -05:00