riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity
ba8c4d0ac0
transformers/tests
History
Thomas Wolf ba8c4d0ac0
[Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 
* splitting fast and slow tokenizers [WIP]

* [WIP] splitting sentencepiece and tokenizers dependencies

* update dummy objects

* add name_or_path to models and tokenizers

* prefix added to file names

* prefix

* styling + quality

* spliting all the tokenizer files - sorting sentencepiece based ones

* update tokenizer version up to 0.9.0

* remove hard dependency on sentencepiece 🎉

* and removed hard dependency on tokenizers 🎉

* update conversion script

* update missing models

* fixing tests

* move test_tokenization_fast to main tokenization tests - fix bugs

* bump up tokenizers

* fix bert_generation

* update ad fix several tokenizers

* keep sentencepiece in deps for now

* fix funnel and deberta tests

* fix fsmt

* fix marian tests

* fix layoutlm

* fix squeezebert and gpt2

* fix T5 tokenization

* fix xlnet tests

* style

* fix mbart

* bump up tokenizers to 0.9.2

* fix model tests

* fix tf models

* fix seq2seq examples

* fix tests without sentencepiece

* fix slow => fast  conversion without sentencepiece

* update auto and bert generation tests

* fix mbart tests

* fix auto and common test without tokenizers

* fix tests without tokenizers

* clean up tests lighten up when tokenizers + sentencepiece are both off

* style quality and tests fixing

* add sentencepiece to doc/examples reqs

* leave sentencepiece on for now

* style quality split hebert and fix pegasus

* WIP Herbert fast

* add sample_text_no_unicode and fix hebert tokenization

* skip FSMT example test for now

* fix style

* fix fsmt in example tests

* update following Lysandre and Sylvain's comments

* Update src/transformers/testing_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/testing_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/tokenization_utils_base.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/tokenization_utils_base.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2020-10-18 20:51:24 +02:00
..
fixtures [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
__init__.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
conftest.py ignore FutureWarning in tests (#7079) 2020-09-14 07:50:51 -04:00
test_activations_tf.py Refactoring the TF activations functions (#7150) 2020-09-16 07:03:47 -04:00
test_activations.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_benchmark_tf.py [Benchmarks] Change all args to from no_... to their positive form (#7075) 2020-09-23 13:25:24 -04:00
test_benchmark.py [Benchmarks] Change all args to from no_... to their positive form (#7075) 2020-09-23 13:25:24 -04:00
test_cli.py [transformers-cli] fix logger getter (#6777) 2020-08-27 20:01:17 -04:00
test_configuration_auto.py Move tests/utils.py -> transformers/testing_utils.py (#5350) 2020-07-01 10:31:17 -04:00
test_configuration_common.py Pass kwargs to configuration (#3147) 2020-03-05 17:16:57 -05:00
test_data_collator.py Mark big downloads slow (#7325) 2020-09-22 12:21:52 -04:00
test_doc_samples.py Move tests/utils.py -> transformers/testing_utils.py (#5350) 2020-07-01 10:31:17 -04:00
test_hf_api.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_hf_argparser.py parse arguments from dict (#4869) 2020-07-31 04:44:23 -04:00
test_logging.py adding TRANSFORMERS_VERBOSITY env var (#6961) 2020-09-09 04:08:01 -04:00
test_model_card.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_model_output.py Add tests and fix various bugs in ModelOutput (#7073) 2020-09-11 12:01:33 -04:00
test_modeling_albert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_auto.py Blenderbot (#7418) 2020-10-07 19:09:23 -04:00
test_modeling_bart.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_bert_generation.py Add "Leveraging Pretrained Checkpoints for Generation" Seq2Seq models. (#6594) 2020-09-10 16:40:51 +02:00
test_modeling_bert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_blenderbot.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_camembert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_common.py Blenderbot (#7418) 2020-10-07 19:09:23 -04:00
test_modeling_ctrl.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_deberta.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_distilbert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_dpr.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_electra.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_encoder_decoder.py clean naming (#7068) 2020-09-11 09:57:53 +02:00
test_modeling_flaubert.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_fsmt.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_funnel.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_gpt2.py Add batch inferencing support for GPT2LMHeadModel (#7552) 2020-10-14 13:40:24 +02:00
test_modeling_layoutlm.py Add LayoutLM Model (#7064) 2020-09-22 09:28:02 -04:00
test_modeling_longformer.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_lxmert.py fix (#6946) 2020-09-04 16:08:54 +02:00
test_modeling_marian.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_mbart.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_mobilebert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_openai.py Gpt1 for sequence classification (#7683) 2020-10-13 05:06:15 -04:00
test_modeling_pegasus.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_rag.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_reformer.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_roberta.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_squeezebert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_t5.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_tf_albert.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_tf_auto.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_tf_bert.py Custom TF weights loading (#7422) 2020-10-05 09:58:45 -04:00
test_modeling_tf_camembert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_tf_common.py [Seq2Seq] Fix a couple of bugs and clean examples (#7474) 2020-10-01 17:38:50 +02:00
test_modeling_tf_ctrl.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_distilbert.py test_tf_common: remove un_used mixin class parameters (#6866) 2020-09-02 10:54:40 -04:00
test_modeling_tf_electra.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_flaubert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_tf_funnel.py Fix saving TF custom models (#7291) 2020-09-22 09:31:13 -04:00
test_modeling_tf_gpt2.py [Seq2Seq] Fix a couple of bugs and clean examples (#7474) 2020-10-01 17:38:50 +02:00
test_modeling_tf_longformer.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_tf_lxmert.py Adding the LXMERT pretraining model (MultiModal languageXvision) to HuggingFace's suite of models (#5793) 2020-09-03 04:02:25 -04:00
test_modeling_tf_mobilebert.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_modeling_tf_openai.py [Generate] Facilitate PyTorch generate using ModelOutputs (#6735) 2020-09-01 12:38:25 +02:00
test_modeling_tf_roberta.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_tf_t5.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_tf_transfo_xl.py test_tf_common: remove un_used mixin class parameters (#6866) 2020-09-02 10:54:40 -04:00
test_modeling_tf_xlm_roberta.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_tf_xlm.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_tf_xlnet.py test_tf_common: remove un_used mixin class parameters (#6866) 2020-09-02 10:54:40 -04:00
test_modeling_transfo_xl.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_xlm_roberta.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_modeling_xlm.py Black 20 release 2020-08-26 17:20:22 +02:00
test_modeling_xlnet.py [Seq2Seq] Fix a couple of bugs and clean examples (#7474) 2020-10-01 17:38:50 +02:00
test_onnx.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_optimization_tf.py Update repo to isort v5 (#6686) 2020-08-24 11:03:01 -04:00
test_optimization.py Format 2020-08-27 18:31:51 +02:00
test_pipelines.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_retrieval_rag.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_skip_decorators.py [testing] skip decorators: docs, tests, bugs (#7334) 2020-09-23 05:16:19 -04:00
test_tokenization_albert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_auto.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_bart.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_bert_generation.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_bert_japanese.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_bert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_bertweet.py Add new pre-trained models BERTweet and PhoBERT (#6129) 2020-09-18 13:16:43 -04:00
test_tokenization_blenderbot.py Blenderbot (#7418) 2020-10-07 19:09:23 -04:00
test_tokenization_camembert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_common.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_ctrl.py Adding Fast tokenizers for SentencePiece based tokenizers - Breaking: remove Transfo-XL fast tokenizer (#7141) 2020-10-08 11:32:16 +02:00
test_tokenization_deberta.py Add DeBERTa model (#5929) 2020-09-30 07:07:30 -04:00
test_tokenization_distilbert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_dpr.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_fsmt.py [ported model] FSMT (FairSeq MachineTranslation) (#6940) 2020-09-17 11:31:29 -04:00
test_tokenization_funnel.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_gpt2.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_herbert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_layoutlm.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_lxmert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_marian.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_mbart.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_openai.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_pegasus.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_phobert.py Add new pre-trained models BERTweet and PhoBERT (#6129) 2020-09-18 13:16:43 -04:00
test_tokenization_rag.py [Rag] Fix loading of pretrained Rag Tokenizer (#7756) 2020-10-13 14:34:22 +02:00
test_tokenization_reformer.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_roberta.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_squeezebert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_t5.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_transfo_xl.py Adding Fast tokenizers for SentencePiece based tokenizers - Breaking: remove Transfo-XL fast tokenizer (#7141) 2020-10-08 11:32:16 +02:00
test_tokenization_utils.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_xlm_roberta.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_xlm.py Adding Fast tokenizers for SentencePiece based tokenizers - Breaking: remove Transfo-XL fast tokenizer (#7141) 2020-10-08 11:32:16 +02:00
test_tokenization_xlnet.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_trainer_callback.py fix/hide warnings (#7837) 2020-10-16 03:19:51 -04:00
test_trainer_distributed.py Add predict step accumulation (#7767) 2020-10-14 11:41:45 -04:00
test_trainer_tpu.py Add predict step accumulation (#7767) 2020-10-14 11:41:45 -04:00
test_trainer_utils.py Add predict step accumulation (#7767) 2020-10-14 11:41:45 -04:00
test_trainer.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_utils_check_copies.py Get a better error when check_copies fails (#7457) 2020-09-30 10:05:14 +02:00