riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-03 03:31:05 +06:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity
cc12e1dbf6
transformers/tests
History
Nicolas Patry cc12e1dbf6
This will reduce "Already borrowed error": (#12550) 
* This will reduce "Already borrowed error":

Original issue https://github.com/huggingface/tokenizers/issues/537

The original issue is caused by transformers calling many times
mutable functions on the rust tokenizers.
Rust needs to guarantee that only 1 agent has a mutable reference
to memory at a given time (for many reasons which don't need explaining
here). Usually, the rust compiler can guarantee that this property is
true at compile time.

Unfortunately, this is impossible for Python to do that, so PyO3, the
bridge between rust and python used by `tokenizers`, will change the
compile guarantee for a dynamic guarantee, so if multiple agents try
to have multiple mutable borrows at the same time, then the runtime will
yell with "Already borrowed".

The proposed fix here in transformers, is simply to reduce the actual
number of calls that really need mutable borrows. By reducing them,
we reduce the risk of running into "Already borrowed" error.
The caveat is now we add a call to read the current configuration of the
`_tokenizer`, so worst case we have 2 calls instead of 1, and best case
we simply have 1 + a Python comparison of a dict (should be negligible).

* Adding a test.

* trivial error :(.

* Update tests/test_tokenization_fast.py

Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>

* Adding reference to original issues in the tests.

* Update the tests with fast tokenizer.

Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>
2021-07-09 09:36:05 +02:00
..
deepspeed [trainer] 2 bug fixes and a rename (#12309) 2021-06-22 11:13:23 -07:00
extended [trainer] 2 bug fixes and a rename (#12309) 2021-06-22 11:13:23 -07:00
fixtures Add DETR (#11653) 2021-06-09 11:51:13 -04:00
sagemaker [Examples] Added context manager to datasets map (#12367) 2021-06-28 09:14:00 -07:00
__init__.py
conftest.py Trainer push to hub (#11328) 2021-04-23 09:17:37 -04:00
test_activations_tf.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_activations.py Replace legacy tensor.Tensor with torch.tensor/torch.empty (#12027) 2021-06-08 13:58:38 +01:00
test_benchmark_tf.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_benchmark.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_cli.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_configuration_auto.py Reorganize repo (#8580) 2020-11-16 21:43:42 -05:00
test_configuration_common.py Clean push to hub API (#12187) 2021-06-23 10:11:19 -04:00
test_data_collator.py Run mlm pad to multiple for fp16 (#11128) 2021-04-08 16:12:49 -04:00
test_doc_samples.py Fix ignore list behavior in doctests (#8213) 2020-11-02 08:47:37 -05:00
test_feature_extraction_auto.py Add the ImageClassificationPipeline (#11598) 2021-05-07 08:08:40 -04:00
test_feature_extraction_clip.py CLIP (#11445) 2021-05-12 13:48:15 +05:30
test_feature_extraction_common.py Add DETR (#11653) 2021-06-09 11:51:13 -04:00
test_feature_extraction_deit.py Add DETR (#11653) 2021-06-09 11:51:13 -04:00
test_feature_extraction_detr.py Rename detr targets to labels (#12280) 2021-06-29 03:07:46 -04:00
test_feature_extraction_speech_to_text.py Auto feature extractor (#11097) 2021-04-06 19:20:08 -04:00
test_feature_extraction_vit.py Add DETR (#11653) 2021-06-09 11:51:13 -04:00
test_feature_extraction_wav2vec2.py W2v2 test require torch (#10665) 2021-03-11 12:56:12 -05:00
test_file_utils.py Patches the full import failure and adds a test (#10750) 2021-03-16 15:37:52 -04:00
test_flax_auto.py [ci, flax] non-existing models are unlikely to pass tests (#10409) 2021-02-26 12:35:36 +03:00
test_generation_beam_search.py Remove max length beam scorer (#11378) 2021-04-27 00:28:40 +02:00
test_generation_flax_logits_process.py [Flax] Add Beam Search (#12131) 2021-06-16 09:43:54 +01:00
test_generation_flax_utils.py [Flax] Add Beam Search (#12131) 2021-06-16 09:43:54 +01:00
test_generation_logits_process.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_generation_stopping_criteria.py Adding new argument max_new_tokens for generate. (#11476) 2021-05-27 14:22:58 +02:00
test_generation_utils.py Fix head masking generate tests (#12110) 2021-06-11 04:04:07 -04:00
test_hf_api.py Trainer push to hub (#11328) 2021-04-23 09:17:37 -04:00
test_hf_argparser.py Fix default bool in argparser (#12424) 2021-06-30 07:57:05 -04:00
test_image_utils.py Add center_crop to ImageFeatureExtractoMixin (#11066) 2021-04-05 15:28:51 -04:00
test_logging.py fix: The 'warn' method is deprecated (#11105) 2021-04-07 09:20:06 -04:00
test_model_card.py
test_model_output.py
test_modeling_albert.py A cleaner and more scalable implementation of symbolic tracing (#11763) 2021-05-20 18:02:29 +02:00
test_modeling_auto.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_bart.py Replace legacy tensor.Tensor with torch.tensor/torch.empty (#12027) 2021-06-08 13:58:38 +01:00
test_modeling_bert_generation.py Replace error by warning when loading an architecture in another (#11207) 2021-04-13 10:33:52 -04:00
test_modeling_bert.py Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5 (#11475) 2021-05-14 20:57:30 +02:00
test_modeling_big_bird.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_bigbird_pegasus.py Add some tests to the slow suite #11860 2021-05-25 04:06:06 -04:00
test_modeling_blenderbot_small.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_blenderbot.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_bort.py ADD BORT (#9813) 2021-01-27 21:25:11 +03:00
test_modeling_camembert.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_canine.py Add CANINE (#12024) 2021-06-30 08:05:44 -04:00
test_modeling_clip.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_modeling_common.py [roberta] fix lm_head.decoder.weight ignore_key handling (#12446) 2021-07-01 10:31:19 -07:00
test_modeling_convbert.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_ctrl.py Ctrl for sequence classification (#8812) 2020-12-01 09:49:27 +01:00
test_modeling_deberta_v2.py Fix integration slow tests (#10670) 2021-03-11 13:43:53 -05:00
test_modeling_deberta.py Fix integration slow tests (#10670) 2021-03-11 13:43:53 -05:00
test_modeling_deit.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_modeling_detr.py Add DETR (#11653) 2021-06-09 11:51:13 -04:00
test_modeling_distilbert.py A cleaner and more scalable implementation of symbolic tracing (#11763) 2021-05-20 18:02:29 +02:00
test_modeling_dpr.py Fix slow dpr test (#10059) 2021-02-08 04:43:25 -05:00
test_modeling_electra.py Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5 (#11475) 2021-05-14 20:57:30 +02:00
test_modeling_encoder_decoder.py EncoderDecoderConfigs should not create new objects (#11300) 2021-04-25 11:45:46 +02:00
test_modeling_flaubert.py Integration test for FlauBert (#10022) 2021-02-08 04:36:50 -05:00
test_modeling_flax_bart.py Flax T5 (#12150) 2021-06-23 13:13:32 +01:00
test_modeling_flax_bert.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_flax_big_bird.py Flax Big Bird (#11967) 2021-06-14 20:01:03 +01:00
test_modeling_flax_clip.py [Flax] [WIP] allow loading head model with base model weights (#12255) 2021-06-21 15:56:42 +01:00
test_modeling_flax_common.py Clean push to hub API (#12187) 2021-06-23 10:11:19 -04:00
test_modeling_flax_electra.py [Flax] Add Electra models (#11426) 2021-05-04 20:56:09 +02:00
test_modeling_flax_gpt_neo.py FlaxGPTNeo (#12493) 2021-07-06 18:55:18 +05:30
test_modeling_flax_gpt2.py Flax Generate (#11777) 2021-05-27 00:18:17 +01:00
test_modeling_flax_mbart.py [Flax] Add FlaxMBart (#12236) 2021-07-07 12:20:38 +05:30
test_modeling_flax_roberta.py [FlaxRoberta] Add FlaxRobertaModels & adapt run_mlm_flax.py (#11470) 2021-05-04 19:57:59 +02:00
test_modeling_flax_t5.py Flax T5 (#12150) 2021-06-23 13:13:32 +01:00
test_modeling_flax_vit.py Flax Big Bird (#11967) 2021-06-14 20:01:03 +01:00
test_modeling_flax_wav2vec2.py [Flax] Add wav2vec2 (#12271) 2021-06-30 18:44:23 +01:00
test_modeling_fsmt.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_modeling_funnel.py Pytorch - Lazy initialization of models (#11471) 2021-05-05 17:22:20 +02:00
test_modeling_gpt_neo.py Added Sequence Classification class in GPTNeo (#11906) 2021-05-28 06:27:02 -04:00
test_modeling_gpt2.py Added Sequence Classification class in GPTNeo (#11906) 2021-05-28 06:27:02 -04:00
test_modeling_hubert.py [Wav2Vec2, Hubert] Fix ctc loss test (#12458) 2021-07-01 08:59:32 -04:00
test_modeling_ibert.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_modeling_layoutlm.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_modeling_led.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_longformer.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_luke.py Fix integration tests (#12066) 2021-06-08 05:21:38 -04:00
test_modeling_lxmert.py create LxmertModelIntegrationTest Pytorch (#9989) 2021-07-05 05:21:25 -04:00
test_modeling_m2m_100.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_marian.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_mbart.py Replace legacy tensor.Tensor with torch.tensor/torch.empty (#12027) 2021-06-08 13:58:38 +01:00
test_modeling_megatron_bert.py A cleaner and more scalable implementation of symbolic tracing (#11763) 2021-05-20 18:02:29 +02:00
test_modeling_megatron_gpt2.py Fix megatron_gpt2 attention block's causal mask (#12007) 2021-06-14 04:57:55 -04:00
test_modeling_mobilebert.py A cleaner and more scalable implementation of symbolic tracing (#11763) 2021-05-20 18:02:29 +02:00
test_modeling_mpnet.py Add MP Net 2 (#9004) 2020-12-09 10:32:43 -05:00
test_modeling_mt5.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_openai.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_pegasus.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_prophetnet.py Fix usage of head masks by PT encoder-decoder models' generate() function (#11621) 2021-05-19 00:44:53 +01:00
test_modeling_rag.py [RAG] Fix rag from pretrained question encoder generator behavior (#11962) 2021-06-02 09:17:14 +01:00
test_modeling_reformer.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_modeling_roberta.py [roberta] fix lm_head.decoder.weight ignore_key handling (#12446) 2021-07-01 10:31:19 -07:00
test_modeling_roformer.py Add new model RoFormer (use rotary position embedding ) (#11684) 2021-05-20 08:00:34 -04:00
test_modeling_speech_to_text.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_squeezebert.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_t5.py Flax T5 (#12150) 2021-06-23 13:13:32 +01:00
test_modeling_tapas.py Fix TAPAS test uncovered by #12446 (#12480) 2021-07-02 04:35:10 -04:00
test_modeling_tf_albert.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_tf_auto.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_tf_bart.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_bert.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_tf_blenderbot_small.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_blenderbot.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_bort.py ADD BORT (#9813) 2021-01-27 21:25:11 +03:00
test_modeling_tf_camembert.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_tf_common.py [RFC] Laying down building stone for more flexible ONNX export capabilities (#11786) 2021-07-08 10:54:42 -04:00
test_modeling_tf_convbert.py ConvBERT fix torch <> tf weights conversion (#10314) 2021-02-24 14:55:34 +03:00
test_modeling_tf_ctrl.py Make TF CTRL compliant with XLA and AMP (#10209) 2021-02-17 18:54:15 +01:00
test_modeling_tf_distilbert.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_tf_dpr.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_tf_electra.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_tf_flaubert.py Making TF XLM-like models XLA and AMP compliant (#10211) 2021-02-17 18:02:48 +01:00
test_modeling_tf_funnel.py Skip Funnel test 2021-05-05 12:38:01 +02:00
test_modeling_tf_gpt2.py Making TF GPT2 compliant with XLA and AMP (#10230) 2021-02-18 09:36:01 +01:00
test_modeling_tf_layoutlm.py Layout lm tf 2 (#10636) 2021-03-25 12:32:38 -04:00
test_modeling_tf_led.py Fix usage of head masks by TF encoder-decoder models' generate() function (#11775) 2021-05-26 14:02:44 +01:00
test_modeling_tf_longformer.py Making TF Longformer-like models compliant with AMP (#10233) 2021-02-22 15:41:56 +01:00
test_modeling_tf_lxmert.py implementing tflxmertmodel integration test (#12497) 2021-07-06 11:44:47 -04:00
test_modeling_tf_marian.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_mbart.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_mobilebert.py Making TF MobileBert model compliant with AMP (#10259) 2021-02-19 06:55:25 -05:00
test_modeling_tf_mpnet.py Making TF MPNet model compliant with XLA (#10260) 2021-02-19 06:56:41 -05:00
test_modeling_tf_mt5.py Adjust loss difference (#10669) 2021-03-12 09:09:46 +03:00
test_modeling_tf_openai.py Making TF OpenAI GPT model compliant with AMP and XLA (#10261) 2021-02-19 09:33:25 -05:00
test_modeling_tf_pegasus.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_pytorch.py Optional layers (#8961) 2020-12-08 09:14:09 -05:00
test_modeling_tf_rag.py make rag tests smaller (#10679) 2021-03-15 10:07:12 +03:00
test_modeling_tf_roberta.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_tf_roformer.py Add new model RoFormer (use rotary position embedding ) (#11684) 2021-05-20 08:00:34 -04:00
test_modeling_tf_t5.py ByT5 model (#11971) 2021-06-01 19:07:37 +01:00
test_modeling_tf_transfo_xl.py Making TF TransfoXL model compliant with AMP (#10264) 2021-02-19 06:58:07 -05:00
test_modeling_tf_wav2vec2.py Add out of vocabulary error to ASR models (#12288) 2021-06-29 08:57:46 +01:00
test_modeling_tf_xlm_roberta.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_tf_xlm.py Making TF XLM-like models XLA and AMP compliant (#10211) 2021-02-17 18:02:48 +01:00
test_modeling_tf_xlnet.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_transfo_xl.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_modeling_visual_bert.py VisualBERT (#10534) 2021-06-02 18:13:08 +05:30
test_modeling_vit.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_modeling_wav2vec2.py [Wav2Vec2, Hubert] Fix ctc loss test (#12458) 2021-07-01 08:59:32 -04:00
test_modeling_xlm_prophetnet.py Ci test tf super slow (#8007) 2020-10-30 10:25:48 -04:00
test_modeling_xlm_roberta.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_xlm.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_xlnet.py Pytorch - Lazy initialization of models (#11471) 2021-05-05 17:22:20 +02:00
test_offline.py offline mode for firewalled envs (part 2) (#10569) 2021-03-08 08:52:20 -08:00
test_onnx_v2.py [RFC] Laying down building stone for more flexible ONNX export capabilities (#11786) 2021-07-08 10:54:42 -04:00
test_onnx.py Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752) 2021-05-18 09:47:28 -04:00
test_optimization_tf.py Use stable functions (#9369) 2021-01-05 03:58:26 -05:00
test_optimization.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_pipelines_automatic_speech_recognition.py Adding support for pipeline("automatic-speech-recognition"). (#11525) 2021-07-07 16:06:48 +02:00
test_pipelines_common.py Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752) 2021-05-18 09:47:28 -04:00
test_pipelines_conversational.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_pipelines_feature_extraction.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_pipelines_fill_mask.py Fixing the pipeline optimization by reindexing targets (V2) (#12330) 2021-07-08 16:58:15 +02:00
test_pipelines_image_classification.py Pipeline update & tests (#12207) 2021-06-17 09:41:16 +02:00
test_pipelines_question_answering.py Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752) 2021-05-18 09:47:28 -04:00
test_pipelines_summarization.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_pipelines_table_question_answering.py Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752) 2021-05-18 09:47:28 -04:00
test_pipelines_text_classification.py Adding pipeline task aliases. (#11247) 2021-04-15 09:51:24 +02:00
test_pipelines_text_generation.py Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752) 2021-05-18 09:47:28 -04:00
test_pipelines_text2text_generation.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_pipelines_token_classification.py Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752) 2021-05-18 09:47:28 -04:00
test_pipelines_translation.py Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752) 2021-05-18 09:47:28 -04:00
test_pipelines_zero_shot.py Fixed: Better names for nlp variables in pipelines' tests and docs. (#11752) 2021-05-18 09:47:28 -04:00
test_processor_clip.py CLIP (#11445) 2021-05-12 13:48:15 +05:30
test_processor_speech_to_text.py Auto feature extractor (#11097) 2021-04-06 19:20:08 -04:00
test_processor_wav2vec2.py [PretrainedFeatureExtractor] + Wav2Vec2FeatureExtractor, Wav2Vec2Processor, Wav2Vec2Tokenizer (#10324) 2021-02-25 17:42:46 +03:00
test_retrieval_rag.py RAG-2nd2end-revamp (#11893) 2021-06-01 07:32:26 +01:00
test_sequence_feature_extraction_common.py Speech2TextTransformer (#10175) 2021-03-10 21:42:04 +05:30
test_skip_decorators.py [testing] rename skip targets + docs (#7863) 2020-10-20 04:39:13 -04:00
test_tokenization_albert.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_auto.py AutoTokenizer: infer the class from the tokenizer config if possible (#12208) 2021-06-17 12:39:22 -04:00
test_tokenization_bart.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_barthez.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_bert_generation.py Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810) 2021-06-14 11:58:44 +02:00
test_tokenization_bert_japanese.py Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810) 2021-06-14 11:58:44 +02:00
test_tokenization_bert.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_bertweet.py Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810) 2021-06-14 11:58:44 +02:00
test_tokenization_big_bird.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_blenderbot.py [PyTorch Bart] Split Bart into different models (#9343) 2021-01-05 22:00:05 +01:00
test_tokenization_byt5.py ByT5 model (#11971) 2021-06-01 19:07:37 +01:00
test_tokenization_camembert.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_canine.py Add CANINE (#12024) 2021-06-30 08:05:44 -04:00
test_tokenization_clip.py Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810) 2021-06-14 11:58:44 +02:00
test_tokenization_common.py Add test for a WordLevel tokenizer model (#12437) 2021-07-01 12:37:07 +02:00
test_tokenization_cpm.py Add a special tokenizer for CPM model (#11068) 2021-04-10 02:07:47 +08:00
test_tokenization_ctrl.py Refactor prepare_seq2seq_batch (#9524) 2021-01-12 18:19:38 -05:00
test_tokenization_deberta_v2.py fix deberta 2 tokenizer integration test (#12017) 2021-06-07 04:55:55 -04:00
test_tokenization_deberta.py Implement Fast Tokenization for Deberta (#11387) 2021-04-30 08:08:15 -04:00
test_tokenization_distilbert.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_tokenization_dpr.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_fast.py This will reduce "Already borrowed error": (#12550) 2021-07-09 09:36:05 +02:00
test_tokenization_fsmt.py Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810) 2021-06-14 11:58:44 +02:00
test_tokenization_funnel.py Reorganize repo (#8580) 2020-11-16 21:43:42 -05:00
test_tokenization_gpt2.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_herbert.py Reorganize repo (#8580) 2020-11-16 21:43:42 -05:00
test_tokenization_layoutlm.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_tokenization_luke.py Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810) 2021-06-14 11:58:44 +02:00
test_tokenization_lxmert.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_tokenization_m2m_100.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_marian.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_mbart.py Enabling multilingual models for translation pipelines. (#10536) 2021-04-16 11:31:35 +02:00
test_tokenization_mbart50.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_mpnet.py [MPNet] Add slow to fast tokenizer converter (#9233) 2020-12-21 15:41:34 +01:00
test_tokenization_openai.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_pegasus.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_phobert.py Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810) 2021-06-14 11:58:44 +02:00
test_tokenization_prophetnet.py Deprecate prepare_seq2seq_batch (#10287) 2021-02-22 12:36:16 -05:00
test_tokenization_rag.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_tokenization_reformer.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_roberta.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_roformer.py [RoFormer] Fix some issues (#12397) 2021-07-06 03:31:57 -04:00
test_tokenization_small_blenderbot.py Feature to use the PreTrainedTokenizerFast class as a stand-alone tokenizer (#11810) 2021-06-14 11:58:44 +02:00
test_tokenization_speech_to_text.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_squeezebert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_t5.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_tapas.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_transfo_xl.py Refactor prepare_seq2seq_batch (#9524) 2021-01-12 18:19:38 -05:00
test_tokenization_utils.py Patch recursive import (#11812) 2021-05-21 06:50:01 -04:00
test_tokenization_wav2vec2.py Fix failing test on Windows Platform (#11589) 2021-05-20 19:54:23 -04:00
test_tokenization_xlm_prophetnet.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_xlm_roberta.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_tokenization_xlm.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_tokenization_xlnet.py Add regression tests for slow sentencepiece tokenizers. (#11737) 2021-06-01 09:24:39 -04:00
test_trainer_callback.py fix: The 'warn' method is deprecated (#11105) 2021-04-07 09:20:06 -04:00
test_trainer_distributed.py [testing] ensure concurrent pytest workers use a unique port for torch.dist (#12166) 2021-06-15 11:12:59 -07:00
test_trainer_seq2seq.py fix bert2bert test (#10063) 2021-02-08 16:04:28 +03:00
test_trainer_tpu.py Examples reorg (#11350) 2021-04-21 11:11:20 -04:00
test_trainer_utils.py [style] consistent nn. and nn.functional: part 3 tests (#12155) 2021-06-14 12:18:22 -07:00
test_trainer.py Clean push to hub API (#12187) 2021-06-23 10:11:19 -04:00
test_utils_check_copies.py Fix failing test on Windows Platform (#11589) 2021-05-20 19:54:23 -04:00
test_versions_utils.py rm require_version_examples (#12088) 2021-06-09 11:02:52 -07:00