riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-02 19:21:31 +06:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity
b88e0e016d
transformers/tests
History
Nicolas Patry b88e0e016d
[TokenClassification] Label realignment for subword aggregation (#11680) 
* [TokenClassification] Label realignment for subword aggregation

Tentative to replace https://github.com/huggingface/transformers/pull/11622/files

- Added `AggregationStrategy`
- `ignore_subwords` and `grouped_entities` arguments are now fused
  into `aggregation_strategy`. It makes more sense anyway because
  `ignore_subwords=True` with `grouped_entities=False` did not have a
  meaning anyway.
- Added 2 new ways to aggregate which are MAX, and AVERAGE
- AVERAGE requires a bit more information than the others, for now this
case is slightly specific, we should keep that in mind for future
changes.
- Testing has been modified to reflect new argument, and to check the
correct deprecation and the new aggregation_strategy.
- Put the testing argument and testing results for aggregation_strategy,
close together, so that readers can understand what is supposed to
happen.
- `aggregate` is now only tested on a small model as it does not mean
anything to test it globally for all models.
- Previous tests are unchanged in desired output.
- Added a new test case that showcases better the difference between the
  FIRST, MAX and AVERAGE strategies.

* Wrong framework.

* Addressing three issues.

1- Tags might not follow B-, I- convention, so any tag should work now
(assumed as B-TAG)
2- Fixed an issue with average that leads to a substantial code change.
3- The testing suite was not checking for the "index" key for "none"
strategy. This is now fixed.

The issue is that "O" could not be chosen by AVERAGE strategy because
those tokens were filtered out beforehand, so their relative scores were
not counted in the average. Now filtering on
ignore_labels will happen at the very end of the pipeline fixing
that issue.
It's a bit hard to make sure this stays like that because we do
not have a end-to-end test for that behavior

* Formatting.

* Adding formatting to code + cleaner handling of B-, I- tags.

Co-authored-by: Francesco Rubbo <rubbo.francesco@gmail.com>
Co-authored-by: elk-cloner <rezakakhki.rk@gmail.com>

* Typo.

Co-authored-by: Francesco Rubbo <rubbo.francesco@gmail.com>
Co-authored-by: elk-cloner <rezakakhki.rk@gmail.com>
2021-05-18 09:53:20 +02:00
..
deepspeed [cuda ext tests] fixing tests (#11619) 2021-05-06 13:35:28 -07:00
extended [cuda ext tests] fixing tests (#11619) 2021-05-06 13:35:28 -07:00
fixtures Vit deit fixes (#11309) 2021-05-12 11:46:02 -04:00
sagemaker Examples reorg (#11350) 2021-04-21 11:11:20 -04:00
__init__.py
conftest.py Trainer push to hub (#11328) 2021-04-23 09:17:37 -04:00
test_activations_tf.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_activations.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_benchmark_tf.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_benchmark.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_cli.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_configuration_auto.py Reorganize repo (#8580) 2020-11-16 21:43:42 -05:00
test_configuration_common.py Give each test a different repo name (#11453) 2021-04-26 11:52:23 -04:00
test_data_collator.py Run mlm pad to multiple for fp16 (#11128) 2021-04-08 16:12:49 -04:00
test_doc_samples.py Fix ignore list behavior in doctests (#8213) 2020-11-02 08:47:37 -05:00
test_feature_extraction_auto.py Add the ImageClassificationPipeline (#11598) 2021-05-07 08:08:40 -04:00
test_feature_extraction_clip.py CLIP (#11445) 2021-05-12 13:48:15 +05:30
test_feature_extraction_common.py [FeatureExtractorSavingUtils] Refactor PretrainedFeatureExtractor (#10594) 2021-03-09 12:16:59 +03:00
test_feature_extraction_deit.py Add DeiT (PyTorch) (#11056) 2021-04-12 18:07:10 -04:00
test_feature_extraction_speech_to_text.py Auto feature extractor (#11097) 2021-04-06 19:20:08 -04:00
test_feature_extraction_vit.py Add DeiT (PyTorch) (#11056) 2021-04-12 18:07:10 -04:00
test_feature_extraction_wav2vec2.py W2v2 test require torch (#10665) 2021-03-11 12:56:12 -05:00
test_file_utils.py Patches the full import failure and adds a test (#10750) 2021-03-16 15:37:52 -04:00
test_flax_auto.py [ci, flax] non-existing models are unlikely to pass tests (#10409) 2021-02-26 12:35:36 +03:00
test_generation_beam_search.py Remove max length beam scorer (#11378) 2021-04-27 00:28:40 +02:00
test_generation_logits_process.py [Generate] Add save mode logits processor to remove nans and infs if necessary (#10769) 2021-03-23 01:00:05 +03:00
test_generation_stopping_criteria.py Removed max_length from being mandatory within generate. (#11314) 2021-04-21 11:56:45 +02:00
test_generation_utils.py Add BigBirdPegasus (#10991) 2021-05-07 09:27:43 +02:00
test_hf_api.py Trainer push to hub (#11328) 2021-04-23 09:17:37 -04:00
test_hf_argparser.py Adding required flags to non-default arguments in hf_argparser (#10688) 2021-03-15 09:27:55 -04:00
test_image_utils.py Add center_crop to ImageFeatureExtractoMixin (#11066) 2021-04-05 15:28:51 -04:00
test_logging.py fix: The 'warn' method is deprecated (#11105) 2021-04-07 09:20:06 -04:00
test_model_card.py
test_model_output.py Add tests and fix various bugs in ModelOutput (#7073) 2020-09-11 12:01:33 -04:00
test_modeling_albert.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_auto.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_bart.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_bert_generation.py Replace error by warning when loading an architecture in another (#11207) 2021-04-13 10:33:52 -04:00
test_modeling_bert.py Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5 (#11475) 2021-05-14 20:57:30 +02:00
test_modeling_big_bird.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_bigbird_pegasus.py [BigBird Pegasus] Make tests faster (#11744) 2021-05-17 06:30:53 -04:00
test_modeling_blenderbot_small.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_blenderbot.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_bort.py ADD BORT (#9813) 2021-01-27 21:25:11 +03:00
test_modeling_camembert.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_clip.py CLIP (#11445) 2021-05-12 13:48:15 +05:30
test_modeling_common.py Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5 (#11475) 2021-05-14 20:57:30 +02:00
test_modeling_convbert.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_ctrl.py Ctrl for sequence classification (#8812) 2020-12-01 09:49:27 +01:00
test_modeling_deberta_v2.py Fix integration slow tests (#10670) 2021-03-11 13:43:53 -05:00
test_modeling_deberta.py Fix integration slow tests (#10670) 2021-03-11 13:43:53 -05:00
test_modeling_deit.py Add DeiT (PyTorch) (#11056) 2021-04-12 18:07:10 -04:00
test_modeling_distilbert.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_dpr.py Fix slow dpr test (#10059) 2021-02-08 04:43:25 -05:00
test_modeling_electra.py Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5 (#11475) 2021-05-14 20:57:30 +02:00
test_modeling_encoder_decoder.py EncoderDecoderConfigs should not create new objects (#11300) 2021-04-25 11:45:46 +02:00
test_modeling_flaubert.py Integration test for FlauBert (#10022) 2021-02-08 04:36:50 -05:00
test_modeling_flax_bert.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_flax_common.py [FlaxRoberta] Add FlaxRobertaModels & adapt run_mlm_flax.py (#11470) 2021-05-04 19:57:59 +02:00
test_modeling_flax_electra.py [Flax] Add Electra models (#11426) 2021-05-04 20:56:09 +02:00
test_modeling_flax_roberta.py [FlaxRoberta] Add FlaxRobertaModels & adapt run_mlm_flax.py (#11470) 2021-05-04 19:57:59 +02:00
test_modeling_fsmt.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_funnel.py Pytorch - Lazy initialization of models (#11471) 2021-05-05 17:22:20 +02:00
test_modeling_gpt_neo.py [GPTNeo] create local attention mask ones (#11335) 2021-04-20 18:37:44 +05:30
test_modeling_gpt2.py GPT2DoubleHeadsModel made parallelizable (#10658) 2021-03-15 09:10:44 -04:00
test_modeling_ibert.py I-BERT model support (#10153) 2021-02-25 10:06:42 -05:00
test_modeling_layoutlm.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_modeling_led.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_longformer.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_luke.py Add LUKE (#11223) 2021-05-03 09:07:29 -04:00
test_modeling_lxmert.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_modeling_m2m_100.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_marian.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_mbart.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_megatron_bert.py Reactivate Megatron tests an use less workers 2021-04-09 18:09:53 -04:00
test_modeling_mobilebert.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_mpnet.py Add MP Net 2 (#9004) 2020-12-09 10:32:43 -05:00
test_modeling_mt5.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_openai.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_pegasus.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_prophetnet.py Add head_mask, decoder_head_mask, cross_head_mask to ProphetNet (#9964) 2021-04-25 11:06:16 +02:00
test_modeling_rag.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_modeling_reformer.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_roberta.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_speech_to_text.py Fix cross-attention head mask for Torch encoder-decoder models (#10605) 2021-04-23 18:58:06 +02:00
test_modeling_squeezebert.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_t5.py Experimental symbolic tracing feature with torch.fx for BERT, ELECTRA and T5 (#11475) 2021-05-14 20:57:30 +02:00
test_modeling_tapas.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_modeling_tf_albert.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_tf_auto.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_tf_bart.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_bert.py Add support for multiple models for one config in auto classes (#11150) 2021-04-08 18:41:36 -04:00
test_modeling_tf_blenderbot_small.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_blenderbot.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_bort.py ADD BORT (#9813) 2021-01-27 21:25:11 +03:00
test_modeling_tf_camembert.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_tf_common.py Give each test a different repo name (#11453) 2021-04-26 11:52:23 -04:00
test_modeling_tf_convbert.py ConvBERT fix torch <> tf weights conversion (#10314) 2021-02-24 14:55:34 +03:00
test_modeling_tf_ctrl.py Make TF CTRL compliant with XLA and AMP (#10209) 2021-02-17 18:54:15 +01:00
test_modeling_tf_distilbert.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_tf_dpr.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_tf_electra.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_tf_flaubert.py Making TF XLM-like models XLA and AMP compliant (#10211) 2021-02-17 18:02:48 +01:00
test_modeling_tf_funnel.py Skip Funnel test 2021-05-05 12:38:01 +02:00
test_modeling_tf_gpt2.py Making TF GPT2 compliant with XLA and AMP (#10230) 2021-02-18 09:36:01 +01:00
test_modeling_tf_layoutlm.py Layout lm tf 2 (#10636) 2021-03-25 12:32:38 -04:00
test_modeling_tf_led.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_modeling_tf_longformer.py Making TF Longformer-like models compliant with AMP (#10233) 2021-02-22 15:41:56 +01:00
test_modeling_tf_lxmert.py Making TF Lxmert model compliant with AMP (#10257) 2021-02-19 06:54:14 -05:00
test_modeling_tf_marian.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_mbart.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_mobilebert.py Making TF MobileBert model compliant with AMP (#10259) 2021-02-19 06:55:25 -05:00
test_modeling_tf_mpnet.py Making TF MPNet model compliant with XLA (#10260) 2021-02-19 06:56:41 -05:00
test_modeling_tf_mt5.py Adjust loss difference (#10669) 2021-03-12 09:09:46 +03:00
test_modeling_tf_openai.py Making TF OpenAI GPT model compliant with AMP and XLA (#10261) 2021-02-19 09:33:25 -05:00
test_modeling_tf_pegasus.py TF BART models - Add cross_attentions to model output and fix cross-attention head masking (#10699) 2021-04-26 14:16:21 +02:00
test_modeling_tf_pytorch.py Optional layers (#8961) 2020-12-08 09:14:09 -05:00
test_modeling_tf_rag.py make rag tests smaller (#10679) 2021-03-15 10:07:12 +03:00
test_modeling_tf_roberta.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_tf_t5.py Fix XLA and AMP (#10262) 2021-02-19 06:57:16 -05:00
test_modeling_tf_transfo_xl.py Making TF TransfoXL model compliant with AMP (#10264) 2021-02-19 06:58:07 -05:00
test_modeling_tf_xlm_roberta.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_tf_xlm.py Making TF XLM-like models XLA and AMP compliant (#10211) 2021-02-17 18:02:48 +01:00
test_modeling_tf_xlnet.py Check TF ops for ONNX compliance (#10025) 2021-02-15 07:55:10 -05:00
test_modeling_transfo_xl.py Pytorch - Lazy initialization of models (#11471) 2021-05-05 17:22:20 +02:00
test_modeling_vit.py Add DeiT (PyTorch) (#11056) 2021-04-12 18:07:10 -04:00
test_modeling_wav2vec2.py Pytorch - Lazy initialization of models (#11471) 2021-05-05 17:22:20 +02:00
test_modeling_xlm_prophetnet.py Ci test tf super slow (#8007) 2020-10-30 10:25:48 -04:00
test_modeling_xlm_roberta.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_modeling_xlm.py Add multi-class, multi-label and regression to transformers (#11012) 2021-05-04 02:23:40 -04:00
test_modeling_xlnet.py Pytorch - Lazy initialization of models (#11471) 2021-05-05 17:22:20 +02:00
test_offline.py offline mode for firewalled envs (part 2) (#10569) 2021-03-08 08:52:20 -08:00
test_onnx.py Onnx fix test (#10663) 2021-03-11 13:38:29 -05:00
test_optimization_tf.py Use stable functions (#9369) 2021-01-05 03:58:26 -05:00
test_optimization.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_pipelines_automatic_speech_recognition.py Adding AutomaticSpeechRecognitionPipeline. (#11337) 2021-04-30 11:54:08 +02:00
test_pipelines_common.py [PretrainedFeatureExtractor] + Wav2Vec2FeatureExtractor, Wav2Vec2Processor, Wav2Vec2Tokenizer (#10324) 2021-02-25 17:42:46 +03:00
test_pipelines_conversational.py Fixing conversation test for torch 1.8 (#10545) 2021-03-05 09:24:14 -05:00
test_pipelines_feature_extraction.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_pipelines_fill_mask.py Adding skip_special_tokens=True to FillMaskPipeline (#9783) 2021-01-26 10:06:28 +01:00
test_pipelines_image_classification.py Add the ImageClassificationPipeline (#11598) 2021-05-07 08:08:40 -04:00
test_pipelines_question_answering.py Fixes NoneType exception when topk is larger than one coupled with a small context in the Question-Answering pipeline (#11628) 2021-05-10 13:28:10 -04:00
test_pipelines_summarization.py Fix torch 1.8.0 segmentation fault (#10546) 2021-03-05 12:10:19 -05:00
test_pipelines_table_question_answering.py Adding a test to prevent late failure in the Table question answering (#9808) 2021-01-27 04:10:53 -05:00
test_pipelines_text_classification.py Adding pipeline task aliases. (#11247) 2021-04-15 09:51:24 +02:00
test_pipelines_text_generation.py Adding a new return_full_text parameter to TextGenerationPipeline. (#9852) 2021-01-29 10:27:32 +01:00
test_pipelines_text2text_generation.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_pipelines_token_classification.py [TokenClassification] Label realignment for subword aggregation (#11680) 2021-05-18 09:53:20 +02:00
test_pipelines_translation.py Enabling multilingual models for translation pipelines. (#10536) 2021-04-16 11:31:35 +02:00
test_pipelines_zero_shot.py zero-shot pipeline multi_class -> multi_label (#10727) 2021-03-15 16:02:46 -06:00
test_processor_clip.py CLIP (#11445) 2021-05-12 13:48:15 +05:30
test_processor_speech_to_text.py Auto feature extractor (#11097) 2021-04-06 19:20:08 -04:00
test_processor_wav2vec2.py [PretrainedFeatureExtractor] + Wav2Vec2FeatureExtractor, Wav2Vec2Processor, Wav2Vec2Tokenizer (#10324) 2021-02-25 17:42:46 +03:00
test_retrieval_rag.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_sequence_feature_extraction_common.py Speech2TextTransformer (#10175) 2021-03-10 21:42:04 +05:30
test_skip_decorators.py [testing] rename skip targets + docs (#7863) 2020-10-20 04:39:13 -04:00
test_tokenization_albert.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_auto.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_bart.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_barthez.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_bert_generation.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_bert_japanese.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_tokenization_bert.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_bertweet.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_big_bird.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_blenderbot.py [PyTorch Bart] Split Bart into different models (#9343) 2021-01-05 22:00:05 +01:00
test_tokenization_camembert.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_clip.py CLIP (#11445) 2021-05-12 13:48:15 +05:30
test_tokenization_common.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_cpm.py Add a special tokenizer for CPM model (#11068) 2021-04-10 02:07:47 +08:00
test_tokenization_ctrl.py Refactor prepare_seq2seq_batch (#9524) 2021-01-12 18:19:38 -05:00
test_tokenization_deberta_v2.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_deberta.py Implement Fast Tokenization for Deberta (#11387) 2021-04-30 08:08:15 -04:00
test_tokenization_distilbert.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_tokenization_dpr.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_fsmt.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_tokenization_funnel.py Reorganize repo (#8580) 2020-11-16 21:43:42 -05:00
test_tokenization_gpt2.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_herbert.py Reorganize repo (#8580) 2020-11-16 21:43:42 -05:00
test_tokenization_layoutlm.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_tokenization_luke.py Add LUKE (#11223) 2021-05-03 09:07:29 -04:00
test_tokenization_lxmert.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_tokenization_m2m_100.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_marian.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_mbart.py Enabling multilingual models for translation pipelines. (#10536) 2021-04-16 11:31:35 +02:00
test_tokenization_mbart50.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_mpnet.py [MPNet] Add slow to fast tokenizer converter (#9233) 2020-12-21 15:41:34 +01:00
test_tokenization_openai.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_pegasus.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_phobert.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_prophetnet.py Deprecate prepare_seq2seq_batch (#10287) 2021-02-22 12:36:16 -05:00
test_tokenization_rag.py Copyright (#8970) 2020-12-07 18:36:34 -05:00
test_tokenization_reformer.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_roberta.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_small_blenderbot.py [PyTorch Bart] Split Bart into different models (#9343) 2021-01-05 22:00:05 +01:00
test_tokenization_speech_to_text.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_squeezebert.py [Dependencies|tokenizers] Make both SentencePiece and Tokenizers optional dependencies (#7659) 2020-10-18 20:51:24 +02:00
test_tokenization_t5.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_tapas.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_tokenization_transfo_xl.py Refactor prepare_seq2seq_batch (#9524) 2021-01-12 18:19:38 -05:00
test_tokenization_utils.py Documentation about loading a fast tokenizer within Transformers (#11029) 2021-04-05 10:51:16 -04:00
test_tokenization_wav2vec2.py [Wav2vec2] Fixed tokenization mistakes while adding single-char tokens to tokenizer (#11538) 2021-05-03 17:19:12 +02:00
test_tokenization_xlm_prophetnet.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_xlm_roberta.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_tokenization_xlm.py make style (#11442) 2021-04-26 13:50:34 +02:00
test_tokenization_xlnet.py Enable option for subword regularization in more tokenizers. (#11417) 2021-05-13 02:44:55 -04:00
test_trainer_callback.py fix: The 'warn' method is deprecated (#11105) 2021-04-07 09:20:06 -04:00
test_trainer_distributed.py Enforce string-formatting with f-strings (#10980) 2021-03-31 10:00:27 -04:00
test_trainer_seq2seq.py fix bert2bert test (#10063) 2021-02-08 16:04:28 +03:00
test_trainer_tpu.py Examples reorg (#11350) 2021-04-21 11:11:20 -04:00
test_trainer_utils.py Accepts BatchEncoding in LengthSampler (#11431) 2021-04-30 08:27:46 -04:00
test_trainer.py Auto modelcard (#11599) 2021-05-11 11:30:34 -04:00
test_utils_check_copies.py Check copies blackify (#10775) 2021-03-17 18:11:20 -04:00
test_versions_utils.py [versions] handle version requirement ranges (#11110) 2021-04-07 09:09:38 -07:00