transformers/tests
TomLim 1bd604d11c
[WIP] Add Tokenizer for MyT5 Model (#31286)
* Initial commit for MyT5 model

* custom implementation of MyT5 tokenizer, unused files deleted

* unittest for myt5 tokenizer

* upadate of import structure and style

* removed remmanents of MyT5Config

* fixed docstrings

* Updates after review: filled documentaion file, new docstrings and tests added

* Fixed code style issues

* fixed copied from to refer to function

* updated loading myt5 tokenizer in tests, added sample byte map file to fixtures

* changes after review

* removed redundant copied from

* removed redundant copied from

* optimalization and loading model from hf

* [run_slow] myt5

* [run-slow] myt5

* Updated en documentation for myt5

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2024-10-06 10:33:16 +02:00
..
agents Decorator for easier tool building (#33439) 2024-09-18 11:07:51 +02:00
benchmark
bettertransformer Fixed malapropism error (#26660) 2023-10-09 11:04:57 +02:00
deepspeed Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00
extended [tests] skip tests for xpu (#33553) 2024-09-19 19:28:04 +01:00
fixtures Implementation of SuperPoint and AutoModelForKeypointDetection (#28966) 2024-03-19 14:43:02 +00:00
fsdp 🚨🚨🚨 Update min version of accelerate to 0.26.0 (#32627) 2024-08-20 11:42:36 +02:00
generation Cache: revert DynamicCache init for BC (#33861) 2024-10-04 22:47:08 +02:00
models [WIP] Add Tokenizer for MyT5 Model (#31286) 2024-10-06 10:33:16 +02:00
optimization fix: Fixed the 1st argument name in classmethods (#31907) 2024-07-11 12:11:50 +01:00
peft_integration [PEFT] Support low_cpu_mem_usage option for PEFT loading adapters (#33725) 2024-10-03 16:15:36 +02:00
pipelines Make ASR pipeline compliant with Hub spec + add tests (#33769) 2024-10-01 18:15:04 +01:00
quantization Bug fix gguf qwen2moe (#33940) 2024-10-05 16:19:01 +02:00
repo_utils Refactor CI: more explicit (#30674) 2024-08-30 18:17:25 +02:00
sagemaker Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00
tokenization Fix for slow the bug tokenizer adding spaces to single id decodes (#32564) 2024-09-18 12:32:02 +02:00
trainer Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00
utils Cache: revert DynamicCache init for BC (#33861) 2024-10-04 22:47:08 +02:00
__init__.py
test_backbone_common.py Align backbone stage selection with out_indices & out_features (#27606) 2023-12-20 18:33:17 +00:00
test_configuration_common.py Refactor: Removed un-necessary object base class (#32230) 2024-07-26 10:33:02 +02:00
test_feature_extraction_common.py
test_image_processing_common.py Update kwargs validation for preprocess with decorator (#32024) 2024-08-06 11:33:05 +01:00
test_image_transforms.py fix: center_crop occasionally outputs off-by-one dimension matrix (#30934) 2024-05-21 13:56:52 +01:00
test_modeling_common.py Paligemma: fix static cache test (#33941) 2024-10-05 09:47:37 +02:00
test_modeling_flax_common.py add sdpa to ViT [follow up of #29325] (#30555) 2024-05-16 10:56:11 +01:00
test_modeling_tf_common.py [TF] Fix Tensorflow XLA Generation on limited seq_len models (#33903) 2024-10-05 16:20:50 +02:00
test_pipeline_mixin.py Make ASR pipeline compliant with Hub spec + add tests (#33769) 2024-10-01 18:15:04 +01:00
test_processing_common.py Uniformize model processors (#31368) 2024-10-02 10:41:08 +02:00
test_sequence_feature_extraction_common.py Fix typo (#25966) 2023-09-05 10:12:25 +02:00
test_tokenization_common.py Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00