riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity
deafc24388
transformers/tests
History
Jonatan Kłosko deafc24388
Add WhisperTokenizerFast (#21222) 
* Add WhisperTokenizerFast

* Fixup

* Up

* Up

* Improve tests

* Update src/transformers/models/whisper/tokenization_whisper_fast.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Keep stride in whisper pipelien test

* Remove unknown token special case

* Reduce vocabulary size in tests

* Fix vocab size assertion

* Sync copied changes from WhisperTokenizer

* Skip pipeline tests

* Update assertion

* Remove Whisper tokenizer dependency on sentencepiece

* Format

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-02-21 06:58:54 +01:00
..
benchmark [Test refactor 1/5] Per-folder tests reorganization (#15725) 2022-02-23 15:46:28 -05:00
deepspeed [tests] add missing report_to none (#21505) 2023-02-08 09:32:40 -08:00
extended Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
fixtures [WIP] add SpeechT5 model (#18922) 2023-02-03 12:43:46 -05:00
generation Generate: filter encoder inputs when its signature does not accept wildcards (#21603) 2023-02-14 10:46:46 +00:00
mixed_int8 [bnb] fix bnb decoders bug (#21688) 2023-02-20 12:21:58 +00:00
models Add WhisperTokenizerFast (#21222) 2023-02-21 06:58:54 +01:00
onnx Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
optimization Add inverse sqrt learning rate scheduler (#21495) 2023-02-07 15:00:50 -05:00
pipelines Add WhisperTokenizerFast (#21222) 2023-02-21 06:58:54 +01:00
repo_utils Cleanup quality (#21493) 2023-02-07 12:27:31 -05:00
sagemaker Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
tokenization Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
trainer Fix epoch number when resuming training (#21478) 2023-02-06 19:34:34 -05:00
utils Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
__init__.py GPU text generation: mMoved the encoded_prompt to correct device 2020-01-06 15:11:12 +01:00
test_configuration_common.py Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
test_feature_extraction_common.py Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
test_image_processing_common.py Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
test_image_transforms.py Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
test_modeling_common.py Fix generation config for empty state dict (#21630) 2023-02-14 10:57:28 -05:00
test_modeling_flax_common.py [Tests] Improve flax test_attention_outputs (#21486) 2023-02-10 11:31:49 -05:00
test_modeling_tf_common.py Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
test_sequence_feature_extraction_common.py Update quality tooling for formatting (#21480) 2023-02-06 18:10:56 -05:00
test_tokenization_common.py Cleanup quality (#21493) 2023-02-07 12:27:31 -05:00