riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity
deafc24388
transformers/tests/models/whisper
History
Jonatan Kłosko deafc24388
Add WhisperTokenizerFast (#21222) 
* Add WhisperTokenizerFast

* Fixup

* Up

* Up

* Improve tests

* Update src/transformers/models/whisper/tokenization_whisper_fast.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Keep stride in whisper pipelien test

* Remove unknown token special case

* Reduce vocabulary size in tests

* Fix vocab size assertion

* Sync copied changes from WhisperTokenizer

* Skip pipeline tests

* Update assertion

* Remove Whisper tokenizer dependency on sentencepiece

* Format

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-02-21 06:58:54 +01:00
..
__init__.py Add WhisperModel to transformers (#19166) 2022-10-05 22:28:31 +02:00
test_feature_extraction_whisper.py Remove CLI spams with Whisper FeatureExtractor (#21267) 2023-02-10 09:15:16 -05:00
test_modeling_flax_whisper.py Fix quality 2023-02-20 03:27:09 -05:00
test_modeling_tf_whisper.py Add WhisperTokenizerFast (#21222) 2023-02-21 06:58:54 +01:00
test_modeling_whisper.py Add WhisperTokenizerFast (#21222) 2023-02-21 06:58:54 +01:00
test_processor_whisper.py [Whisper] Fix forced decoder ids (#20652) 2022-12-07 16:44:13 +00:00
test_tokenization_whisper.py Add WhisperTokenizerFast (#21222) 2023-02-21 06:58:54 +01:00