transformers/docs/source
Jonatan Kłosko deafc24388
Add WhisperTokenizerFast (#21222)
* Add WhisperTokenizerFast

* Fixup

* Up

* Up

* Improve tests

* Update src/transformers/models/whisper/tokenization_whisper_fast.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Keep stride in whisper pipelien test

* Remove unknown token special case

* Reduce vocabulary size in tests

* Fix vocab size assertion

* Sync copied changes from WhisperTokenizer

* Skip pipeline tests

* Update assertion

* Remove Whisper tokenizer dependency on sentencepiece

* Format

---------

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2023-02-21 06:58:54 +01:00
..
de Add EfficientNet (#21563) 2023-02-20 16:37:11 +03:00
en Add WhisperTokenizerFast (#21222) 2023-02-21 06:58:54 +01:00
es Add EfficientNet (#21563) 2023-02-20 16:37:11 +03:00
fr [i18n-fr] Translate quicktour page to French (#21589) 2023-02-13 13:05:31 -05:00
it Add EfficientNet (#21563) 2023-02-20 16:37:11 +03:00
ja Add EfficientNet (#21563) 2023-02-20 16:37:11 +03:00
ko Add EfficientNet (#21563) 2023-02-20 16:37:11 +03:00
pt Add EfficientNet (#21563) 2023-02-20 16:37:11 +03:00
zh translate index to zh(#20095) (#21351) 2023-01-30 16:50:57 -05:00
_config.py Adding evaluate to the list of libraries required in generated notebooks (#20850) 2022-12-21 14:04:08 +01:00