mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-31 02:02:21 +06:00
06d488061f
* [Whisper Tokenizer] Make more user-friendly * use property * make indexing rigorous * small clean-up * tests * skip seq2seq tests * remove multilingual arg * reorder args * collapse to one function Co-authored-by: ArthurZucker <arthur@huggingface.co> * option to override attributes Co-authored-by: ArthurZucker <arthur@huggingface.co> * add to docs * Apply suggestions from code review Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * make comment more clear Co-authored-by: sgugger <sylvain@huggingface.co> * don't add special tokens in get_decoder_prompt_ids * add test for set_prefix_tokens Co-authored-by: ArthurZucker <arthur@huggingface.co> Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> Co-authored-by: sgugger <sylvain@huggingface.co> |
||
|---|---|---|
| .. | ||
| benchmark | ||
| deepspeed | ||
| extended | ||
| fixtures | ||
| generation | ||
| mixed_int8 | ||
| models | ||
| onnx | ||
| optimization | ||
| pipelines | ||
| repo_utils | ||
| sagemaker | ||
| tokenization | ||
| trainer | ||
| utils | ||
| __init__.py | ||
| test_configuration_common.py | ||
| test_feature_extraction_common.py | ||
| test_image_transforms.py | ||
| test_modeling_common.py | ||
| test_modeling_flax_common.py | ||
| test_modeling_tf_common.py | ||
| test_sequence_feature_extraction_common.py | ||
| test_tokenization_common.py | ||