transformers/tests
eustlb 54aae121eb
[Whisper] Fix whisper tokenizer (#34537)
* handle single timestamp ending

* include last timestamp token

* handle single timestamp ending

* avoid floating points arithm limitations

* ensure float64 operations

* new test

* make fixup

* make copies

* handle edge case double tokens ending with different tokens

* handle single timestamp ending

* make fixup

* handle conditioning on prev segments

* fix

* Update src/transformers/models/whisper/generation_whisper.py

Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com>

* [run-slow] whisper

* don't call item() to avoid unnecessary sync

* fix

---------

Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com>
Co-authored-by: Eustache Le Bihan <eustlb@users.noreply.huggingface.co>
2024-12-05 13:46:29 +01:00
..
agents Add token cost + runtime monitoring to Agent and HfEngine children (#34548) 2024-12-03 13:14:52 +01:00
benchmark
bettertransformer Fixed malapropism error (#26660) 2023-10-09 11:04:57 +02:00
deepspeed Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00
extended [tests] skip tests for xpu (#33553) 2024-09-19 19:28:04 +01:00
fixtures Implementation of SuperPoint and AutoModelForKeypointDetection (#28966) 2024-03-19 14:43:02 +00:00
fsdp FSDP grad accum fix (#34645) 2024-11-15 22:28:06 +01:00
generation Offloaded cache: fix generate (#34921) 2024-11-28 15:05:56 +01:00
models [Whisper] Fix whisper tokenizer (#34537) 2024-12-05 13:46:29 +01:00
optimization fix: Fixed the 1st argument name in classmethods (#31907) 2024-07-11 12:11:50 +01:00
peft_integration [PEFT] Set eval mode when loading PEFT adapter (#34509) 2024-11-28 13:56:25 +01:00
pipelines allow unused input parameters passthrough when chunking in asr pipelines (#33889) 2024-11-25 11:36:44 +01:00
quantization Skipping aqlm non working inference tests till fix merged (#34865) 2024-11-26 11:09:30 +01:00
repo_utils Refactor CI: more explicit (#30674) 2024-08-30 18:17:25 +02:00
sagemaker Trainer - deprecate tokenizer for processing_class (#32385) 2024-10-02 14:08:46 +01:00
tokenization VLM: special multimodal Tokenizer (#34461) 2024-11-04 16:37:51 +01:00
tp Simplify Tensor Parallel implementation with PyTorch TP (#34184) 2024-11-18 19:51:49 +01:00
trainer Remove FSDP wrapping from sub-models. (#34452) 2024-11-15 23:00:03 +01:00
utils Fix: take into account meta device (#34134) 2024-11-20 11:32:07 +01:00
__init__.py
test_backbone_common.py Align backbone stage selection with out_indices & out_features (#27606) 2023-12-20 18:33:17 +00:00
test_configuration_common.py Load sub-configs from composite configs (#34410) 2024-11-05 11:34:01 +01:00
test_feature_extraction_common.py
test_image_processing_common.py Add DetrImageProcessorFast (#34063) 2024-10-21 09:05:05 -04:00
test_image_transforms.py fix: center_crop occasionally outputs off-by-one dimension matrix (#30934) 2024-05-21 13:56:52 +01:00
test_modeling_common.py Fix test_eager_matches_sdpa_inference for XPU backend (#34889) 2024-12-02 16:21:04 +01:00
test_modeling_flax_common.py add sdpa to ViT [follow up of #29325] (#30555) 2024-05-16 10:56:11 +01:00
test_modeling_tf_common.py [TF] Fix Tensorflow XLA Generation on limited seq_len models (#33903) 2024-10-05 16:20:50 +02:00
test_pipeline_mixin.py Add image text to text pipeline (#34170) 2024-10-31 15:48:11 -04:00
test_processing_common.py Separate chat templates into a single file (#33957) 2024-11-26 14:18:04 +00:00
test_sequence_feature_extraction_common.py
test_tokenization_common.py Separate chat templates into a single file (#33957) 2024-11-26 14:18:04 +00:00