Tom Aarsen 32e0db8a69 [tokenizers] Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer (#35593) ... * Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer in PreTrainedTokenizerFast, rather than relying on subclasses to take care of this. * Simplify setting self.add_prefix_space, ensure pre_tok exists * Wrap in try-except to catch 'Custom PreTokenizer cannot be serialized' 862d1a346a/bindings/python/src/pre_tokenizers.rs (L672) produces the Exception. They're triggered by the roformer tests, as the RoFormerTokenizerFast uses a custom PreTokenizer. * Propagate add_prefix_space in T5TokenizerFast to superclass		2025-01-09 17:46:50 +01:00
..
agents	Change is_soundfile_availble to is_soundfile_available (#35030)	2025-01-03 14:37:42 +01:00
benchmark	[Test refactor 1/5] Per-folder tests reorganization (#15725)	2022-02-23 15:46:28 -05:00
bettertransformer	Fixed malapropism error (#26660)	2023-10-09 11:04:57 +02:00
deepspeed	Trainer - deprecate tokenizer for processing_class (#32385)	2024-10-02 14:08:46 +01:00
extended	[tests] skip tests for xpu (#33553)	2024-09-19 19:28:04 +01:00
fixtures	Implementation of SuperPoint and AutoModelForKeypointDetection (#28966)	2024-03-19 14:43:02 +00:00
fsdp	FSDP grad accum fix (#34645)	2024-11-15 22:28:06 +01:00
generation	fix: Qwen2-VL generate with inputs_embeds (#35466)	2025-01-08 16:36:03 +01:00
models	Fix flaky SwitchTransformersModelTest::test_training_gradient (#35587)	2025-01-09 15:36:22 +01:00
optimization	fix: Fixed the 1st argument name in classmethods (#31907)	2024-07-11 12:11:50 +01:00
peft_integration	added logic for deleting adapters once loaded (#34650)	2025-01-06 18:36:40 +00:00
pipelines	Pipeline: simple API for assisted generation (#34504)	2025-01-08 17:08:02 +00:00
quantization	Add Gemma2 GGUF support (#34002)	2025-01-03 14:50:07 +01:00
repo_utils	Fix modular edge case + modular sorting order (#35562)	2025-01-09 17:17:52 +01:00
sagemaker	Trainer - deprecate tokenizer for processing_class (#32385)	2024-10-02 14:08:46 +01:00
tokenization	tokenizer train from iterator without pre_tokenizers (#35396)	2025-01-09 15:34:43 +01:00
tp	Simplify Tensor Parallel implementation with PyTorch TP (#34184)	2024-11-18 19:51:49 +01:00
trainer	Fix all output_dir in test_trainer.py to use tmp_dir (#35266)	2025-01-08 19:44:39 +01:00
utils	More model refactoring! (#35359)	2025-01-09 11:09:09 +01:00
__init__.py	GPU text generation: mMoved the encoded_prompt to correct device	2020-01-06 15:11:12 +01:00
test_backbone_common.py	Align backbone stage selection with out_indices & out_features (#27606)	2023-12-20 18:33:17 +00:00
test_configuration_common.py	Load sub-configs from composite configs (#34410)	2024-11-05 11:34:01 +01:00
test_feature_extraction_common.py	Split common test from core tests (#24284)	2023-06-15 07:30:24 -04:00
test_image_processing_common.py	Fix Qwen2VL processor to handle odd number of frames (#35431)	2025-01-08 13:49:00 +01:00
test_image_transforms.py	fix: center_crop occasionally outputs off-by-one dimension matrix (#30934)	2024-05-21 13:56:52 +01:00
test_modeling_common.py	Fix flaky SwitchTransformersModelTest::test_training_gradient (#35587)	2025-01-09 15:36:22 +01:00
test_modeling_flax_common.py	🚨All attention refactor🚨 (#35235)	2024-12-18 16:53:39 +01:00
test_modeling_tf_common.py	🚨All attention refactor🚨 (#35235)	2024-12-18 16:53:39 +01:00
test_pipeline_mixin.py	Add image text to text pipeline (#34170)	2024-10-31 15:48:11 -04:00
test_processing_common.py	VLMs: major clean up 🧼 (#34502)	2025-01-08 10:35:23 +01:00
test_sequence_feature_extraction_common.py	Fix typo (#25966)	2023-09-05 10:12:25 +02:00
test_tokenization_common.py	[tokenizers] Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer (#35593)	2025-01-09 17:46:50 +01:00