Nicolas Patry 2da28c4b41 [WIP] Hard error when ignoring tensors. (#27484) ... * [WIP] Hard error when ignoring tensors. * Better selection/error when saving a checkpoint. - Find all names we should normally drop (those are in the transformers config) - Find all disjoint tensors (for those we can safely trigger a copy to get rid of the sharing before saving) - Clone those disjoint tensors getting rid of the issue - Find all identical names (those should be declared in the config but we try to find them all anyway.) - For all identical names: - If they are in the config, just ignore them everything is fine - If they are not, warn about them. - For all remainder tensors which are shared yet neither identical NOR disjoint. raise a hard error. * Adding a failing test on `main` that passes here. * We don't need to keep the subfolder logic in this test. * Apply suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>		2024-02-05 09:17:24 +01:00
..
benchmark	[Test refactor 1/5] Per-folder tests reorganization (#15725)	2022-02-23 15:46:28 -05:00
bettertransformer	Fixed malapropism error (#26660)	2023-10-09 11:04:57 +02:00
deepspeed	DeepSpeed: hardcode torch.arange dtype on float usage to avoid incorrect initialization (#28760)	2024-01-31 14:39:07 +00:00
extended	Device agnostic trainer testing (#27131)	2023-10-30 18:16:40 +00:00
fixtures	[WIP] add SpeechT5 model (#18922)	2023-02-03 12:43:46 -05:00
fsdp	fix resuming from ckpt when using FSDP with FULL_STATE_DICT (#27891)	2023-12-16 19:41:43 +05:30
generation	Add tf_keras imports to prepare for Keras 3 (#28588)	2024-01-30 17:26:36 +00:00
models	Mark test_encoder_decoder_model_generate for vision_encoder_deocder as flaky (#28842)	2024-02-02 16:57:08 +00:00
optimization	Make schedulers picklable by making lr_lambda fns global (#21768)	2023-03-02 12:08:43 -05:00
peft_integration	[Peft] modules_to_save support for peft integration (#27466)	2023-11-14 10:32:57 +01:00
pipelines	[Whisper] Refactor forced_decoder_ids & prompt ids (#28687)	2024-01-31 14:02:07 +02:00
quantization	HfQuantizer class for quantization-related stuff in modeling_utils.py (#26610)	2024-01-30 02:48:25 +01:00
repo_utils	Allow # Ignore copy (#27328)	2023-12-07 10:00:08 +01:00
sagemaker	Add tf_keras imports to prepare for Keras 3 (#28588)	2024-01-30 17:26:36 +00:00
tokenization	[Styling] stylify using ruff (#27144)	2023-11-16 17:43:19 +01:00
tools	Add support for for loops in python interpreter (#24429)	2023-06-26 09:58:14 -04:00
trainer	Avoid root logger's level being changed (#28638)	2024-01-22 14:45:30 +01:00
utils	fix / skip (for now) some tests before switch to torch 2.2 (#28838)	2024-02-02 14:11:50 +01:00
__init__.py	GPU text generation: mMoved the encoded_prompt to correct device	2020-01-06 15:11:12 +01:00
test_backbone_common.py	Align backbone stage selection with out_indices & out_features (#27606)	2023-12-20 18:33:17 +00:00
test_cache_utils.py	Generate: SinkCache can handle iterative prompts (#27907)	2023-12-08 20:02:20 +00:00
test_configuration_common.py	[ PretrainedConfig] Improve messaging (#27438)	2023-11-15 14:10:39 +01:00
test_configuration_utils.py	Config: warning when saving generation kwargs in the model config (#28514)	2024-01-16 18:31:01 +00:00
test_feature_extraction_common.py	Split common test from core tests (#24284)	2023-06-15 07:30:24 -04:00
test_feature_extraction_utils.py	Remove-auth-token (#27060)	2023-11-13 14:20:54 +01:00
test_image_processing_common.py	Fix a couple of typos and add an illustrative test (#26941)	2023-12-11 15:51:51 +00:00
test_image_processing_utils.py	Remove-auth-token (#27060)	2023-11-13 14:20:54 +01:00
test_image_transforms.py	Normalize floating point cast (#27249)	2023-11-10 15:35:27 +00:00
test_modeling_common.py	Fix symbolic_trace with kv cache (#28724)	2024-02-01 09:45:02 +01:00
test_modeling_flax_common.py	[Flax] Update no init test for Flax v0.7.1 (#28735)	2024-01-26 18:20:39 +00:00
test_modeling_flax_utils.py	Enable safetensors conversion from PyTorch to other frameworks without the torch requirement (#27599)	2024-01-23 10:28:23 +01:00
test_modeling_tf_common.py	Add tf_keras imports to prepare for Keras 3 (#28588)	2024-01-30 17:26:36 +00:00
test_modeling_tf_utils.py	Add tf_keras imports to prepare for Keras 3 (#28588)	2024-01-30 17:26:36 +00:00
test_modeling_utils.py	[WIP] Hard error when ignoring tensors. (#27484)	2024-02-05 09:17:24 +01:00
test_pipeline_mixin.py	Shorten the conversation tests for speed + fixing position overflows (#26960)	2023-10-31 14:20:04 +00:00
test_processing_common.py	Don't save processor_config.json if a processor has no extra attribute (#28584)	2024-01-19 09:59:14 +00:00
test_sequence_feature_extraction_common.py	Fix typo (#25966)	2023-09-05 10:12:25 +02:00
test_tokenization_common.py	[ TokenizationUtils] Fix add_special_tokens when the token is already there (#28520)	2024-01-16 16:36:29 +01:00
test_tokenization_utils.py	Remove-auth-token (#27060)	2023-11-13 14:20:54 +01:00