CalOmnie
547911e727
Uses Collection in transformers.image_transforms.normalize ( #36301 )
...
* Uses Collection instead of Sequence in transformers.image_transforms.normalize
* Uses collections.abc.Collection in lieu of deprecated typing one
2025-02-21 18:38:41 +01:00
Fanli Lin
7c5bd24ffa
[tests] make quanto tests device-agnostic ( #36328 )
...
* make device-agnostic
* name change
2025-02-21 14:20:40 +01:00
Joao Gante
678885bbbd
[CI] Check test if the GenerationTesterMixin inheritance is correct 🐛 🔫 ( #36180 )
2025-02-21 10:18:20 +00:00
Pavel Iakubovskii
a957b7911a
Add SigLIP 2 ( #36323 )
...
* Docs
* Inits
* Auto classes
* Add siglip base
* Add base tests
* Fix Siglip V1 for fix res version
* Add image processor
* Update conversion
* Experimenting with vectorized embeddings
* Fixup
* Add modular Siglip2Processor
* Add modular configuration
* Rename num patches
* Correct image and text features merging
* Working conversion script
* Refactoring conversion script
* Remove unused code in conversion script
* Shorten dict a bit
* Refactoring conversion
* Done conversion refactoring
* Fixup
* Modular siglip2
* Make model exportable and compilable without graph breaks
* Remove position_ids from image_processor
* REmove position ids from modeling file
* Update modular
* Type hint
* Fixup
* Set defaults to processor
* Add integration test
* Revert spatial shapes back to tensor
* Change order
* Fix most of the tests
* Fix docstring
* Remove interpolate_pos_encoding arg (not needed)
* Update docs
* Standardize processing
* Fix attention_mask in vision head
* Siglip v1: remove double transpose in FA2
* Update modular file
* Update FA2 test
* Update expected logits
* Fix interpolation for siglip2 image processor
* Skip init test
* Skip dispatch on flash test
* Fix modeling tests
* Fixup
* Add dummy objects
* Fix some docstrings
* Add siglip2 in index.md
* Fix consistency
* Add docs
* Remove size and data format
* Add image processor tests
* Fix
* Add fast image processor
* Fix style
* Fix
* Docs
* Set lowercase for tokenizer
* Adjust head size for Siglip v1
* Update siglip2 for consistency with siglip1
* Update siglip2 conversion
* Update pipeline
* Update checkpoints in tests
* Update checkpoint name
* Fix pooling for image classification model
* Fix FA2 test
* Update processor
* Fix check repo
* Update docs
* Fix typos
* Fix docstring for fast image processor
* Add siglip2 to FA2 docs
* Fix fast ip tests
* Fix constitency
* Fix tokenizer class for siglip v1
* Fix missing header
* Refactor scaling for clip, siglip, siglip2
* Remove unused imports
* Make fast IP default for siglip2
* Update docs
* Update checkpoints
* Update modular
* Update paper link
* Fixup
* Fix name in toctree
* Fix test
2025-02-21 09:04:19 +00:00
Raushan Turganbay
14552cbd7c
VLMs: even more clean-up ( #36249 )
...
* squash
* style
2025-02-21 09:46:31 +01:00
Cyan
e18f233f6c
Fix default attention mask of generate in MoshiForConditionalGeneration ( #36171 )
2025-02-20 19:53:27 +00:00
Joao Gante
27d1707586
[smolvlm] make CI green ( #36306 )
...
* add smolvlm to toctree
* add requirements
* dev-ci
* no docker changes
* dev-ci
* update torch-light.dockerfile
* derp
* dev-ci
2025-02-20 18:56:11 +01:00
Nosimus
effaef334b
fix: prevent second save in the end of training if last step was saved already ( #36219 )
...
* fix: prevent second save in the end of training
* fix: prevent second save in the end of training
* test: added test for no duplicate save on epoch save strategy
* fix: removed TrainerControl
* chore: style formatting
---------
Co-authored-by: JaktensTid <jaktenstid1@gmail.com>
2025-02-20 17:38:52 +01:00
12v
5412ff1a13
Fix typo in Pixtral example ( #36302 )
...
Fix typo
2025-02-20 14:13:48 +00:00
Orr Zohar
4397dfcb71
SmolVLM2 ( #36126 )
...
* smolvlm init
* updates
* fixing bugs
* minimal run, no checks
* minimal run, no checks
* passing first check + adding url support
* updating video dataloading logic
* fixing image logic
* trying modular, but fails
* modular is working, changing processor to match PR comments and general transformers logic
* fixing kwargs
* offloading video loading logic to image_util
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* fixing circleci code formatting errors
* update
* add idefics3-based tests
* add keyword to all
* add PreTrainedModel
* updateing video loading logic
* working inference
* updates for PR comments
* updates for PR comments
* moving SmolVLMPretrainedModel higher to fix import error
* CI test pass
* CI test pass
* removing lambda
* CI test pass
* CI test pass
* CI test pass
* CI test pass
* CI test pass
* CI test pass
* processor tests
* add example in docs
* typo
* fix copies
* skip compile tests - sdpa for VisionTransformer
* fix init
* raise import error for num2words
* update doc for FA2
* more doc fix
* CI
* updates for PR comments
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* fixing processor -- tokenizer not defined properly, (gpt2 tokenizer), and does not have the attributes of fake image token, etc
* adding smolvlm to VQA models
* removing vqa auto class
* Update src/transformers/models/smolvlm/processing_smolvlm.py
Co-authored-by: Joshua Lochner <admin@xenova.com>
* removing smolvlmvisiontransformer from index.md
* my bad, video processing had typos
* fixing docs
* renaming params in SmolVLMModel.inputs_merger
* removing un-needed dtype/device in model forward
* ruff for CI
* update docs
* Update docs/source/en/model_doc/smolvlm.md
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* return cache position
* return cache position
* return cache also in modular
* needed to run modular again
* fix training tests
* push vectorized inputs merger
* format
* format
* reduce number of mappings
* addressing PR comments
* happy CI, happy me :)
* skip non-nested images
* adjust integration test for smaller GPUs
* format
* fix kwargs in chat template apply
* skip this for now
---------
Co-authored-by: raushan <raushan@huggingface.co>
Co-authored-by: Pablo <pablo.montalvo.leroux@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Joshua Lochner <admin@xenova.com>
2025-02-20 15:00:26 +01:00
Yih-Dar
f2ab182dca
Ignore conversion files in test fetcher ( #36251 )
...
fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2025-02-20 13:32:02 +01:00
Yih-Dar
e8531a0e33
Fix broken CI on release branch due to missing conversion files ( #36275 )
...
* fix
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2025-02-20 13:22:10 +01:00
Ilyas Moutawwakil
5e2183f344
Make cache traceable ( #35873 )
...
simply make cache traceable
2025-02-20 09:59:25 +01:00
Marc Sun
31bb662db1
Fix callback handler reference ( #36250 )
...
* fix reference
* style
2025-02-19 18:17:33 +01:00
hyjbrave
78d6484675
docs: Update README_zh-hans.md ( #36269 )
...
Update README_zh-hans.md
docs: Fix awkward sentence in README
2025-02-19 09:04:46 -08:00
Mohamed Mekkouri
e5cea20743
Add Example for Custom quantization ( #36286 )
...
* add example
* rename
2025-02-19 17:09:23 +01:00
Joao Gante
e3d99ec2f5
[tests] make test_from_pretrained_low_cpu_mem_usage_equal less flaky ( #36255 )
...
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2025-02-19 15:14:02 +00:00
Joao Gante
99adc74462
[tests] remove flax-pt equivalence and cross tests ( #36283 )
2025-02-19 15:13:27 +00:00
Joao Gante
fa8cdccd91
[tests] deflake dither test ( #36284 )
2025-02-19 15:13:10 +00:00
Cyril Vallez
60226c6ff3
TP initialization module-by-module ( #35996 )
...
* module-by-module loading!
* Update modeling_utils.py
* dtyle and comments
* Update modeling_utils.py
* Update modeling_utils.py
* Update test
* Update modeling_utils.py
* Update modeling_utils.py
* Update test_tp.py
* Update test_tp.py
* Update modeling_utils.py
* re-trigger CIs
* re-trigger CIs
2025-02-19 14:04:57 +01:00
Joao Gante
0863eef248
[tests] remove pt_tf equivalence tests ( #36253 )
2025-02-19 11:55:11 +00:00
Karel Vesely
1a81d774b1
Add dithering to the Speech2TextFeatureExtractor API. ( #34638 )
...
* Add dithering to the `Speech2TextFeatureExtractor` API.
- in kaldi : 4a8b7f6732/src/feat/feature-window.cc (L145)
2025-02-19 11:50:02 +01:00
Yoni Gozlan
9f51dc2535
Add support for post-processing kwargs in image-text-to-text pipeline ( #35374 )
...
* fix error and improve pipeline
* add processing_kwargs to apply_chat_template
* change default post_process kwarg to args
* Fix slow tests
* fix copies
2025-02-18 17:43:36 -05:00
Yoni Gozlan
9b479a245b
Uniformize LlavaNextVideoProcessor kwargs ( #35613 )
...
* Uniformize processor kwargs and add tests
* add videos_kwargs tests
* fix copies
* fix llava_next_video chat template tests
* remove unnecessary default kwargs
2025-02-18 14:13:51 -05:00
Ardalan
8ee50537fe
Qwen2VL fix cos,sin dtypes to float when used with deepspeed ( #36188 )
...
* fix dtype of cos,sin when used with deepspeed
* move sin,cos casting withing flash attention functions
* fix cos,sin float casting in modular
---------
Co-authored-by: ardalan.mehrani <ardalan.mehrani@ardalanmehranis-MacBook-Pro.local>
Co-authored-by: ardalan.mehrani <ardalan.mehrani@bytedance.com>
2025-02-18 19:18:29 +01:00
Parteek
8eaae6bee9
Added Support for Custom Quantization ( #35915 )
...
* Added Support for Custom Quantization
* Update code
* code reformatted
* Updated Changes
* Updated Changes
---------
Co-authored-by: Mohamed Mekkouri <93391238+MekkCyber@users.noreply.github.com>
2025-02-18 16:14:19 +01:00
ivarflakstad
07182b2e10
GitModelIntegrationTest - flatten the expected slice tensor ( #36260 )
...
Flatten the expected slice tensor
2025-02-18 16:04:19 +01:00
Damiano Amatruda
4d2de5f63c
Fix XGLM loss computation (PyTorch and TensorFlow) ( #35878 )
...
* Fix XGLM loss computation (PyTorch and TensorFlow)
* Update expected output string in XGLM sample test
This updates the expected output string of test_xglm_sample for torch
2.0 to the correct one and removes the one for torch 1.13.1 + cu116
(transformers moved to torch 2.0 with PR #35358 ).
* Update expected output IDs in XGLM generation test
2025-02-18 15:37:48 +01:00
Mehant Kammakomati
c3ba53303b
feat: add support for tensor parallel training workflow with accelerate ( #34194 )
...
* feat: add support for tensor parallel flow using accelerate
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com>
* fix: add tp degree to env variable
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com>
* fix: add version check for accelerate to allow TP
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com>
* docs: tensor parallelism
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com>
* nit: rename plugin name
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com>
* fix: guard accelerate version before allow tp
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com>
* docs: add more docs and updates related to TP
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com>
---------
Signed-off-by: Mehant Kammakomati <mehant.kammakomati2@ibm.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
2025-02-18 14:05:46 +01:00
Raushan Turganbay
e6cc410d5b
Remove flakiness in VLMs ( #36242 )
...
* fix
* nit
* no logits processor needed
* two more tests on assisted decoding
2025-02-18 11:41:07 +01:00
andrewor14
fdcfdbfd22
Fix TorchAoConfig not JSON serializable ( #36206 )
...
**Summary:** TorchAoConfig optionally contains a
`torchao.dtypes.Layout` object which is a dataclass and not
JSON serializable, and so the following fails:
```
import json
from torchao.dtypes import TensorCoreTiledLayout
from transformers import TorchAoConfig
config = TorchAoConfig("int4_weight_only", layout=TensorCoreTiledLayout())
config.to_json_string()
json.dumps(config.to_dict())
```
This also causes `quantized_model.save_pretrained(...)` to
fail because the first step of this call is to JSON serialize
the config. Fixes https://github.com/pytorch/ao/issues/1704 .
**Test Plan:**
python tests/quantization/torchao_integration/test_torchao.py -k test_json_serializable
Co-authored-by: Mohamed Mekkouri <93391238+MekkCyber@users.noreply.github.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
2025-02-18 11:05:42 +01:00
Yih-Dar
626666c444
Au revoir flaky test_fast_is_faster_than_slow ( #36240 )
...
* fix
* fix
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2025-02-17 18:30:07 +01:00
Joao Gante
429f1a682d
[tests] remove test_export_to_onnx ( #36241 )
2025-02-17 16:52:44 +00:00
Marc Sun
dae8708c36
Add compressed tensor in quant dockerfile ( #36239 )
...
add compressed_tensors in the dockerfile
2025-02-17 17:48:57 +01:00
dependabot[bot]
3e970dbbf1
Bump transformers from 4.38.0 to 4.48.0 in /examples/research_projects/codeparrot/examples ( #36237 )
...
Bump transformers in /examples/research_projects/codeparrot/examples
Bumps [transformers](https://github.com/huggingface/transformers ) from 4.38.0 to 4.48.0.
- [Release notes](https://github.com/huggingface/transformers/releases )
- [Commits](https://github.com/huggingface/transformers/compare/v4.38.0...v4.48.0 )
---
updated-dependencies:
- dependency-name: transformers
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-02-17 16:28:43 +00:00
eustlb
77aa9fc076
[generate] Fix encoder decoder models attention mask ( #36018 )
2025-02-17 15:42:28 +00:00
Joao Gante
55493f1390
[tests] remove tf/flax tests in /generation ( #36235 )
2025-02-17 14:59:22 +00:00
Arthur Zucker
c877c9fa5b
v4.45.0-dev0
2025-02-17 15:21:20 +01:00
ivarflakstad
7ec35bc3bd
Add missing atol to torch.testing.assert_close where rtol is specified ( #36234 )
2025-02-17 14:57:50 +01:00
Joao Gante
dad513e0c2
[generate] remove cache v4.47 deprecations ( #36212 )
2025-02-17 13:55:03 +00:00
ivarflakstad
936aeb70ab
AMD DeepSpeed image additional HIP dependencies ( #36195 )
...
* Add hipsolver and hipblastlt as dependencies
* Upgrade torch libs with rocm6.2.4 index
2025-02-17 11:50:49 +01:00
Yih-Dar
23d6095e8f
Fix LlavaForConditionalGenerationModelTest::test_config after #36077 ( #36230 )
...
fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2025-02-17 11:49:07 +01:00
Fanli Lin
fae0f3dde8
[tests] fix EsmModelIntegrationTest::test_inference_bitsandbytes ( #36225 )
...
fix failed test
2025-02-17 11:10:33 +01:00
Yih-Dar
dd16acb8a3
set test_torchscript = False for Blip2 testing ( #35972 )
...
* just skip
* fix
* fix
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2025-02-14 17:43:32 +01:00
Yih-Dar
0a9923a609
Use args.num_workers in check_modular_conversion.py ( #36200 )
...
fix
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2025-02-14 17:31:03 +01:00
Mayank Mishra
a570e2ba87
add shared experts for upcoming Granite 4.0 language models ( #35894 )
...
* Modular GraniteMoE with shared Experts.
Signed-off-by: Shawn Tan <shawntan@ibm.com>
* Modified
* Import order.
* Modified for style
* Fix space.
* Test
* Remove extra granitemoe file.
* New converted file and tests
* Modified __init__ files.
* Formatting.
* Dummy PT objects
* register granitemoe shared model
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
* fix linting of a file
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
* fix import in modeling file
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
* update generated modeling file
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
* add documentation
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
* update docstrings
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
* update generated modeling file
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
* fix docstrings in config class
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
* merge main
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
---------
Signed-off-by: Shawn Tan <shawntan@ibm.com>
Signed-off-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
Co-authored-by: Shawn Tan <shawntan@ibm.com>
Co-authored-by: Shawn Tan <shawn@wtf.sg>
Co-authored-by: Sukriti-Sharma4 <sukriti.sharma4@ibm.com>
Co-authored-by: Sukriti Sharma <Ssukriti@users.noreply.github.com>
2025-02-14 16:55:28 +01:00
ivarflakstad
7ae7e87a09
Add @require_bitsandbytes to Aria test_batched_generation ( #36192 )
2025-02-14 15:48:47 +01:00
Kyle Sayers
bcfc9d795e
[Bugfix] Fix reloading of pixtral/llava configs ( #36077 )
...
* add is_composition flag to LlavaConfig
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
* WIP: pixtral text config
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
* fix style
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
* add test
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
* use is_composition for pixtral
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
* Revert "use is_composition for pixtral"
This reverts commit a53d5f9fc53ab1c99404
2025-02-14 15:27:05 +01:00
Raushan Turganbay
0c78ef6cd3
🔴 VLM: compile compatibility ( #35724 )
...
* llavas
* add mroe models
* fix `compile_forward` test for all models
* fix copies
* make style
* also doesn't support cache class
* fix some tests
* not copied from
* ci green?
* fix tests
* fix copies
* fix tests
* check with `numel` and remove `item`
* fix copies
* fix copies
* Update src/transformers/models/cohere2/modeling_cohere2.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* opt remove cross attn
* gemma2
* fixup
* fixup
* fix newly added test
* maybe fixed?
* green please?
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2025-02-14 15:23:49 +01:00
David LaPalomento
b45cf0e90a
Guard against unset resolved_archive_file ( #35628 )
...
* archive_file may not be specified
When loading a pre-trained model from a gguf file, resolved_archive_file may not be set. Guard against that case in the safetensors availability check.
* Remap partial disk offload to cpu for GGUF files
GGUF files don't support disk offload so attempt to remap them to the CPU when device_map is auto. If device_map is anything else but None, raise a NotImplementedError.
* Don't remap auto device_map and raise RuntimeError
If device_map=auto and modules are selected for disk offload, don't attempt to map them to any other device. Raise a runtime error when a GGUF model is configured to map any modules to disk.
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
2025-02-14 14:44:31 +01:00
