transformers/utils
Orr Zohar 4397dfcb71
SmolVLM2 (#36126)
* smolvlm init

* updates

* fixing bugs

* minimal run, no checks

* minimal run, no checks

* passing first check + adding url support

* updating video dataloading logic

* fixing image logic

* trying modular, but fails

* modular is working, changing processor to match PR comments and general transformers logic

* fixing kwargs

* offloading video loading logic to  image_util

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* fixing circleci code formatting errors

* update

* add idefics3-based tests

* add keyword to all

* add PreTrainedModel

* updateing video loading logic

* working inference

* updates for PR comments

* updates for PR comments

* moving SmolVLMPretrainedModel higher to fix import error

* CI test pass

* CI test pass

* removing lambda

* CI test pass

* CI test pass

* CI test pass

* CI test pass

* CI test pass

* CI test pass

* processor tests

* add example in docs

* typo

* fix copies

* skip compile tests - sdpa for VisionTransformer

* fix init

* raise import error for num2words

* update doc for FA2

* more doc fix

* CI

* updates for PR comments

* Update docs/source/en/model_doc/smolvlm.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update docs/source/en/model_doc/smolvlm.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update docs/source/en/model_doc/smolvlm.md

Co-authored-by: Joshua Lochner <admin@xenova.com>

* Update docs/source/en/model_doc/smolvlm.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* Update docs/source/en/model_doc/smolvlm.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* fixing processor -- tokenizer not defined properly, (gpt2 tokenizer), and does not have the attributes of fake image token, etc

* adding smolvlm to VQA models

* removing vqa auto class

* Update src/transformers/models/smolvlm/processing_smolvlm.py

Co-authored-by: Joshua Lochner <admin@xenova.com>

* removing smolvlmvisiontransformer from index.md

* my bad, video processing had typos

* fixing docs

* renaming params in SmolVLMModel.inputs_merger

* removing un-needed dtype/device in model forward

* ruff for CI

* update docs

* Update docs/source/en/model_doc/smolvlm.md

Co-authored-by: Pedro Cuenca <pedro@huggingface.co>

* return cache position

* return cache position

* return cache also in modular

* needed to run modular again

* fix training tests

* push vectorized inputs merger

* format

* format

* reduce number of mappings

* addressing PR comments

* happy CI, happy me :)

* skip non-nested images

* adjust integration test for smaller GPUs

* format

* fix kwargs in chat template apply

* skip this for now

---------

Co-authored-by: raushan <raushan@huggingface.co>
Co-authored-by: Pablo <pablo.montalvo.leroux@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Joshua Lochner <admin@xenova.com>
2025-02-20 15:00:26 +01:00
..
test_module
tf_ops
add_pipeline_model_mapping_to_test.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
check_bad_commit.py Fix utils/check_bad_commit.py (for auto ping in CI) (#34943) 2024-11-28 15:34:38 +01:00
check_build.py Fix import of FalconMambaForCausalLM (#33381) 2024-09-10 09:14:54 +02:00
check_config_attributes.py Adding RT-DETRv2 for object detection (#34773) 2025-02-06 19:28:45 +00:00
check_config_docstrings.py Add TimmWrapper (#34564) 2024-12-11 12:40:30 +00:00
check_copies.py [Modular] skip modular checks based on diff (#36130) 2025-02-13 12:53:21 +00:00
check_doc_toc.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
check_docstrings.py Refactoring of ImageProcessorFast (#35069) 2025-02-04 17:52:31 -05:00
check_doctest_list.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
check_dummies.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
check_inits.py Fix import of FalconMambaForCausalLM (#33381) 2024-09-10 09:14:54 +02:00
check_model_tester.py
check_modular_conversion.py Use args.num_workers in check_modular_conversion.py (#36200) 2025-02-14 17:31:03 +01:00
check_repo.py SmolVLM2 (#36126) 2025-02-20 15:00:26 +01:00
check_self_hosted_runner.py
check_support_list.py [RoBERTa-based] Add support for sdpa (#30510) 2024-08-28 10:26:00 +02:00
check_table.py SmolVLM2 (#36126) 2025-02-20 15:00:26 +01:00
check_tf_ops.py
create_dependency_mapping.py [Modular] skip modular checks based on diff (#36130) 2025-02-13 12:53:21 +00:00
create_dummy_models.py CI: fix efficientnet pipeline timeout and prevent future similar issues due to large image size (#33123) 2024-08-27 11:58:27 +01:00
custom_init_isort.py Import structure & first three model refactors (#31329) 2024-09-10 11:10:53 +02:00
deprecate_models.py Remove copied froms for deprecated models (#31153) 2024-06-03 09:42:53 +01:00
download_glue_data.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
extract_warnings.py
get_ci_error_statistics.py
get_github_job_time.py
get_modified_files.py
get_previous_daily_ci.py Ping team members for new failed tests in daily CI (#34171) 2024-10-17 16:11:52 +02:00
get_test_info.py CI: fix efficientnet pipeline timeout and prevent future similar issues due to large image size (#33123) 2024-08-27 11:58:27 +01:00
important_models.txt
models_to_deprecate.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
modular_model_converter.py [Modular] skip modular checks based on diff (#36130) 2025-02-13 12:53:21 +00:00
not_doctested.txt [commands] remove deprecated/inoperational commands (#35718) 2025-02-12 12:23:58 +00:00
notification_service_doc_tests.py
notification_service_quantization.py Revive Nightly/Past CI (#31159) 2024-06-20 18:57:24 +02:00
notification_service.py Remove old benchmark code (#35730) 2025-01-21 17:56:43 +00:00
past_ci_versions.py
patch_helper.py [Patch helper] update to not have to checkout main (#34006) 2024-10-09 09:21:46 +02:00
pr_slow_ci_models.py Commont bot CI for other jobs (generation / quantization) (#35341) 2025-02-04 14:42:51 +01:00
print_env.py
process_bad_commit_report.py Tiny update after #34383 (#34404) 2024-10-28 12:01:05 +01:00
process_circleci_workflow_test_reports.py Aggeregate test summary files in CircleCI workflow runs (#34989) 2024-12-16 11:06:17 +01:00
process_test_artifacts.py fix the parallel number of CI nodes when it is smaller than number of tests (#33276) 2024-09-03 16:53:21 +02:00
release.py 🚨🚨🚨 Delete conversion scripts when making release wheels (#35296) 2024-12-17 14:18:42 +00:00
set_cuda_devices_for_ci.py Fix Cohere CI (#31263) 2024-06-10 15:16:58 +02:00
slow_documentation_tests.txt
sort_auto_mappings.py update ruff version (#30932) 2024-05-22 06:40:15 +02:00
split_doctest_jobs.py
split_model_tests.py
tests_fetcher.py Ignore conversion files in test fetcher (#36251) 2025-02-20 13:32:02 +01:00
update_metadata.py Add ColPali to 🤗 transformers (#33736) 2024-12-17 11:26:43 +01:00
update_tiny_models.py Mention model_info.id instead of model_info.modelId (#32106) 2024-07-22 14:14:47 +01:00