* add unit tests for splinter_tokenizer
* add unit test for splinter tokenizer, pass in the question_token to be saved on save_pretrained called
* remove unused import
* remove vocab_splinter.txt, add Copied from, use fmt:on and fmt:off to prevent autoformatting on long lines
* remove all the spaces
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Use all state dict keys when checking if root module is initialized.
* Apply style corrections
* Add comment explaining change.
* Change comment phrasing.
* try fixing push-ci
* move to new runners
* move benchmark.yml to new runners
* move doctest_job.yml to new runners
* move doctests.yml to new runners
* move push-important-models.yml to new runners
* move self-pr-slow-ci.yml to new runners
* fix typo
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* fix working directory
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* fix working directory
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* improve code
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
---------
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* Update an keyerror on _save_check_point prevent confusion of missing metric keys
* Update grammar error and case sensitive.
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* adding update KeyError on _evaluate function to align with _save_checkpoint function
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* When we set self.dt_proj.bias = None, it removes the bias parameter from the model. When we later tried to assign a tensor to self.dt_proj.bias, it caused a TypeError because PyTorch expects a Parameter object.
* When we set self.dt_proj.bias = None, it removes the bias parameter from the model. When we later tried to assign a tensor to self.dt_proj.bias, it caused a TypeError because PyTorch expects a Parameter object.
* When we set self.dt_proj.bias = None, it removes the bias parameter from the model. When we later tried to assign a tensor to self.dt_proj.bias, it caused a TypeError because PyTorch expects a Parameter object.
* Trainer - deprecate tokenizer for processing_class
* Extend chage across Seq2Seq trainer and docs
* Add tests
* Update to FutureWarning and add deprecation version
* add support for custom inputs and batched inputs in ProcessorTesterMixin
* Fix batch_size behavior ProcessorTesterMixin
* Change format prepare inputs batched
* Remove override test pixtral processor
* Remove unnecessary tests and cleanup after new prepare_inputs functions
* Fix instructBlipVideo image processor
* fix(copy): fixup copy
* fix(deformable_detr): move weight initialization to the right place
* fix(grounding_dino): move weight initialization to the right place
* fix(rt_detr): move weight initialization to the right place
* [run-slow] deformable_detr, grounding_dino, rt_detr
* Remove max_new_tokens arg
* Add ASR pipeline to testing
* make fixup
* Factor the output test out into a util
* Full error reporting
* Full error reporting
* Update src/transformers/pipelines/automatic_speech_recognition.py
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Small comment
---------
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Add include_loss_for_metrics
* Fix styling
* Initialize inputs and losses to avoid AttributeError
* Ruff styling
* Refactor compute_metrics and update EvalPrediction
* Change Naming
* Added include_for_metrics to group both args
* Fix style
* Change warnings to logger
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Validate the eval dataset in advance.
* format
* format
* format
* Update src/transformers/trainer.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* format
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* fix(m2m_100): skip dropout in eval for flash_attn
* fix(misc): skip dropout in eval for flash attn various models
* chore(m2m_100): copy flash attn from bart
* chore: run make fix-copies
* [run-slow] bart, m2m_100
* refactor image features selection
* break line
* remove whitespace
* add pr comments: include projection and rename function
* make fix-copies
* fix get_image_feature in vip llava
* Fix Mamba slow path bug with dtype mismatch.
* Update test_modeling_mamba.py
* Improve style.
* Fix issue with cache position of dtype mismatch test.
* Change test for slow path.
* Revert changes.
* Switch to buggy code and add test to catch it.
* Fix the dtype mismatch bug and add test code to verify it.
* Fix minor bug with test.
* Fix incorrect dtype of model output.
* Fix incorrect dtype of cache.
* Fix incorrect dtype of ssm cache.
* Fix incorrect dtype of conv state.
* Remove assertion for ssm state.
* Add assertion for conv state dtype.
* Fix all issues with dtype mismatch test.
* HQQ model serialization attempt
* fix hqq dispatch and unexpected keys
* style
* remove check_old_param
* revert to check HQQLinear in quantizer_hqq.py
* revert to check HQQLinear in quantizer_hqq.py
* update HqqConfig default params
* make ci happy
* make ci happy
* revert to HQQLinear check in quantizer_hqq.py
* check hqq_min version 0.2.0
* set axis=1 as default in quantization_config.py
* validate_env with hqq>=0.2.0 version message
* deprecated hqq kwargs message
* make ci happy
* remove run_expected_keys_check hack + bump to 0.2.1 min hqq version
* fix unexpected_keys hqq update
* add pre_quantized check
* add update_expected_keys to base quantizerr
* ci base.py fix?
* ci base.py fix?
* fix "quantization typo" src/transformers/utils/quantization_config.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* fix post merge
---------
Co-authored-by: Marc Sun <marc@huggingface.co>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
