* Error condition bug fix
* Update error message
* Update src/transformers/models/qwen2_vl/modeling_qwen2_vl.py
Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>
* Making change in the rest of the repo
* Formatting
* Formatting with ruff
---------
Co-authored-by: Pavel Iakubovskii <qubvel@gmail.com>
* Add support for `weights_only` flag when loading state_dict
Summary:
This is to enable loading a state_dict with wrapper tensor subclasses (used in torchao to
for quantized weights)
Test Plan:
tested locally with torchao weights, also need https://github.com/huggingface/transformers/pull/32306:
```
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import TorchAoConfig
from torchao.utils import benchmark_model
import torchao
DEVICE_TYPE = "cuda"
def init_model_and_benchmark(model_id, torch_dtype=torch.bfloat16, quantization_config=None):
tokenizer = AutoTokenizer.from_pretrained(model_id)
if quantization_config is not None:
model = AutoModelForCausalLM.from_pretrained(model_id, device_map=DEVICE_TYPE, torch_dtype=torch.\bfloat16, quantization_config=quantization_config)
else:
model = AutoModelForCausalLM.from_pretrained(model_id, device_map=DEVICE_TYPE, torch_dtype=torch.\bfloat16, weights_only=False)
# sanity check: run the model
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to(DEVICE_TYPE)
output = model.generate(**input_ids, max_new_tokens=1000)
print(tokenizer.decode(output[0], skip_special_tokens=True))
NUM_WARMUP = 1
NUM_RUNS = 5
if quantization_config is not None:
torchao.quantization.utils.recommended_inductor_config_setter()
model = torch.compile(model, mode="max-autotune")
benchmark_model(model.generate, NUM_WARMUP, kwargs=input_ids, device_type=DEVICE_TYPE)
print("running benchmark")
results = benchmark_model(model.generate, NUM_RUNS, kwargs=input_ids, device_type=DEVICE_TYPE)
return model, results
model_id = "jerryzh168/test-model"
torchao.quantization.utils.recommended_inductor_config_setter()
bf16_model, bf16_time = init_model_and_benchmark(model_id)
print(f"bf16: {bf16_time}")
```
Reviewers:
Subscribers:
Tasks:
Tags:
* format
* [PEFT] Support low_cpu_mem_usage for PEFT loading
PEFT added support for low_cpu_mem_usage=True when loading adapters in
https://github.com/huggingface/peft/pull/1961. This feature is now
available when installing PEFT v0.13.0. With this PR, this option is
also supported when loading PEFT adapters directly into transformers
models.
Additionally, with this PR,
https://github.com/huggingface/diffusers/pull/9510 will be unblocked,
which implements this option in diffusers.
* Fix typo
* fix beam indices in token_timestamps
* fix attention_mask in FA2
* correct translation example with the right example
* correct how somes tests are using outputs + correct num_frames
* fix shortform batch prev cond tests
* make fix-copies
* make fix-copies
* take care of shifting beam indices
* [run-slow] whisper
* [run-slow] whisper
* Fix: use unidic-lite instead of ipadic as the tokenizer dictionary of Japanese
Signed-off-by: Kan Takahiro <kan@Kans-Mac-mini.local>
* fix the default name
---------
Signed-off-by: Kan Takahiro <kan@Kans-Mac-mini.local>
Co-authored-by: Kan Takahiro <kan@Kans-Mac-mini.local>
Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com>
* add unit tests for splinter_tokenizer
* add unit test for splinter tokenizer, pass in the question_token to be saved on save_pretrained called
* remove unused import
* remove vocab_splinter.txt, add Copied from, use fmt:on and fmt:off to prevent autoformatting on long lines
* remove all the spaces
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* Use all state dict keys when checking if root module is initialized.
* Apply style corrections
* Add comment explaining change.
* Change comment phrasing.
* try fixing push-ci
* move to new runners
* move benchmark.yml to new runners
* move doctest_job.yml to new runners
* move doctests.yml to new runners
* move push-important-models.yml to new runners
* move self-pr-slow-ci.yml to new runners
* fix typo
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* fix working directory
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* fix working directory
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* improve code
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
---------
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
* Update an keyerror on _save_check_point prevent confusion of missing metric keys
* Update grammar error and case sensitive.
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* adding update KeyError on _evaluate function to align with _save_checkpoint function
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* When we set self.dt_proj.bias = None, it removes the bias parameter from the model. When we later tried to assign a tensor to self.dt_proj.bias, it caused a TypeError because PyTorch expects a Parameter object.
* When we set self.dt_proj.bias = None, it removes the bias parameter from the model. When we later tried to assign a tensor to self.dt_proj.bias, it caused a TypeError because PyTorch expects a Parameter object.
* When we set self.dt_proj.bias = None, it removes the bias parameter from the model. When we later tried to assign a tensor to self.dt_proj.bias, it caused a TypeError because PyTorch expects a Parameter object.
* Trainer - deprecate tokenizer for processing_class
* Extend chage across Seq2Seq trainer and docs
* Add tests
* Update to FutureWarning and add deprecation version
* add support for custom inputs and batched inputs in ProcessorTesterMixin
* Fix batch_size behavior ProcessorTesterMixin
* Change format prepare inputs batched
* Remove override test pixtral processor
* Remove unnecessary tests and cleanup after new prepare_inputs functions
* Fix instructBlipVideo image processor
* fix(copy): fixup copy
* fix(deformable_detr): move weight initialization to the right place
* fix(grounding_dino): move weight initialization to the right place
* fix(rt_detr): move weight initialization to the right place
* [run-slow] deformable_detr, grounding_dino, rt_detr
* Remove max_new_tokens arg
* Add ASR pipeline to testing
* make fixup
* Factor the output test out into a util
* Full error reporting
* Full error reporting
* Update src/transformers/pipelines/automatic_speech_recognition.py
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Small comment
---------
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Add include_loss_for_metrics
* Fix styling
* Initialize inputs and losses to avoid AttributeError
* Ruff styling
* Refactor compute_metrics and update EvalPrediction
* Change Naming
* Added include_for_metrics to group both args
* Fix style
* Change warnings to logger
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* Validate the eval dataset in advance.
* format
* format
* format
* Update src/transformers/trainer.py
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* format
---------
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
* fix(m2m_100): skip dropout in eval for flash_attn
* fix(misc): skip dropout in eval for flash attn various models
* chore(m2m_100): copy flash attn from bart
* chore: run make fix-copies
* [run-slow] bart, m2m_100
* refactor image features selection
* break line
* remove whitespace
* add pr comments: include projection and rename function
* make fix-copies
* fix get_image_feature in vip llava
