Pavel Iakubovskii
3c2d4d60d7
Correct @is_flaky test decoration ( #31480 )
...
* Correct @is_flaky decorator
2024-06-24 08:09:21 +01:00
Nicholi Caron
4b822560a1
Update mask_generation.md ( #31543 )
...
Minor bug fixes -- rearrange import & add missing parentheses
2024-06-23 20:27:21 +01:00
Sangbum Daniel Choi
74a207404e
New model support RTDETR ( #29077 )
...
* fill out docs string in configuration
75dcd3a0e8 (r1506391856)https://github.com/huggingface/transformers/pull/29077#discussion_r1516657790
* simple typo
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* intermediate fix
* fix with load_backbone format
* remove unused configuration
* 3 fix test left
* make style
* Update src/transformers/models/rt_detr/image_processing_rt_detr.py
Co-authored-by: Sounak Dey <dey.sounak@gmail.com>
* change last_hidden_state to first index
* all pass fix
TO DO: minor update in comments
* make fix-copies
* remove deepcopy
* pr_document fix
* revert deepcopy due to the issue of unexpceted behavior in decoderlayer
* add atol in final
* add no_split_module
* _no_split_modules = None
* device transfer for model parallelism
* minor fix
* make fix-copies
* fix typo
* add test_image_processor with post_processing
* Update src/transformers/models/rt_detr/configuration_rt_detr.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* add config in RTDETRPredictionHead
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* set lru_cache with max_size 32
* Update src/transformers/models/rt_detr/configuration_rt_detr.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* add lru_cache import and configuration change
* change the order of definition
* make fix-copies
* add docs and change config error
* revert strange make-fix
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* test pass
* fix get_clones related and remove deepcopy
* Update src/transformers/models/rt_detr/configuration_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/configuration_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/image_processing_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/image_processing_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/image_processing_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/image_processing_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* nit for paper section
* Update src/transformers/models/rt_detr/configuration_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* rename denoising related parameters
* Update src/transformers/models/rt_detr/image_processing_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* check the image transformation logic
* make style
* make style
* Update src/transformers/models/rt_detr/configuration_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/modeling_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* pe_encoding -> positional_encoding_temperature
* remove TODO
* Update src/transformers/models/rt_detr/image_processing_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* remove eval_idx since transformer DETR is giving all decoder output
* Update src/transformers/models/rt_detr/configuration_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/rt_detr/configuration_rt_detr.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* change variable name
* make style and docs import update
* Revert "Update src/transformers/models/rt_detr/image_processing_rt_detr.py"
This reverts commit 74aa3e1de0ccb7a34051d136225cd3
2024-06-21 17:50:08 +01:00
Nate Cibik
8b7cd40273
Removed torch.cuda.empty_cache from train loop. ( #31530 )
2024-06-21 14:45:27 +01:00
Ita Zaporozhets
1e79eade41
SPLIT PR: add user defined symbols and control symbols ( #31305 )
...
* PR SPLIT: moving origina changes for adding user defined symbols
* adding gemma test and generalizing gemma converter
* ruff
* update common test
* update serialization test
* deberta v2 tests updates as rust version adds '.' as a user added token, so a space is not added
* removing commented lines
* applying feedback - user only added_tokens to add and check piece.type instead of trainer_spec for user_defined_symbols
* add comment referencing sentencepiece
2024-06-21 01:48:10 -07:00
Raushan Turganbay
730a440734
Deprecate legacy cache + use cache position ( #31491 )
...
* tmp
* update models
* revert utils
* delete
* Update src/transformers/models/dbrx/modeling_dbrx.py
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
* modify warning msg
---------
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
2024-06-21 09:28:14 +05:00
dependabot[bot]
12b1620e61
Bump urllib3 from 1.26.18 to 1.26.19 in /examples/research_projects/lxmert ( #31524 )
...
Bump urllib3 in /examples/research_projects/lxmert
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 1.26.18 to 1.26.19.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.18...1.26.19 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-20 19:45:53 +01:00
Yih-Dar
d4564df1d4
Revive Nightly/Past CI ( #31159 )
...
* build
* build
* build
* build
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2024-06-20 18:57:24 +02:00
Yih-Dar
ec905f3a76
unskip 2 tests in cohere ( #31517 )
...
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2024-06-20 17:21:08 +02:00
Joao Gante
1fd60fec75
RWKV: enable generation tests ( #31490 )
...
* add rwkv tests
* has_attentions set in individual tests
2024-06-20 14:15:01 +01:00
Jiahui Wei
d28e647f28
Fix mismatched ` in doc & other common typos ( #31516 )
...
fix common doc typos
Co-authored-by: Jiahui Wei <jiahui.wei@tusen.ai>
2024-06-20 14:03:07 +01:00
Younes Belkada
6d4306160a
GGUF: Fix llama 3 GGUF ( #31358 )
...
* Create push-important-models.yml
* llama3 support for GGUF
* fixup
* Update src/transformers/integrations/ggml.py
* fix pre-tokenizer
* fix
* fix
* fix
* fix
* fix
* fix
* address final comment
* handle special tokens + add tests
2024-06-20 14:29:58 +02:00
Sadra Barikbin
35b112d344
Fix a teeny-tiny typo in tokenization_utils_base.py's docstring ( #31510 )
...
Update tokenization_utils_base.py
2024-06-20 10:35:52 +01:00
arthasking123
0ed3ffcb44
Add valid columns check in _remove_unused_columns method ( #31466 )
...
* Add valid columns checking in _remove_unused_columns method
https://github.com/huggingface/datasets/issues/6973#issue-2355517362
https://github.com/huggingface/datasets/issues/6535
https://discuss.huggingface.co/t/indexerror-invalid-key-16-is-out-of-bounds-for-size-0/14298/25
* Update modeling_mixtral.py
* Update modeling_mixtral.py
* Update modeling_mixtral.py
2024-06-19 13:26:37 +01:00
Daemyung Jang
547b5582ec
Consider inheritance in type checking for tensors ( #31378 )
...
* Consider inheritance in type checking for tensors
Add an additional check to bypass type assertion when both tensors are
torch.Tensor instances.
* Fix the quality issue
2024-06-19 14:05:20 +02:00
Timothé Pearce
3b5fa14fb8
Fix wandb integration with SetFit model ( #30021 )
...
Fix W&B integration with SetFit model
Co-authored-by: PEARCE Timothe <timothe_pearce@ext.connect-tech.sncf>
2024-06-19 13:23:05 +02:00
nikkie
f4d189441d
Fix typo: pas_token_id ( #30894 )
...
Fix typo
2024-06-19 11:23:08 +01:00
Fanli Lin
4144c354e9
auto-detect device when no device is passed to pipeline ( #31398 )
...
* fix device
* Update src/transformers/pipelines/base.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* bug fix
* add warning
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-06-19 11:12:39 +01:00
Billy Cao
cd5f7c1790
Add docs on zeroshot image classification prompt templates ( #31343 )
...
* Add docs on pipeline templates
* Fix example and comments
Update usage tips
* Update docs/source/en/tasks/zero_shot_image_classification.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update docs/source/en/model_doc/siglip.md
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Trigger CI
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-06-19 11:11:44 +01:00
linlin
1c1aec2ef1
Update object_detection.md ( #31488 )
...
Define MAX_SIZE before it is used.
2024-06-19 10:36:44 +01:00
Joao Gante
83259e406d
Mamba: add generative tests ( #31478 )
2024-06-19 10:27:23 +01:00
Younes Belkada
7d683f7bae
Docs / AQLM: Clarify torch.compile support for AQLM ( #31473 )
...
Update overview.md
2024-06-19 11:26:25 +02:00
Fanli Lin
077c139f57
[tests] rename test_config_object to test_ds_config_object ( #31403 )
...
fix name
2024-06-19 11:19:15 +02:00
amyeroberts
609e662243
Use self.config_tester.run_common_tests() ( #31431 )
...
* First testing updating config tests
* Use run_common_tests
2024-06-19 10:18:08 +01:00
Phillip Rust
7c71b61dae
Fix autocast incompatibility in RecurrentGemma ( #30832 )
2024-06-19 09:59:34 +02:00
Anton Vlasjuk
b275a41005
[GPT2] Add SDPA support ( #31172 )
...
* `gpt2` sdpa support
* fix (at least) one test, style, repo consistency
* fix sdpa mask in forward --> fixes generation
* test
* test2
* test3
* test4
* simplify shapes for attn mask creation and small comments
* hub fail test
* benchmarks
* flash attn 2 mask should not be inverted on enc-dec setup
* fix comment
* apply some suggestion from code review
- only save _attn_implentation once
- remove unnecessary comment
* change elif logic
* [run-slow] gpt2
* modify `test_gpt2_sample_max_time` to follow previous assertion patterns
2024-06-19 09:40:57 +02:00
Rémy Léone
22b41b3f8a
Update perf_train_gpu_many.md ( #31451 )
...
* Update perf_train_gpu_many.md
* Update docs/source/en/perf_train_gpu_many.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/en/perf_train_gpu_many.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
2024-06-18 11:00:26 -07:00
Tom Aarsen
280cef51b3
Give more useful metric_for_best_model errors ( #31450 )
...
Give more useful metric_for_best_model errors
2024-06-18 16:56:30 +01:00
Quentin Gallouédec
2505357e4f
Fix documentation typos ( #31476 )
...
Fix doc typo
2024-06-18 16:09:50 +01:00
dependabot[bot]
4691ffbd41
Bump urllib3 from 1.26.18 to 1.26.19 in /examples/research_projects/visual_bert ( #31472 )
...
Bump urllib3 in /examples/research_projects/visual_bert
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 1.26.18 to 1.26.19.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.18...1.26.19 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-18 16:08:15 +01:00
Yih-Dar
1c7c34bc64
Improve PreTrainedTokenizerFast loading time when there are many added tokens ( #31404 )
...
* use hash
* use hash
* update
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2024-06-18 15:20:14 +02:00
Matt
6e56b83453
Update chat template docs and bump Jinja version ( #31455 )
...
* Update chat template docs
* Minor bug in the version check
* Update docs/source/en/chat_templating.md
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Update docs/source/en/chat_templating.md
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Update docs/source/en/chat_templating.md
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Replace backticks with bolding because the doc builder was trying to parse them
* Replace backticks with bolding because the doc builder was trying to parse them
* Replace backticks with bolding because the doc builder was trying to parse them
* More cleanups to avoid upsetting the doc builder
* Add one more tip at the end
---------
Co-authored-by: Joshua Lochner <admin@xenova.com>
2024-06-18 14:16:30 +01:00
Matt
28316d0e8b
Fix single letter stop strings ( #31448 )
...
* Fix single letter stop strings
* Change the 0 to a 1 to avoid potential empty vector headaches later
* Restructure for clarity
* Update tests/generation/test_stopping_criteria.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Add the unsqueeze
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-06-18 14:07:16 +01:00
Matt
dabf01973a
Make "tool_use" the default chat template key when tools are passed ( #31429 )
...
* Make "tool_use" the default when tools are passed
* Add some opinionated text to the docs
* Add some opinionated text to the docs
2024-06-18 13:54:42 +01:00
Joao Gante
cd71f9381b
Donut: fix generate call from local path ( #31470 )
...
* local donut path fix
* engrish
* Update src/transformers/generation/utils.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-06-18 13:28:06 +01:00
dependabot[bot]
76289fbc7c
Bump urllib3 from 1.26.18 to 1.26.19 in /examples/research_projects/decision_transformer ( #31459 )
...
Bump urllib3 in /examples/research_projects/decision_transformer
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 1.26.18 to 1.26.19.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.18...1.26.19 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-18 12:22:25 +01:00
Aymeric Roucher
b38612d312
Agents: Improve python interpreter ( #31409 )
...
* Improve Python interpreter
* Add with and assert statements
* Prevent overwriting existing tools
* Check interpreter errors are well logged in code agent
* Add lazy evaluation for and and or
* Improve variable assignment
* Fix early return statements in functions
* Add small import fix on interpreter tool
2024-06-18 11:55:36 +02:00
Kevin Hu
1f9387d33d
Fix typing errors in Qwen2ForTokenClassification ( #31440 )
...
* Update modeling_qwen2.py
* Fix llama
* More fixes
2024-06-18 10:27:18 +01:00
Kerim
9ba9369a25
simple fix ( #31456 )
2024-06-17 22:30:37 +01:00
Ella Charlaix
02300273e2
🚨 Remove dataset with restrictive license ( #31452 )
...
remove dataset with restrictive license
2024-06-17 17:56:51 +01:00
Albert Villanova del Moral
a14b055b65
Pass datasets trust_remote_code ( #31406 )
...
* Pass datasets trust_remote_code
* Pass trust_remote_code in more tests
* Add trust_remote_dataset_code arg to some tests
* Revert "Temporarily pin datasets upper version to fix CI"
This reverts commit b7672826ca833fc17a3e
2024-06-17 17:29:13 +01:00
Bastien Le Chenadec
485fd81471
Support multiple validation datasets when dataloader_persistent_workers=True ( #30627 )
...
* Support multiple validation datasets when dataloader_persistent_workers=True
* Test support of multiple validation datasets
2024-06-17 16:58:39 +01:00
dependabot[bot]
147c404fb1
Bump idna from 2.8 to 3.7 in /examples/research_projects/visual_bert ( #30201 )
...
Bumps [idna](https://github.com/kjd/idna ) from 2.8 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases )
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst )
- [Commits](https://github.com/kjd/idna/compare/v2.8...v3.7 )
---
updated-dependencies:
- dependency-name: idna
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-17 16:39:42 +01:00
Fanli Lin
9454f437b0
[tests] make TestDeepSpeedModelZoo device-agnostic ( #31402 )
...
* fix
* use accelerator device count
* ci fix
2024-06-17 16:42:57 +02:00
dependabot[bot]
7977f206dc
Bump idna from 2.8 to 3.7 in /examples/research_projects/lxmert ( #30200 )
...
Bumps [idna](https://github.com/kjd/idna ) from 2.8 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases )
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst )
- [Commits](https://github.com/kjd/idna/compare/v2.8...v3.7 )
---
updated-dependencies:
- dependency-name: idna
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-17 15:13:33 +01:00
dependabot[bot]
ee197e2b9e
Bump idna from 3.3 to 3.7 in /examples/research_projects/decision_transformer ( #30203 )
...
Bump idna in /examples/research_projects/decision_transformer
Bumps [idna](https://github.com/kjd/idna ) from 3.3 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases )
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst )
- [Commits](https://github.com/kjd/idna/compare/v3.3...v3.7 )
---
updated-dependencies:
- dependency-name: idna
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-17 11:13:16 +01:00
Joao Gante
377e903928
Generate: fix tokenizer being popped twice ( #31427 )
2024-06-17 10:36:10 +01:00
amyeroberts
02c525d226
Rename misnamed image processor test files ( #31430 )
2024-06-17 10:21:28 +01:00
Yoach Lacombe
7ae4fc271d
Fix Bark logits processors device misplacement ( #31416 )
...
Fix Logits Processors device misplacement
2024-06-17 09:54:06 +02:00
Raushan Turganbay
9af1b6a80a
Musicgen special tokens in tensors ( #31420 )
...
fix
2024-06-17 10:09:27 +05:00
