linlin
1c1aec2ef1
Update object_detection.md ( #31488 )
...
Define MAX_SIZE before it is used.
2024-06-19 10:36:44 +01:00
Joao Gante
83259e406d
Mamba: add generative tests ( #31478 )
2024-06-19 10:27:23 +01:00
Younes Belkada
7d683f7bae
Docs / AQLM: Clarify torch.compile support for AQLM ( #31473 )
...
Update overview.md
2024-06-19 11:26:25 +02:00
Fanli Lin
077c139f57
[tests] rename test_config_object to test_ds_config_object ( #31403 )
...
fix name
2024-06-19 11:19:15 +02:00
amyeroberts
609e662243
Use self.config_tester.run_common_tests() ( #31431 )
...
* First testing updating config tests
* Use run_common_tests
2024-06-19 10:18:08 +01:00
Phillip Rust
7c71b61dae
Fix autocast incompatibility in RecurrentGemma ( #30832 )
2024-06-19 09:59:34 +02:00
Anton Vlasjuk
b275a41005
[GPT2] Add SDPA support ( #31172 )
...
* `gpt2` sdpa support
* fix (at least) one test, style, repo consistency
* fix sdpa mask in forward --> fixes generation
* test
* test2
* test3
* test4
* simplify shapes for attn mask creation and small comments
* hub fail test
* benchmarks
* flash attn 2 mask should not be inverted on enc-dec setup
* fix comment
* apply some suggestion from code review
- only save _attn_implentation once
- remove unnecessary comment
* change elif logic
* [run-slow] gpt2
* modify `test_gpt2_sample_max_time` to follow previous assertion patterns
2024-06-19 09:40:57 +02:00
Rémy Léone
22b41b3f8a
Update perf_train_gpu_many.md ( #31451 )
...
* Update perf_train_gpu_many.md
* Update docs/source/en/perf_train_gpu_many.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/en/perf_train_gpu_many.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
2024-06-18 11:00:26 -07:00
Tom Aarsen
280cef51b3
Give more useful metric_for_best_model errors ( #31450 )
...
Give more useful metric_for_best_model errors
2024-06-18 16:56:30 +01:00
Quentin Gallouédec
2505357e4f
Fix documentation typos ( #31476 )
...
Fix doc typo
2024-06-18 16:09:50 +01:00
dependabot[bot]
4691ffbd41
Bump urllib3 from 1.26.18 to 1.26.19 in /examples/research_projects/visual_bert ( #31472 )
...
Bump urllib3 in /examples/research_projects/visual_bert
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 1.26.18 to 1.26.19.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.18...1.26.19 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-18 16:08:15 +01:00
Yih-Dar
1c7c34bc64
Improve PreTrainedTokenizerFast loading time when there are many added tokens ( #31404 )
...
* use hash
* use hash
* update
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2024-06-18 15:20:14 +02:00
Matt
6e56b83453
Update chat template docs and bump Jinja version ( #31455 )
...
* Update chat template docs
* Minor bug in the version check
* Update docs/source/en/chat_templating.md
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Update docs/source/en/chat_templating.md
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Update docs/source/en/chat_templating.md
Co-authored-by: Joshua Lochner <admin@xenova.com>
* Replace backticks with bolding because the doc builder was trying to parse them
* Replace backticks with bolding because the doc builder was trying to parse them
* Replace backticks with bolding because the doc builder was trying to parse them
* More cleanups to avoid upsetting the doc builder
* Add one more tip at the end
---------
Co-authored-by: Joshua Lochner <admin@xenova.com>
2024-06-18 14:16:30 +01:00
Matt
28316d0e8b
Fix single letter stop strings ( #31448 )
...
* Fix single letter stop strings
* Change the 0 to a 1 to avoid potential empty vector headaches later
* Restructure for clarity
* Update tests/generation/test_stopping_criteria.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Add the unsqueeze
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-06-18 14:07:16 +01:00
Matt
dabf01973a
Make "tool_use" the default chat template key when tools are passed ( #31429 )
...
* Make "tool_use" the default when tools are passed
* Add some opinionated text to the docs
* Add some opinionated text to the docs
2024-06-18 13:54:42 +01:00
Joao Gante
cd71f9381b
Donut: fix generate call from local path ( #31470 )
...
* local donut path fix
* engrish
* Update src/transformers/generation/utils.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-06-18 13:28:06 +01:00
dependabot[bot]
76289fbc7c
Bump urllib3 from 1.26.18 to 1.26.19 in /examples/research_projects/decision_transformer ( #31459 )
...
Bump urllib3 in /examples/research_projects/decision_transformer
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 1.26.18 to 1.26.19.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/1.26.18...1.26.19 )
---
updated-dependencies:
- dependency-name: urllib3
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-18 12:22:25 +01:00
Aymeric Roucher
b38612d312
Agents: Improve python interpreter ( #31409 )
...
* Improve Python interpreter
* Add with and assert statements
* Prevent overwriting existing tools
* Check interpreter errors are well logged in code agent
* Add lazy evaluation for and and or
* Improve variable assignment
* Fix early return statements in functions
* Add small import fix on interpreter tool
2024-06-18 11:55:36 +02:00
Kevin Hu
1f9387d33d
Fix typing errors in Qwen2ForTokenClassification ( #31440 )
...
* Update modeling_qwen2.py
* Fix llama
* More fixes
2024-06-18 10:27:18 +01:00
Kerim
9ba9369a25
simple fix ( #31456 )
2024-06-17 22:30:37 +01:00
Ella Charlaix
02300273e2
🚨 Remove dataset with restrictive license ( #31452 )
...
remove dataset with restrictive license
2024-06-17 17:56:51 +01:00
Albert Villanova del Moral
a14b055b65
Pass datasets trust_remote_code ( #31406 )
...
* Pass datasets trust_remote_code
* Pass trust_remote_code in more tests
* Add trust_remote_dataset_code arg to some tests
* Revert "Temporarily pin datasets upper version to fix CI"
This reverts commit b7672826ca833fc17a3e
2024-06-17 17:29:13 +01:00
Bastien Le Chenadec
485fd81471
Support multiple validation datasets when dataloader_persistent_workers=True ( #30627 )
...
* Support multiple validation datasets when dataloader_persistent_workers=True
* Test support of multiple validation datasets
2024-06-17 16:58:39 +01:00
dependabot[bot]
147c404fb1
Bump idna from 2.8 to 3.7 in /examples/research_projects/visual_bert ( #30201 )
...
Bumps [idna](https://github.com/kjd/idna ) from 2.8 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases )
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst )
- [Commits](https://github.com/kjd/idna/compare/v2.8...v3.7 )
---
updated-dependencies:
- dependency-name: idna
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-17 16:39:42 +01:00
Fanli Lin
9454f437b0
[tests] make TestDeepSpeedModelZoo device-agnostic ( #31402 )
...
* fix
* use accelerator device count
* ci fix
2024-06-17 16:42:57 +02:00
dependabot[bot]
7977f206dc
Bump idna from 2.8 to 3.7 in /examples/research_projects/lxmert ( #30200 )
...
Bumps [idna](https://github.com/kjd/idna ) from 2.8 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases )
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst )
- [Commits](https://github.com/kjd/idna/compare/v2.8...v3.7 )
---
updated-dependencies:
- dependency-name: idna
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-17 15:13:33 +01:00
dependabot[bot]
ee197e2b9e
Bump idna from 3.3 to 3.7 in /examples/research_projects/decision_transformer ( #30203 )
...
Bump idna in /examples/research_projects/decision_transformer
Bumps [idna](https://github.com/kjd/idna ) from 3.3 to 3.7.
- [Release notes](https://github.com/kjd/idna/releases )
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst )
- [Commits](https://github.com/kjd/idna/compare/v3.3...v3.7 )
---
updated-dependencies:
- dependency-name: idna
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-17 11:13:16 +01:00
Joao Gante
377e903928
Generate: fix tokenizer being popped twice ( #31427 )
2024-06-17 10:36:10 +01:00
amyeroberts
02c525d226
Rename misnamed image processor test files ( #31430 )
2024-06-17 10:21:28 +01:00
Yoach Lacombe
7ae4fc271d
Fix Bark logits processors device misplacement ( #31416 )
...
Fix Logits Processors device misplacement
2024-06-17 09:54:06 +02:00
Raushan Turganbay
9af1b6a80a
Musicgen special tokens in tensors ( #31420 )
...
fix
2024-06-17 10:09:27 +05:00
Dmitry Rogozhkin
eed9ed6798
xpu: support xpu backend from stock pytorch (>=2.4) ( #31238 )
...
* xpu: support xpu backend from stock pytorch (>=2.4)
Fixes: https://github.com/huggingface/transformers/issues/31237
XPU backend is available in the stock PyTorch starting from
version 2.4, see [1]. This commit extends huggingface transformers
to support XPU from both IPEX and the stock pytorch. IPEX is being
tried first.
See: https://github.com/pytorch/pytorch/issues/114842
Requires: https://github.com/huggingface/accelerate/pull/2825
Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
* xpu: enable gpt2 and decision_transformer tests for xpu pytorch backend
Note that running xpu tests requires TRANSFORMERS_TEST_DEVICE_SPEC=spec.py
passed to the test runner:
import torch
DEVICE_NAME = 'xpu'
MANUAL_SEED_FN = torch.xpu.manual_seed
EMPTY_CACHE_FN = torch.xpu.empty_cache
DEVICE_COUNT_FN = torch.xpu.device_count
Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
---------
Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
2024-06-14 21:31:35 +02:00
amyeroberts
20812237ce
Remove empty create_and_test_config_common_properties tests ( #31359 )
...
Remove empty tests
2024-06-14 20:15:48 +01:00
amyeroberts
3d0bd86915
Install the tensorflow example requirements in docker ( #31428 )
2024-06-14 19:35:43 +01:00
amyeroberts
11f43c15d3
Remove duplicate image processor in auto map ( #31383 )
2024-06-14 18:23:55 +01:00
Ian McKenzie
c212ac9a02
Change potential inputs_embeds padding logger.warning to logger.warning_once ( #31411 )
...
change embeddings padding warning to warning_once
2024-06-14 17:36:15 +01:00
Yoach Lacombe
7e1c7dc8b6
Fix SpeechT5 decoder_attention_mask shape ( #28071 )
...
* Fix SpeechT5
* add test foward with labels and attention mask
* make style
2024-06-14 15:20:11 +02:00
Yoach Lacombe
d9daeff297
Set seed for M4T retain grad test ( #31419 )
2024-06-14 14:48:04 +02:00
Yoach Lacombe
43ee58588b
Fix MusicGen SDPA ( #31208 )
...
* fix sdpa musicgen
* make style
* remove copied from statement from Musicgen SDPA
2024-06-14 13:30:44 +02:00
amyeroberts
833fc17a3e
Pin datasets<2.20.0 for examples ( #31417 )
2024-06-14 12:06:56 +01:00
Mansu Kim
cfb22e035e
Support Clip QKV for MPT ( #31307 )
2024-06-14 11:47:06 +01:00
Albert Villanova del Moral
b7672826ca
Temporarily pin datasets upper version to fix CI ( #31407 )
...
Temporarily pin datasets upper version
2024-06-13 18:01:18 +01:00
Jade Choghari
67a4ef89d4
Add missing French translation of tutoriel_pipeline.md ( #31396 )
...
* Update french translation of tutoriel_pipeline.md
* Update docs/source/fr/tutoriel_pipeline.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update docs/source/fr/tutoriel_pipeline.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update docs/source/fr/tutoriel_pipeline.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update docs/source/fr/tutoriel_pipeline.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update docs/source/fr/tutoriel_pipeline.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update docs/source/fr/tutoriel_pipeline.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update docs/source/fr/tutoriel_pipeline.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
* Update docs/source/fr/tutoriel_pipeline.md
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
---------
Co-authored-by: Jade Choghari <chogharijade@icloud.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
2024-06-13 17:48:54 +02:00
Pablo Montalvo
c624d5ba0b
add initial design for uniform processors + align model ( #31197 )
...
* add initial design for uniform processors + align model
* fix mutable default 👀
* add configuration test
* handle structured kwargs w defaults + add test
* protect torch-specific test
* fix style
* fix
* fix assertEqual
* move kwargs merging to processing common
* rework kwargs for type hinting
* just get Unpack from extensions
* run-slow[align]
* handle kwargs passed as nested dict
* add from_pretrained test for nested kwargs handling
* [run-slow]align
* update documentation + imports
* update audio inputs
* protect audio types, silly
* try removing imports
* make things simpler
* simplerer
* move out kwargs test to common mixin
* [run-slow]align
* skip tests for old processors
* [run-slow]align, clip
* !$#@!! protect imports, darn it
* [run-slow]align, clip
* [run-slow]align, clip
* update doc
* improve documentation for default values
* add model_max_length testing
This parameter depends on tokenizers received.
* Raise if kwargs are specified in two places
* fix
* expand VideoInput
* fix
* fix style
* remove defaults values
* add comment to indicate documentation on adding kwargs
* protect imports
* [run-slow]align
* fix
* remove set() that breaks ordering
* test more
* removed unused func
* [run-slow]align
2024-06-13 16:27:16 +02:00
Matt
15b3923d65
Make chat templates part of ProcessorMixin ( #30744 )
...
* Let's try moving chat templates out of IDEFICS and into the generic ProcessorMixin
* Chat templates should not be mandatory
* Chat templates should not be mandatory
* Not all classes will have default chat templates
* stash commit
* Add chat template docstring
* Clean up docstring
* Add chat templates to LLaVA/LLaVA-next
* Docstring fixup
* Quick IDEFICS2 fixup
* Remove some old references to the Conversation class
* make fixup
2024-06-13 14:35:30 +01:00
Billy Cao
3c4a8dca0c
[QoL fix] [Image processing] Add warning on assumption of channel dim and avoid infering when inputs are PIL.Image ( #31364 )
...
* Add warning on assumption of channel dim
Use PIL info whenever possible to decide channel axis
* Fix ruff format
* Remove type checking
Improve warning message
* Update src/transformers/models/siglip/image_processing_siglip.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/image_utils.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
* Update src/transformers/image_utils.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2024-06-13 10:34:58 +01:00
Luc Georges
348e2294ac
feat(ci): add trufflehog secrets detection ( #31344 )
2024-06-12 18:00:43 +02:00
조준래
17896f6783
Change JSON serialization to custom json.dumps ( #31100 )
...
* Change JSON serialization to custom json.dumps to prevent escaping of "<", ">", "&", "'"
* caller has control over the order, remove sort_key=True
* Move tojson into a proper function and expose a couple of other args
---------
Co-authored-by: jun.4 <jun.4@kakaobrain.com>
Co-authored-by: Matt <rocketknight1@gmail.com>
2024-06-12 14:59:35 +01:00
dependabot[bot]
1c77b3d9cf
Bump jupyter-core from 4.6.3 to 4.11.2 in /examples/research_projects/visual_bert ( #31386 )
...
Bump jupyter-core in /examples/research_projects/visual_bert
Bumps [jupyter-core](https://github.com/jupyter/jupyter_core ) from 4.6.3 to 4.11.2.
- [Release notes](https://github.com/jupyter/jupyter_core/releases )
- [Changelog](https://github.com/jupyter/jupyter_core/blob/main/CHANGELOG.md )
- [Commits](https://github.com/jupyter/jupyter_core/compare/4.6.3...4.11.2 )
---
updated-dependencies:
- dependency-name: jupyter-core
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-06-12 14:12:53 +01:00
Marc Sun
254b25abd9
Use huggingface_hub helper function to split state dict ( #31091 )
...
* shard saving from hf hub
* index = None
* fix tests
* indent
2024-06-12 14:10:32 +02:00
