riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 10:12:23 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity
13,720 Commits 887 Branches 232 Tags 2.1 GiB
b4d5548800
Commit Graph

13720 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Arthur
b4d5548800
🚨🚨🚨 [SPM] Finish fix spm models 🚨🚨🚨 (#25224) 
* fix EVERYTHING

* more fixes

* ⚗️⚗️ Tokenizer magic ⚗️⚗️

* wrong value but test passes for the TODO

* update

* updat

* safe protobuf import?

* style

* non gated repo

* update

* fixup

* Update src/transformers/models/llama/tokenization_llama.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update src/transformers/models/llama/tokenization_llama.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update tests/models/t5/test_tokenization_t5.py

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* nits

* fix t5 too

* use assert equal

* fix llama decoding

* nits on t5

* fixup

* only remove the prefix space, not other spaces

* more deconding tests and more todos

* fix CI as well

* fixup

* skip failing test on CI (its tf its ok)

* skip test_subword_regularization_tokenizer that is also crashing on the CI for TF

* update llama

* revert good fixes

* fixup

* empty

* explain why we need to encode with an additional token

* better warning?

* nits

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-08-17 17:08:05 +02:00
Arthur
5347d00092
[SwitchTransformers] Remove unused module (#25427) 
* remove unused module

* remove old feed_forward_proj

* fixup
 2023-08-17 17:03:41 +02:00
Arthur
d6bf08f7f6
[resize_embedding] Introduce pad_to_multiple_of and guidance (#25088) 
* fix

* revert cahnges and update resizing of embedding layer

* use wraning

* fixup

* more styling nits

* fix all tests that overload the embedding tests

* 👀👀 remove breakpoint

* remove useless overload + overload correctly where needed

* resize lm head with new vocab size

* reverse not necessary changes

* style

* fix CIs!

* fix last CI tests, adapt bark and Marian

* fixup
 2023-08-17 17:00:32 +02:00
Yih-Dar
d2871b2975
Skip test_beam_search_xla_generate_simple for T5 (#25566) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-17 15:30:46 +02:00
Alex McKinney
1791ef8df6
Adds TRANSFORMERS_TEST_DEVICE (#25506) 
* Adds `TRANSFORMERS_TEST_DEVICE`
Mirrors the same API in the diffusers library. Useful in transformers
too.

* replace backend checking with trying `torch.device`

* Adds better error message for unknown test devices

* `make style`

* adds documentation showing `TRANSFORMERS_TEST_DEVICE` usage.
 2023-08-17 13:41:34 +02:00
Younes Belkada
e7e9261a20
[Docs] Fix un-rendered images (#25561) 
fix un-rendered images
 2023-08-17 12:08:11 +02:00
Yih-Dar
8992589dd6
Skip test_onnx_runtime_optimize for now (#25560) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-17 11:23:16 +02:00
amyeroberts
e50c9253f3
YOLOS - reset default return_pixel_mask value (#25559) 
Remove added back copied from statement
 2023-08-17 09:48:38 +01:00
amyeroberts
c8346cb267
🚨🚨🚨 Vivit update default rescale_factor value (#25547) 
* Update default rescale_factor value

* Formatting
 2023-08-17 09:35:56 +01:00
Yih-Dar
8fd6561981
Fix torch.fx tests on nightly CI (#25549) 
* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-17 10:02:54 +02:00
Yih-Dar
ec25306b39
Fix MPT CI (#25548) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-17 09:06:26 +02:00
Sylvain Gugger
297a6a7aea
Add documentation to dynamic module utils (#25534) 
* Add documentation to dynamic module utils

* Address review comments
 2023-08-17 08:28:06 +02:00
Yun Dai
d1832dd808
Update trainer.py (#25553) 2023-08-17 08:10:33 +02:00
Juntae
db816c6e02
[i18n-KO] Translated docs: ko: pr_checks.md to Korean (#24987) 
* docs: ko: pr_checks.mdx

* feat: chatgpt draft

* fix: manual edits

* fix: resolve suggestions

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>

* feat: chatgpt draft

* fix: manual edits

---------

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
 2023-08-17 08:03:17 +02:00
Sylvain Gugger
2defb6b048
More utils doc (#25457) 
* Document and clean more utils.

* More documentation and fixes

* Switch to Lysandre's token

* Address review comments

* Actually put else
 2023-08-17 07:58:35 +02:00
Sanchit Gandhi
36f183ebab
[ASR Pipeline] Fix init with timestamps (#25438) 
* [ASR Pipeline] Fix init

* refactor test

* change default kwarg setting

* only perform checks if we have to

* override init

* move pre/forward/post checks to sanitize
 2023-08-16 18:04:19 +01:00
amyeroberts
6bca43bb90
Input data format (#25464) 
* Add copied from statements for image processors

* Move out rescale and normalize to base image processor

* Remove rescale and normalize from vit (post rebase)

* Update docstrings and tidy up

* PR comments

* Add input_data_format as preprocess argument

* Resolve tests and tidy up

* Remove num_channels argument

* Update doc strings -> default ints not in code formatting
 2023-08-16 17:45:02 +01:00
Zach Mueller
a6609caf4e
More frozen args (#25540) 2023-08-16 12:19:51 -04:00
Yih-Dar
f61f072b61
Fix MaskFormerModelIntegrationTest OOM (#25544) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-16 18:11:24 +02:00
Marc Sun
0ed23e4db2
fix vit hybrid test (#25543) 
fix test
 2023-08-16 17:02:57 +02:00
Joao Gante
3f9cb33504
Generate: fix default max length warning (#25539) 2023-08-16 15:30:54 +01:00
Sylvain Gugger
e13d5b6048
Document the test fetcher (#25521) 
* Document the test fetcher

* Address review comments
 2023-08-16 14:18:32 +02:00
Joao Gante
0b568291d7
Marian: post-hack-fix correction (#25459) 2023-08-16 11:49:29 +01:00
Sylvain Gugger
5ccf343aeb
Fix nested configs of Jukebox (#25533) 2023-08-16 11:48:24 +02:00
lishukan
c385de2441
[TYPO] fix typo/format in quicktour.md (#25519) 
* fix_all_language_quicktour

* give up ! before bash command

---------

Co-authored-by: lishukan <lishukan@dxy.cn>
 2023-08-16 08:03:23 +02:00
Matt
eec5841e9f
Use dynamic past key-values shape in TF-Whisper (#25523) 2023-08-15 17:57:58 +01:00
Zach Mueller
ca51499248
Make training args fully immutable (#25435) 
* Make training args fully immutable

* Working tests, PyTorch

* In test_trainer

* during testing

* Use proper dataclass way

* Fix test

* Another one

* Fix tf

* Lingering slow

* Exception

* Clean
 2023-08-15 11:47:47 -04:00
YQ
f11518a542
add __repr__ to the BitsAndBytesConfig class (#25517) 
add __repr__
 2023-08-15 11:11:28 +02:00
dependabot[bot]
7a94ea4c64
Bump tornado from 6.3.2 to 6.3.3 in /examples/research_projects/lxmert (#25511) 
Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-15 08:52:30 +02:00
dependabot[bot]
2552b8c5bd
Bump tornado from 6.3.2 to 6.3.3 in /examples/research_projects/visual_bert (#25512) 
Bump tornado in /examples/research_projects/visual_bert

Bumps [tornado](https://github.com/tornadoweb/tornado) from 6.3.2 to 6.3.3.
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.3.2...v6.3.3)

---
updated-dependencies:
- dependency-name: tornado
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-15 08:52:20 +02:00
Michael Murray
df91ff5314
Check for case where auxiliary_head is None in UperNetPreTrainedModel (#25514) 
check for case where auxiliary_head is None in UperNetPreTrainedModel
 2023-08-15 08:44:21 +02:00
Matt
b42010bb1d
Conditional DETR type hint fix (#25505) 2023-08-14 18:12:06 +01:00
amyeroberts
c41291965f
🚨🚨🚨 Remove softmax for EfficientNetForImageClassification 🚨🚨🚨 (#25501) 
* Remove softmax for EfficientNet

* Update integration test values

* Fix up
 2023-08-14 17:08:47 +01:00
Marc Sun
06a1d75bd5
fix gptq nits (#25500) 
* fix nits

* fix docstring

* fix doc

* fix damp_percent

* fix doc
 2023-08-14 11:43:38 -04:00
amyeroberts
80f29a25a7
MaskFormer post_process_instance_segmentation bug fix convert out side of loop (#25497) 
Bug fix - convert out side of loop
 2023-08-14 16:00:57 +01:00
Yoach Lacombe
ee7d6694ed
Set can_generate for SpeechT5ForTextToSpeech (#25493) 
add can_generate=True to SpeechT5ForTextToSpeech
 2023-08-14 15:41:47 +01:00
David Reguera
87c9d8a10f
Add type hints to Blip2QFormer, BigBirdForQA and ConditionalDetr family models (#25488) 
* Add missing type hints to `BigBirdForQuestionAnswering`

* Add type hints to `Blip2QFormerModel`

* Add type hints for `ConditionalDetr` family
 2023-08-14 14:44:34 +01:00
Matt
b1b0fc4f56
Remove logging code in TF Longformer that fails to compile (#25496) 
Remove wonky logger block
 2023-08-14 14:22:15 +01:00
nour-elkamel
e97deca9a3
fix : escape key of start_token from special characters before search end_token in token2json function of DonutProcessor (#25472) 
fix : escape key of start_token from special characters before searching for end_token
 2023-08-14 13:46:17 +02:00
dependabot[bot]
0ebe7ae160
Bump gitpython from 3.1.30 to 3.1.32 in /examples/research_projects/decision_transformer (#25467) 
Bump gitpython in /examples/research_projects/decision_transformer

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.32.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.32)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-13 19:47:16 +02:00
dependabot[bot]
2b22cde71e
Bump gitpython from 3.1.30 to 3.1.32 in /examples/research_projects/distillation (#25468) 
Bump gitpython in /examples/research_projects/distillation

Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.30 to 3.1.32.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases)
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES)
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.30...3.1.32)

---
updated-dependencies:
- dependency-name: gitpython
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-08-13 19:47:04 +02:00
Erfan Zekri Esfahani
892f9ea0db
import required torch and numpy libraries (#25483) 2023-08-13 19:26:40 +02:00
Yih-Dar
fe3c8ab1af
Revert "Reuse the cache created for latest main on PRs/branches" (#25466) 
Revert "Reuse the cache created for latest `main` on PRs/branches if `setup.py` is not modified (#25445)"

This reverts commit 1d75768695.
 2023-08-11 21:07:08 +02:00
amyeroberts
5e5fa0d88c
Mark flaky tests (#25463) 
Make CI less brittle
 2023-08-11 15:26:45 +01:00
amyeroberts
11757e2bbd
Add input_data_format argument, image transforms (#25462) 
* Enable specifying input data format - overriding inferring

* Add tests
 2023-08-11 15:09:31 +01:00
Gema Parreño
0acf56224b
Update run_translation.py broken link example Pytoch (#25461) 
* Update run_translation.py

Fixed link

* Update run_translation.py
 2023-08-11 15:41:24 +02:00
Yih-Dar
1d75768695
Reuse the cache created for latest main on PRs/branches if setup.py is not modified (#25445) 
* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-11 14:40:51 +02:00
Joao Gante
4692d26194
Switch Transformers: remove overwritten beam sample test (#25458) 2023-08-11 13:16:01 +01:00
amyeroberts
41d56ea6dd
Refactor image processor testers (#25450) 
* Refactor image processor test mixin

- Move test_call_numpy, test_call_pytorch, test_call_pil to mixin
- Rename mixin to reflect handling of logic more than saving
- Add prepare_image_inputs, expected_image_outputs for tests

* Fix for oneformer
 2023-08-11 11:30:18 +01:00
Yih-Dar
454957c9bb
Fix for #25437 (#25454) 
* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-11 11:39:57 +02:00