Commit Graph

13662 Commits

Author SHA1 Message Date
Sylvain Gugger
16edf4d9fd
Doc checks (#25408)
* Document check_dummies

* Type hints and doc in other files

* Document check inits

* Add documentation to

* Address review comments
2023-08-10 10:53:22 +02:00
TaeYupNoh
b14d4641f6
🌐 [i18n-KO] Translated philosophy.md to Korean (#25010)
* docs: ko: philosophy.md

* feat: chatgpt draft

* fix: manual edits

* fix: resolve suggestions
2023-08-10 09:50:51 +02:00
NielsRogge
b175fc39d9
[DINOv2] Update pooler output (#25392)
Update pooler output
2023-08-10 09:13:52 +02:00
Joao Gante
d0c1aebea4
Bark: flexible generation config overload (#25414) 2023-08-09 18:51:51 +01:00
amyeroberts
944ddce8bf
Enable passing number of channels when inferring data format (#25412) 2023-08-09 17:41:21 +01:00
hukuda222
cb3c821cb7
aligned sample_beam output selection with beam_search (#25375)
* aligned sample_beam specs with beam_search

* pull origin main

* Revert "pull origin main"

This reverts commit 06d356f113.

* update test_utils.py

* fix format

* remove comment

---------

Co-authored-by: Shogo Fujita <shogo.fujita@legalontech.jp>
2023-08-09 18:28:57 +02:00
Yoach Lacombe
704bf595eb
Update Bark generation configs and tests (#25409)
* update bark generation configs for more coherent parameter

* make style

* update bark hub repo
2023-08-09 18:28:02 +02:00
Hyeonseo Yun
cf84738d2e
🌐 [i18n-KO] Translated model_summary.md to Korean (#24625)
* docs: ko: model_summary.md

* feat: nmt and manual edit model_summary.mdx

* fix: resolve suggestions

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>

* fix: resolve suggestions2

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>

---------

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
2023-08-09 18:27:27 +02:00
MinJae Kang
133aac09b0
🌐 [i18n-KO] Translated add_new_model.md to Korean (#24957)
* docs: ko: add_new_model.md

* feat: chatgpt draft

* fix: manual edits

* fix: change document title

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>

* fix: edit with reviewers

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

* fix: add anchor to header

* Update docs/source/ko/add_new_model.md

Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>

* Update docs/source/ko/add_new_model.md

Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>

* Update docs/source/ko/add_new_model.md

Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>

* fix: edit with reviews

* feat: edit toctree

---------

Co-authored-by: Wonhyeong Seo <wonhseo@kakao.com>
Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>
Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com>
Co-authored-by: 이서정 <97655267+sjlee-wise@users.noreply.github.com>
2023-08-09 18:24:29 +02:00
Maria Khalusova
f2a43c7383
VQA task guide (#25244)
* initial commit

* semi-finished task guide draft

* image link

* Apply suggestions from code review

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/tasks/visual_question_answering.md

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* feedback addressed

* Apply suggestions from code review

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* nits addressed

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2023-08-09 08:29:06 -04:00
Joao Gante
eb3ded16f7
Generate: lower severity of parameterization checks (#25407) 2023-08-09 13:15:06 +01:00
David Reguera
ef74da6582
16059 - Add extra type hints for AltCLIPModel (#25399) 2023-08-09 13:13:33 +01:00
Joao Gante
f456b4d10b
Generate: generation config validation fixes in docs (#25405) 2023-08-09 13:07:11 +01:00
Alan Ji
00b93cda21
Improve training args (#25401)
* enhanced tips for some training args

* make style
2023-08-09 13:50:13 +02:00
Joao Gante
3deed1f97e
Generate: length validation (#25384) 2023-08-09 11:48:32 +01:00
Joao Gante
d59b872c9e
Docs: introduction to generation with LLMs (#25240)
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
2023-08-09 11:09:20 +01:00
amyeroberts
ea5dda2290
YOLOS - Revert default return_pixel_mask value (#25404)
Revert default return_pixel_mask value
2023-08-09 11:09:09 +01:00
Sylvain Gugger
599377161b
Fix path for dynamic module creation (#25402) 2023-08-09 10:46:05 +02:00
jiqing-feng
85447bb22e
rm useless condition since the previous condition contains it. (#25403) 2023-08-09 09:31:24 +02:00
David Reguera
1564a81ac5
16059 - Add missing type hints for ASTModel (#25364)
* 16059 - Add missing type hints for ASTModel

* Add an additional type hint

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

---------

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
2023-08-09 08:31:57 +02:00
SeongWooChoi
1367142afd
🌐 [i18n-KO] Translated perf_train_cpu_many.md to Korean (#24923)
* docs: ko: perf_train_cpu_many.md

* feat: chatgpt draft

* fix: manual edits

* fix: resolve suggestions

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>

---------

Co-authored-by: Jungnerd <46880056+jungnerd@users.noreply.github.com>
2023-08-09 08:15:31 +02:00
Abhipsha Das
41c5f45bfe
[DOCS] Add example for TopPLogitsWarper (#25361)
* [DOCS] Add example for `TopPLogitsWarper`

* fix typo

* address review feedback

* address review nits
2023-08-08 19:18:33 +02:00
Marc Sun
3a05e010e0
change version (#25387) 2023-08-08 13:05:41 -04:00
amyeroberts
e3490104da
Add copied from for image processor methods (#25121)
* Add copied from statements for image processors

* Move out rescale and normalize to base image processor

* Remove rescale and normalize from vit (post rebase)

* Update docstrings and tidy up

* PR comments
2023-08-08 17:02:49 +01:00
Yih-Dar
5b517e1764
Use small config for OneFormerModelTest.test_model_with_labels (#25383)
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-08 17:15:34 +02:00
Yih-Dar
9c7b744795
Fix missing usage of token (#25382)
* add missing tokens

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-08 16:27:24 +02:00
Joao Gante
5bd8c011bb
Generate: add config-level validation (#25381) 2023-08-08 13:53:03 +01:00
Yih-Dar
9e57e0c063
Fix torch_job worker(s) crashing (#25374)
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-08 14:12:56 +02:00
나건주
6247d1b2b6
🌐 [i18n-KO] Translated add_tensorflow_model.md to Korean (#25017)
* docs: ko: add_tensorflow_model.md

* feat: chatgpt draft

* fix: manual edits

* fix: manual edits

* fix: resolve suggestions

* fix: manual edits
2023-08-08 13:56:34 +02:00
Alan Ji
26ce4dd8b7
Enable tests to run on third-party devcies (#25327)
* enable unit tests to run on third-party devcies other than CUDA and CPU.

* remove the modification that enabled ut on MPS

* control test on third-party device by env variable

* update

---------

Co-authored-by: statelesshz <jihuazhong1@huawei.com>
2023-08-08 13:48:50 +02:00
Yih-Dar
5744482abc
Fix token in example template (#25351)
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-08 12:00:31 +02:00
Zach Mueller
01ab39b65f
Load state in else (#25318)
* Load else

* New approach

* Propagate
2023-08-08 05:41:00 -04:00
amyeroberts
36d5b8b06c
MaskFormer, Mask2Former - replace einsum for tracing (#25297)
* Replace einsum with ops for tracing

* Fix comment
2023-08-08 10:37:14 +01:00
Sanchit Gandhi
dedd11160d
[ASR Pipeline] Clarify return timestamps (#25344)
* [ASR Pipeline] Clarify return timestamps

* fix indentation

* fix ctc check

* fix ctc error message!

* fix test

* fix other test

* add new tests

* final comment
2023-08-08 10:16:00 +01:00
JB (Don)
5ea2595ecd
Add warning for missing attention mask when pad tokens are detected (#25345)
* Add attention mask and pad token warning to many of the models

* Remove changes under examples/research_projects

These files are not maintained by HG.

* Skip the warning check during torch.fx or JIT tracing

* Switch ordering for the warning and input shape assignment

This ordering is a little cleaner for some of the cases.

* Add missing line break in one of the files
2023-08-08 10:49:21 +02:00
Yih-Dar
6ea3ee3cd2
Fix test_model_parallelism (#25359)
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-08 10:48:45 +02:00
Matthew Hoffman
d4bd33cc9f
Register ModelOutput subclasses as supported torch.utils._pytree nodes (#25358)
* Register ModelOutput subclasses as supported torch.utils._pytree nodes

Fixes #25357 where DDP with static_graph=True does not sync gradients when calling backward() over tensors contained in ModelOutput subclasses

* Add test for torch pytree ModelOutput serialization and deserialization
2023-08-08 08:12:11 +02:00
David Reguera
a23ac36f8c
[DOCS] Add descriptive docstring to MinNewTokensLength (#25196)
* Add descriptive docstring to MinNewTokensLength

It addresses https://github.com/huggingface/transformers/issues/24783

* Refine the differences between `min_length` and `min_new_tokens`

* Remove extra line

* Remove extra arguments in generate

* Add a missing space

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Run the linter

* Add clarification comments

---------

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2023-08-08 08:09:17 +02:00
Pedro Lira
080a97119c
Add mask2former fp16 support (#25093)
* Add mask2former fp16 support

* Clear consistency/quality issues

* Fix consistency/quality (2)

* Add integration test for mask2former (fp16 case)

* Fix code quality

* Add integration test for maskformer (fp16 case)

* Add integration test for oneformer (fp16 case)

* Remove slow decorator from fp16 tests

* Fix lint

* Remove usage of full inference and value checks for fp16

* Temporarily comment slow for {mask, mask2, one}former

* Add fp16 support to oneformer

* Revert "Temporarily comment slow for {mask, mask2, one}former"

This reverts commit e5371edabd.

* Remove dtype conversion noop
2023-08-07 20:07:29 +01:00
Merve Noyan
5ee9693a1c
Docs: Added benchmarks for torch.compile() for vision models (#24748)
* added benchmarks for compile

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* added more models

* added more models fr

* added visualizations

* minor fix

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Update docs/source/en/perf_torch_compile.md

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* Added links to models and put charts side by side

* Added batch comparisons

* Added more comparisons

* Fix table

* Added link to wheel

* Update perf_torch_compile.md

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2023-08-07 17:18:43 +01:00
Rishab26
676247fd6b
[DOCS] Add NoRepeatNGramLogitsProcessor Example for LogitsProcessor class (#25186)
* Add Description And Example to Docstring

* make style corrections

* make style

* Doc Style Consistent With HF

* Apply make style

* Modify Docstring

* Edit Type in Docstring

* Feedback Incorporated

* Edit Docstring

* make style

* Post Review Changes

* Review Feedback Incorporated

* Styling

* Formatting

* make style

* pep8
2023-08-07 17:02:14 +01:00
Phuc Van Phan
5fe36970e5
Adding more information in help parser on train_file and validation_file (#25324)
chorse: adding new doc on train and val
2023-08-07 17:56:13 +02:00
Sylvain Gugger
baf1daa58e
Migrate Trainer from Repository to upload_folder (#25095)
* First draft

* Deal with progress bars

* Update src/transformers/utils/hub.py

Co-authored-by: Lucain <lucainp@gmail.com>

* Address review comments

* Forgot one

* Pin hf_hub

* Add argument for push all and fix tests

* Fix tests

* Address review comments

---------

Co-authored-by: Lucain <lucainp@gmail.com>
2023-08-07 17:47:22 +02:00
Yih-Dar
c177606fb4
Fix more offload edge cases (#25342)
* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-07 17:45:41 +02:00
Joao Gante
7d65697da7
Generate: remove Marian hack (#25294)
Remove Marian hack
2023-08-07 15:38:24 +01:00
Jackmin801
145109382a
Allow trust_remote_code in example scripts (#25248)
* pytorch examples

* pytorch mim no trainer

* cookiecutter

* flax examples

* missed line in pytorch run_glue

* tensorflow examples

* tensorflow run_clip

* tensorflow run_mlm

* tensorflow run_ner

* tensorflow run_clm

* pytorch example from_configs

* pytorch no trainer examples

* Revert "tensorflow run_clip"

This reverts commit 261f86ac1f.

* fix: duplicated argument
2023-08-07 16:32:25 +02:00
calpt
65001cb1c8
Loosen output shape restrictions on GPT-style models (#25188)
* Loosen output shape restrictions on GPT-style models

* Use more self-explanatory variables

* Revert "Use more self-explanatory variables"

This reverts commit 5fd9ab3911.
2023-08-07 16:31:15 +02:00
oobabooga
d6bfba76be
Generalize CFG to allow for positive prompts (#25339)
* Generalize CFG to allow for positive prompts

* Add documentation, fix the correct class
2023-08-07 16:25:15 +02:00
Yih-Dar
b0f23036f1
Update TF pin in docker image (#25343)
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-07 12:32:34 +02:00
Injin Paek
b9da44bd3e
🌐 [i18n-KO] Translated perf_infer_gpu_one.md to Korean (#24978)
* docs: ko: perf_infer_gpu_one

* feat: chatgpt draft

* fix: manual edits

* fix: manual edits

* fix: resolve suggestions

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: TaeYupNoh <107118671+TaeYupNoh@users.noreply.github.com>

* fix: resolve suggestions

* fix: resolve suggestions

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

---------

Co-authored-by: Sohyun Sim <96299403+sim-so@users.noreply.github.com>
Co-authored-by: TaeYupNoh <107118671+TaeYupNoh@users.noreply.github.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
2023-08-07 08:37:29 +02:00