riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-03 12:50:06 +06:00
Code Issues Actions 28 Packages Projects Releases Wiki Activity
19,205 Commits 859 Branches 224 Tags 2 GiB
5399c1d670
Commit Graph

576 Commits

Author SHA1 Message Date
Lucain
f01e1609bf
Set custom_container in build docs workflows (#29855) 2024-03-26 14:46:02 +01:00
Ilyas Moutawwakil
aa17cf986f
Enable AMD docker build CI (#29803) 
* enable amd ci

* remove unnecessary clean up
 2024-03-22 11:56:47 +01:00
Yih-Dar
17e4467f0e
Fix docker image build (#29762) 
update

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2024-03-20 19:17:26 +01:00
Marc Sun
f54d82cace
[CI] Quantization workflow (#29046) 
* [CI] Quantization workflow

* build dockerfile

* fix dockerfile

* update self-cheduled.yml

* test build dockerfile on push

* fix torch install

* udapte to python 3.10

* update aqlm version

* uncomment build dockerfile

* tests if the scheduler works

* fix docker

* do not trigger on psuh again

* add additional runs

* test again

* all good

* style

* Update .github/workflows/self-scheduled.yml

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* test build dockerfile with torch 2.2.0

* fix extra

* clean

* revert changes

* Revert "revert changes"

This reverts commit 4cb52b8822.

* revert correct change

---------

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
 2024-02-28 10:09:25 -05:00
Yih-Dar
93f8617afd
Use DS_DISABLE_NINJA=1 (#29290) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2024-02-26 17:41:01 +08:00
Yih-Dar
c8d98405a8
Use torch 2.2 for daily CI (model tests) (#29208) 
* Use torch 2.2 for daily CI (model tests)

* update

* update

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2024-02-23 21:37:08 +08:00
Yuki Watanabe
ebf3ea2788
Fix a wrong link to CONTRIBUTING.md section in PR template (#28941) 2024-02-09 15:10:47 +00:00
Yih-Dar
4735866141
Split daily CI using 2 level matrix (#28773) 
* update / add new workflow files

* Add comment

* Use env.NUM_SLICES

* use scripts

* use scripts

* use scripts

* Fix

* using one script

* Fix

* remove unused file

* update

* fail-fast: false

* remove unused file

* fix

* fix

* use matrix

* inputs

* style

* update

* fix

* fix

* no model name

* add doc

* allow args

* style

* pass argument

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2024-01-31 18:04:43 +01:00
Yih-Dar
95346e9dcd
Add artifact name in job step to maintain job / artifact correspondence (#28682) 
* avoid using job name

* apply to other files

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2024-01-31 15:58:17 +01:00
Rishit Ratna
78d767e3c8
Fixed minor typos (#28489) 2024-01-15 16:45:15 +00:00
Yih-Dar
59cd9de39d
Byebye torch 1.10 (#28207) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2024-01-11 16:18:27 +01:00
Patrick von Platen
cbbe30749b
[Whisper] Fix slow test (#28407) 
* [Whisper] Fix slow test

* update

* update

* update

* update

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2024-01-10 22:35:36 +01:00
Joao Gante
ee2482b6f8
CI: limit natten version (#28432) 2024-01-10 12:39:05 +00:00
dependabot[bot]
c2d283a64a
Bump tj-actions/changed-files from 22.2 to 41 in /.github/workflows (#28311) 
Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 22.2 to 41.
- [Release notes](https://github.com/tj-actions/changed-files/releases)
- [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md)
- [Commits](https://github.com/tj-actions/changed-files/compare/v22.2...v41)

---
updated-dependencies:
- dependency-name: tj-actions/changed-files
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-01-03 09:12:53 +01:00
Yih-Dar
dde6c427a1
Fix AMD push CI not triggered (#28029) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-12-14 12:44:00 +01:00
Ella Charlaix
39acfe84ba
Add deepspeed test to amd scheduled CI (#27633) 
* add deepspeed scheduled test for amd

* fix image

* add dockerfile

* add comment

* enable tests

* trigger

* remove trigger for this branch

* trigger

* change runner env to trigger the docker build image test

* use new docker image

* remove test suffix from docker image tag

* replace test docker image with original image

* push new image

* Trigger

* add back amd tests

* fix typo

* add amd tests back

* fix

* comment until docker image build scheduled test fix

* remove deprecated deepspeed build option

* upgrade torch

* update docker & make tests pass

* Update docker/transformers-pytorch-deepspeed-amd-gpu/Dockerfile

* fix

* tmp disable test

* precompile deepspeed to avoid timeout during tests

* fix comment

* trigger deepspeed tests with new image

* comment tests

* trigger

* add sklearn dependency to fix slow tests

* enable back other tests

* final update

---------

Co-authored-by: Felix Marty <felix@hf.co>
Co-authored-by: Félix Marty <9808326+fxmarty@users.noreply.github.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-12-11 16:33:36 +01:00
Yih-Dar
0f59d2f173
Fix AMD scheduled CI not triggered (#27951) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-12-11 16:22:10 +01:00
Yih-Dar
9f1f11a2e7
Show new failing tests in a more clear way in slack report (#27881) 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-12-07 15:09:30 +01:00
Maria Khalusova
d6392482bd
removed the delete doc workflows (#27852) 2023-12-06 01:30:56 -08:00
Yih-Dar
e0d2e69582
restructure AMD scheduled CI (#27743) 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-12-04 15:32:05 +01:00
Yih-Dar
62ab32b299
Remove check_runner_status.yml (#27767) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-30 10:17:25 +01:00
Yih-Dar
7a757bb694
Fix AMD Push CI not triggered (#27732) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-28 09:30:21 +01:00
Yih-Dar
e11e26df93
Fix Past CI (#27696) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-27 09:11:58 +01:00
fxmarty
f93c1e9ece
Add RoCm scheduled CI & upgrade RoCm CI to PyTorch 2.1 (#26940) 
* add scheduled ci on amdgpu

* fix likely typo

* more tests, avoid parallelism

* precise comment

* fix report channel

* trigger docker build on this branch

* fix

* fix

* run rocm scheduled ci

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-21 14:55:13 +01:00
Yih-Dar
d903abfccc
Fix AMD CI not showing GPU (#27555) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-17 10:44:37 +01:00
Yih-Dar
acb5b4aff5
Disable docker image build job latest-pytorch-amd for now (#27541) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-16 17:00:46 +01:00
Yih-Dar
64e21ca2a4
Make some jobs run on the GitHub Actions runners (#27512) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-15 10:43:16 +01:00
Yih-Dar
00dc856233
At most 2 GPUs for CI (#27435) 
At most 2 GPUs

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-10 16:19:06 +01:00
Mert Yanık
eb30a49b20
Translate index.md to Turkish (#27093) 
* Add index.md for tukish language

* Fix index.md (huggingface/transformers#27088)

* Add 'tr' to additional files

* Update docs/source/tr/_toctree.yml

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update index.md

---------

Co-authored-by: Mert Yanık <mert.yanik@lcwaikiki.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-11-08 08:35:20 -05:00
Yih-Dar
eef7ea98c3
Update doctest workflow file (#27306) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-11-06 11:27:48 +01:00
Lysandre
bc78fd1274 Dev version 2023-11-02 18:15:36 +01:00
Yih-Dar
9dc4ce9ea7
Disable CI runner check (#27170) 
Disable runner check

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-31 11:59:21 +01:00
Aarya Balwadkar
a2f55a65cd
Hindi translation of pipeline_tutorial.md (#26837) 
* hindi translation of pipeline_tutorial.md

* Update pipeline_tutorial.md

* Update build_documentation.yml

* Update build_pr_documentation.yml

* Updated build_documentation.yml

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-10-25 11:21:49 -07:00
Akhil
093848d3cc
Added Telugu [te] translations (#26828) 
* Create index.md

* Create _toctree.yml

* Updated index.md in telugu

* Update _toctree.yml

* Create quicktour.md

* Update quicktour.md

* Create index.md

* Update quicktour.md

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Delete docs/source/hi/index.md

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/te/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update build_documentation.yml

Added telugu [te]

* Update build_pr_documentation.yml

Added Telugu [te]

* Update _toctree.yml

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2023-10-20 15:27:55 -07:00
Yih-Dar
12cc123359
Better way to run AMD CI with different flavors (#26634) 
* Enable testing against mi250

* Change BERT to trigger tests

* Revert BERT's change

* AMD CI

* AMD CI

---------

Co-authored-by: Morgan Funtowicz <funtowiczmo@gmail.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-10-16 16:24:30 +02:00
Rockerz
9b7668c03a
add japanese documentation (#26138) 
* udpaet

* update

* Update docs/source/ja/autoclass_tutorial.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* add codes workflows/build_pr_documentation.yml

* Create preprocessing.md

* added traning.md

* Create Model_sharing.md

* add quicktour.md

* new

* ll

* Create benchmark.md

* Create Tensorflow_model

* add

* add community.md

* add create_a_model

* create custom_model.md

* create_custom_tools.md

* create fast_tokenizers.md

* create

* add

* Update docs/source/ja/_toctree.yml

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* md

* add

* commit

* add

* h

* Update docs/source/ja/peft.md

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* Update docs/source/ja/_toctree.yml

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* Update docs/source/ja/_toctree.yml

Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>

* Suggested Update

* add perf_train_gpu_one.md

* added perf based MD files

* Modify toctree.yml and Add transmartion to md codes

* Add `serialization.md` and edit `_toctree.yml`

* add task summary and tasks explained

* Add and Modify files starting from T

* Add testing.md

* Create main_classes files

* delete main_classes folder

* Add toctree.yml

* Update llm_tutorail.md

* Update docs/source/ja/_toctree.yml

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update misspelled filenames

* Update docs/source/ja/_toctree.yml

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/_toctree.yml

* Update docs/source/ja/_toctree.yml

* missplled file names inmrpovements

* Update _toctree.yml

* close tip block

* close another tip block

* Update docs/source/ja/quicktour.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/pipeline_tutorial.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/pipeline_tutorial.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/preprocessing.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/peft.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/add_new_model.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/testing.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/task_summary.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/tasks_explained.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update glossary.md

* Update docs/source/ja/transformers_agents.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/llm_tutorial.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/create_a_model.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/torchscript.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/benchmarks.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/troubleshooting.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/troubleshooting.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/troubleshooting.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/ja/add_new_model.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update perf_torch_compile.md

* Update Year to default in en documentation

* Final Update

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com>
 2023-10-11 10:26:37 -07:00
Funtowicz Morgan
3632fb3c25
[AMD] Add initial version for run_tests_multi_gpu (#26346) 
* Add initial version for run_tests_multi_gpu

* Trigger change in BERT

* fix typo setup -> setup_gpu

* Add tag mi210

* Enable multi-gpu jobs

* One more

* Use dynamic device allocation

* Attempt to fix syntax for docker create

* fix script path

* fix

* temp machine type

* fix label

* Enable multi-gpu tests

* Rename multi-amd-gpu to multi-gpu

* Let's not be lazy dude

* Update rocm-smi output

* Add gpu_flavour in the matrix

* Fix typos

* merge single/multi dispatch into the matrix

* Format.

* Revert BERT's change

---------

Co-authored-by: Guillaume LEGENDRE <glegendre01@gmail.com>
 2023-10-03 11:13:45 +02:00
Yih-Dar
6ae71ec836
Update runs-on in workflow files (#26435) 
* update

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-09-27 19:25:52 +02:00
Funtowicz Morgan
2d71307dc0
Integrate AMD GPU in CI/CD environment (#26007) 
* Add a Dockerfile for PyTorch + ROCm based on official AMD released artifact

* Add a new artifact single-amdgpu testing on main

* Attempt to test the workflow without merging.

* Changed BERT to check if things are triggered

* Meet the dependencies graph on workflow

* Revert BERT changes

* Add check_runners_amdgpu to correctly mount and check availability

* Rename setup to setup_gpu for CUDA and add setup_amdgpu for AMD

* Fix all the needs.setup -> needs.setup_[gpu|amdgpu] dependencies

* Fix setup dependency graph to use check_runner_amdgpu

* Let's do the runner status check only on AMDGPU target

* Update the Dockerfile.amd to put ourselves in / rather than /var/lib

* Restore the whole setup for CUDA too.

* Let's redisable them

* Change BERT to trigger tests

* Restore BERT

* Add torchaudio with rocm 5.6 to AMD Dockerfile (#26050)

fix dockerfile

Co-authored-by: Felix Marty <felix@hf.co>

* Place AMD GPU tests in a separate workflow (correct branch) (#26105)

AMDGPU CI lives in an other workflow

* Fix invalid job name is dependencies.

* Remove tests multi-amdgpu for now.

* Use single-amdgpu

* Use --net=host for now.

* Remote host networking.

* Removed duplicated check_runners_amdgpu step

* Let's tag machine-types with mi210 for now.

* Machine type should be only mi210

* Remove unnecessary push.branches item

* Apply review suggestions moving from `x-amdgpu` to `x-gpu` introducing `amd-gpu` and `miXXX` labels.

* Remove amdgpu from step names.

* finalize

* delete

---------

Co-authored-by: fxmarty <9808326+fxmarty@users.noreply.github.com>
Co-authored-by: Felix Marty <felix@hf.co>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-09-20 14:48:49 +02:00
Ralf Müller-Zimmermann
e469be3406
Fix the gitlab user mention in issue templates to the correct user (#26237) 2023-09-19 01:49:03 +02:00
Yih-Dar
9219d1427b
pin pandas==2.0.3 (#25875) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-30 18:10:01 +02:00
Yih-Dar
62399d6f35
Add flax installation in daily doctest workflow (#25860) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-08-30 15:13:50 +02:00
Sylvain Gugger
c2123626aa
Update list of persons to tag (#25708) 2023-08-24 10:13:30 +02:00
Lysandre Debut
8657ec68fc
Sets the stalebot to 10 AM CEST (#25678) 
This sets the stale bot trigger time at 10 AM CEST rather than 5 PM CEST as all core maintainers on watch duty are now in the European timezone
 2023-08-23 14:21:07 +02:00
Yih-Dar
b413e0610b
Remove utils/documentation_tests.txt (#25680) 
* fix

* fix

* fix

* fix

* fix

* fix

* Apply suggestions from code review

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
 2023-08-23 11:14:45 +02:00
Sylvain Gugger
2defb6b048
More utils doc (#25457) 
* Document and clean more utils.

* More documentation and fixes

* Switch to Lysandre's token

* Address review comments

* Actually put else
 2023-08-17 07:58:35 +02:00
Sylvain Gugger
d561408cc3
Skip Add model like job (#24865) 2023-07-17 15:52:04 -04:00
Yih-Dar
11cb6e0f7e
Unpin DeepSpeed and require DS >= 0.9.3 (#24541) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-28 14:01:22 +02:00
Yih-Dar
e84bf1f734
⚠️ Time to say goodbye to py37 (#24091) 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-28 07:22:39 +02:00
Sylvain Gugger
eb849f6604
Migrate doc files to Markdown. (#24376) 
* Rename index.mdx to index.md

* With saved modifs

* Address review comment

* Treat all files

* .mdx -> .md

* Remove special char

* Update utils/tests_fetcher.py

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

---------

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
 2023-06-20 18:07:47 -04:00
Yih-Dar
0b259a3b7e
Clean up disk sapce during docker image build for transformers-pytorch-gpu (#24346) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-19 12:54:02 +02:00
Yih-Dar
896a58de15
Byebye pytorch 1.9 (#24080) 
byebye

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-16 16:38:23 +02:00
Mishig
ef010071ee
Update delete_doc_comment_trigger.yml (#24084) 
fix base workflow name
 2023-06-07 17:55:48 +02:00
Mishig
5c9394b54c
[doc build] Use secrets (#24079) 2023-06-07 17:33:39 +02:00
Yih-Dar
7631db0fdc
Pin deepspeed to 0.9.2 for now (#24024) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-06-05 20:00:28 +02:00
Lysandre Debut
dc67da0182
Revert "Update stale.yml to use HuggingFaceBot" (#23943) 
Revert "Update stale.yml to use HuggingFaceBot (#23941)"

This reverts commit 5929f86ebb.
 2023-06-01 11:58:11 -04:00
Lysandre Debut
5929f86ebb
Update stale.yml to use HuggingFaceBot (#23941) 2023-06-01 10:54:50 -04:00
Sourab Mangrulkar
a73b1d59a3
accelerate deepspeed and gradient accumulation integrate (#23236) 
* mixed precision support via accelerate

* fix issues

* fix for the sharded ddp case

* fix flax and tf failing tests

* `refactor the place to create `Accelerator` object

* move ddp prep to accelerate

* fix 😅

* resolving comments

* move fsdp handling to accelerate

* fixex

* fix saving

* shift torch dynamo handling to accelerate

* shift deepspeed integration and save & load utils to accelerate

* fix accelerate launcher support

* oops

* fix 🐛

* save ckpt fix

* Trigger CI

* nasty 🐛 😅

* as deepspeed needs grad_acc fixes, transfer grad_acc to accelerate

* make tests happy

* quality ✨

* loss tracked needs to account for grad_acc

* fixing the deepspeed tests

* quality ✨

* 😅😅😅

* tests 😡

* quality ✨

* Trigger CI

* resolve comments and fix the issue with the previous merge from branch

* Trigger CI

* accelerate took over deepspeed integration

---------

Co-authored-by: Stas Bekman <stas@stason.org>
 2023-05-31 15:16:22 +05:30
Yih-Dar
e69feab8a1
Update workflow files (#23658) 
* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-22 21:26:51 +02:00
Yih-Dar
9728f1134b
Fix confusing transformers installation in CI (#23465) 
* fix

* fix

* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-19 22:10:18 +02:00
Yih-Dar
1f2c00d671
Fix DeepSpeed stuff in the nightly CI (#23478) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-19 20:31:55 +02:00
Yih-Dar
22a0769933
Update 3 docker files to use cu118 (#23406) 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-17 14:26:50 +02:00
Yih-Dar
6a6225beab
Fix doctest files fetch issue (#23277) 
* fix

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-11 17:14:06 +02:00
Yih-Dar
9088fcae82
Bring back the PR Refactor doctests + add CI to main (#23271) 
* Revert "Revert "[Doctests] Refactor doctests + add CI" (#23245)"

This reverts commit 69ee46243c.

* try not expose HfDocTestParser

* move into testing_utils.py

* remove pytest install

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-10 22:00:48 +02:00
Sylvain Gugger
69ee46243c
Revert "[Doctests] Refactor doctests + add CI" (#23245) 
Revert "[Doctests] Refactor doctests + add CI (#22987)"

This reverts commit 627f44799a.
 2023-05-09 15:26:15 -04:00
Arthur
627f44799a
[Doctests] Refactor doctests + add CI (#22987) 
* intiial commit

* new styling

* update

* just run doctest in CI

* remove more test for fast dev

* update

* update refs

* update path and fetch upstream

* update documentatyion trests

* typo

* parse pwd

* don't check for files that are in hidden folders

* just give paths relative to transformers

* update

* update

* update

* major refactoring

* make sure options is ok

* lest test that mdx is tested

* doctest glob

* nits

* update doctest nightly

* some cleaning

* run correct test on diff

* debug

* run on a single worker

* skip_cuda_test tampkate

* updates

* add rA and continue on failure

* test options

* parse `py` codeblock?

* we don't need to replace ignore results, don't remember whyu I put it

* cleanup

* more cleaning

* fix arg

* more cleaning

* clean an todo

* more pre-processing

* doctest-module has none so extra `- ` is needed

* remove logs

* nits

* doctest-modules ....

* oups

* let's use sugar

* make dataset go quiet

* add proper timeout

* nites

* spleling timeout

* update

* properly skip tests that have CUDSA

* proper skipping

* cleaning main and get tests to run

* remove make report?

* remove tee

* some updates

* tee was removed but is the full output still available?

* [all-test]

* only our tests

* don't  touch tee in this PR

* no atee-sys

* proper sub

* monkey

* only replace call

* fix sub

* nits

* nits

* fix invalid syntax

* add skip cuda doctest env variable

* make sure all packages are installed

* move file

* update check repo

* revert changes

* nit

* finish cleanup

* fix re

* findall

* update don't test init files

* ignore pycache

* `-ignore-pycache` when running pytests

* try to fix the import missmatch error

* install dec

* pytest is required as doctest_utils imports things from it

* the only log issues were dataset, ignore results should work

* more cleaning

* Update .circleci/create_circleci_config.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

* [ydshieh] empty string if cuda is found

* [ydshieh] fix condition

* style

* [ydshieh] fix

* Add comment

* style

* style

* show failure

* trigger CI

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Yih-Dar <2521628+ydshieh@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-05-09 20:34:48 +02:00
Yih-Dar
073baf7f22
Install accelerete@main in PyTorch Past CI jobs (#22963) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-24 21:19:06 +02:00
Yih-Dar
aa4316757d
Change schedule CI time (#22884) 
* fix

* Update .github/workflows/self-nightly-past-ci-caller.yml

Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
 2023-04-20 14:01:08 +02:00
Yih-Dar
648bd5a8aa
Show diff between 2 CI runs on Slack reports (#22798) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-19 19:27:37 +02:00
Yih-Dar
1ebc1dee92
Fix Past CI not running against the latest main (#22823) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-18 15:41:41 +02:00
Sylvain Gugger
e13d6ef7dc
Simplify update metadata job (#22811) 
* Simplify update metadata job

* Match more branch names

* Install all what is necessary

* Install all what is necessary

* Forgot the dev

* Install less stuff

* This syntax?
 2023-04-17 13:54:20 -04:00
Yih-Dar
656d41ab4c
Remove DS_BUILD_AIO=1 (#22741) 
fix

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-13 18:08:22 +02:00
Yih-Dar
2c22bc79c2
Make tiny model creation + pipeline testing more robust (#22500) 
* Final Tiny things

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-04-06 17:45:55 +02:00
Yih-Dar
0fe6c6bdca
(Re-)Enable Nightly + Past CI (#22393) 
* Enable Nightly + Past CI

* put schedule

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-30 21:06:35 +02:00
Yih-Dar
97440e9c75
Avoid using personal HF token in CI (#22453) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-29 19:45:06 +02:00
Yih-Dar
8894b81742
Use real tokenizers if tiny version(s) creation has issue(s) (#22428) 
Fix some tiny model creation issues

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-29 16:16:23 +02:00
Yih-Dar
e8cc02555e
Automatically create/update tiny models (#22275) 
* Automatically create or update tiny models

* Skip failed tests

* update workflow file

* use revision

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-23 19:14:17 +01:00
Yih-Dar
ba9e0191de
Prepare daily CI for torch 2.0.0 (#22135) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-03-13 22:21:15 +01:00
Yih-Dar
aab895c396
Make Slack CI reporting stronger (#21823) 
* Use token

* Avoid failure

* better error

* Fix

* fix style

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-02-28 17:12:44 +01:00
Yih-Dar
ff143ae10e
Update doctest GH workflow file (#21744) 
update

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-02-23 09:40:53 +01:00
Sylvain Gugger
22888d3082
Remove Niels from templates (#21564) 2023-02-14 09:47:43 -05:00
Zachary Mueller
8c5026628a
Add in big model inference to issue template (#21611) 
* Add in big model inference to issue template

* Trigger

* Untrigger

* empty test commit
 2023-02-13 16:40:34 -05:00
Nolwenn Bernard
baf4bacb1f
[i18n-fr] Translate index page to French (#21458) 
* Translate index page to French

* Fix indent

* Fix toctree

* Replace missing file by in_translation

* Add index

* Update docs/source/fr/index.mdx

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

---------

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2023-02-06 12:25:49 -05:00
Stas Bekman
6d67664380
[issues template] update deepspeed owners (#21027) 
* [issues template] update deepspeed owners

add the right contact for deepspeed@accelerate

* pr-template
 2023-01-18 17:23:36 -08:00
Clémentine Fourrier
4bbbabcb2c
Added clefourrier as ref point for graph models in bug reports (#21139) 
* Added clefourrier as ref point for graph models in bug reports

* Update PULL_REQUEST_TEMPLATE.md
 2023-01-16 15:12:42 +01:00
Yih-Dar
94db82573e
Fix (DeepSpeed) docker image build issue (#21002) 
* Fix docker image build issue

* remove comment

* Add comment

* Update docker/transformers-pytorch-deepspeed-latest-gpu/Dockerfile

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
 2023-01-04 21:28:33 +01:00
Steven Liu
d53f329d88
Update PR template (#21006) 
add maria to pr template
 2023-01-04 11:01:52 -08:00
Steven Liu
9dcc881fa6
Update bug report template (#21004) 
add maria to bug report
 2023-01-04 10:33:15 -08:00
Sylvain Gugger
c83703cbdb
Update the list of contributors to reflect current organization (#20603) 
* Update the list of contributors to reflect current organization

* Proper indent
 2022-12-08 10:05:43 -05:00
Yih-Dar
bf9a5882a7
Update some GH action versions (#20537) 
* update actions versions

* update actions versions

* update actions versions

* update actions versions

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-12-06 16:54:40 +01:00
Yih-Dar
67d32f4649
Replace set-output by $GITHUB_OUTPUT (#20547) 
* remove set-output

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-12-05 18:25:13 +01:00
Yih-Dar
373bfe70a0
Change Doctests CI launch time (#20523) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-12-01 16:38:41 +01:00
Yih-Dar
e8d448edcf
extract warnings in GH workflows (#20487) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-11-29 15:58:54 +01:00
BFSS
05d80d856c
translate zh quicktour(#20095) (#20181) 
* zh quicktour(#20095)

* add zh to doc workflow

* remove untranslation from toctree

Co-authored-by: BeifangSusu <BeifangSusu@bfss.com>
 2022-11-21 08:44:18 -05:00
Yih-Dar
700e0cd65f
Add missing report button for Example test (#20293) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-11-17 15:55:00 +01:00
Mishig
e627e9b5ae
Complete doc migration (#20267) 2022-11-16 08:43:37 -05:00
Wonhyeong Seo
7ecb039176
feat: add i18n issue template (#20199) 
Part of #20183
docs: add relevant labels to i18n issue template
fix: typo on completion count
 2022-11-14 12:36:58 -05:00
Wonhyeong Seo
07d8d6e2f7
docs: translated index page to korean (#20180) 
docs: i18n: first draft of index page
docs: fix: first revision of index page
docs: i18n: missed section - supported frameworks
docs: fix: second revision of index page
review by @ArthurZucker

refactor: remove untranslated files from korean
docs: fix: remove untranslated references from toctree.yml
feat: enable korean docs in gh actions
docs: feat: add in_translation page as placeholder
docs: bug: testing if internal toc need alphabet chars
docs: fix: custom english anchor for non-alphanumeric headings
review by @sgugger

docs: i18n: translate comments on install methods in _config.py
docs: refactor: more concise wording for translations
 2022-11-14 12:09:21 -05:00
Mishig
e2a23b6ce9
Update github pr docs actions (#20125) 2022-11-08 10:37:24 -05:00
Yih-Dar
c06d555647
Show installed libraries and their versions in GA jobs (#20069) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-11-04 18:03:18 +01:00
Yih-Dar
12ce2941c7
Fix docker image build (#19759) 
* Use 2 jobs for the docker image build (latest torch + DS)

* fix

* Add comment

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-10-20 20:36:13 +02:00
Yih-Dar
5418e3cef0
Build Push CI images also in a daily basis (#19532) 
* Build Push CI images also in a daily basis

* update

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-10-13 07:31:12 +02:00
Yih-Dar
e81cb010f8
Avoid Push CI failing to report due to many commits being merged (#19496) 
* Change the depth to 20

* Add comment

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-10-12 09:25:05 +02:00
Yih-Dar
8d68878cc0
python3 instead of python in push CI setup job (#19492) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-10-11 19:18:39 +02:00
Sylvain Gugger
9ac586b3c8
Rework pipeline tests (#19366) 
* Rework pipeline tests

* Try to fix Flax tests

* Try to put it before

* Use a new decorator instead

* Remove ignore marker since it doesn't work

* Filter pipeline tests

* Woopsie

* Use the fitlered list

* Clean up and fake modif

* Remove init

* Revert fake modif
 2022-10-07 18:01:58 -04:00
Sushrut1101
6dce9e0cdd
docker-build: Update actions/checkout to v3 (#19288) 2022-10-04 16:26:52 +02:00
Oscar Dominguez
cd024da6f8
ci(workflows): update actions/checkout to v3 (#19280) 
in stale.yml
 2022-10-04 16:07:53 +02:00
Oscar Dominguez
ca3ebc44e0
ci(stale.yml): upgrade actions/setup-python to v4 (#19281) 2022-10-04 16:07:33 +02:00
Sylvain Gugger
18c06208c4
Don't automatically add bug label (#19302) 2022-10-03 12:42:04 -04:00
Yih-Dar
163cd15279
Add job names in Past CI artifacts (#19235) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-29 19:18:24 +02:00
Yih-Dar
99c32493e0
Fix confusing working directory in Push CI (#19234) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-29 08:36:46 +02:00
Yih-Dar
71fc331746
Separate Push CI images from Scheduled CI (#19170) 
* separate images

* Fix condition

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-26 10:55:42 +02:00
Yih-Dar
0cea8d5555
Add offline runners info in the Slack report (#19169) 
* send slack report for offline runners

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-23 19:23:05 +02:00
Yih-Dar
cc567e0063
Fix the wrong schedule (#19117) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-20 13:46:55 +02:00
Yih-Dar
22264f933d
fix working dir (#19101) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-19 07:09:24 -04:00
Yih-Dar
ba7f2173cc
Add runner availability check (#19054) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-19 12:27:06 +02:00
Yih-Dar
7a8118947f
Add checks for more workflow jobs (#18905) 
* add check for scheduled CI

* Add check to other CIs

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-07 12:51:37 +02:00
Yih-Dar
7d5fde991d
unpin slack_sdk version (#18901) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-06 18:42:00 +02:00
Yih-Dar
ecdf9b06bc
Remove cached torch_extensions on CI runners (#18868) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-02 18:17:58 +02:00
Yih-Dar
0ab465a5d2
pin Slack SDK to 3.18.1 to avoid failing issue (#18869) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-09-02 16:49:08 +02:00
Lysandre Debut
ee407024c4
Update location identification (#18834) 2022-08-31 15:10:25 +02:00
Yih-Dar
e88e9ff045
Disable nightly CI temporarily (#18820) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-08-30 18:33:09 +02:00
Yih-Dar
81ab11124f
Add checks for some workflow jobs (#18583) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-08-16 13:53:47 +02:00
flozi00
5d3f037433
german docs translation (#18544) 
* Create _config.py

* Create _toctree.yml

* Create index.mdx

not sure about "du / ihr" oder "sie"

* Create quicktour.mdx

* Update _toctree.yml

* Update build_documentation.yml

* Update build_pr_documentation.yml

* fix build

* Update index.mdx

* Update quicktour.mdx

* Create installation.mdx

* Update _toctree.yml
 2022-08-11 09:52:27 -04:00
Yih-Dar
d2704c4143
Add machine type in the artifact of Examples directory job (#18459) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-08-04 18:52:01 +02:00
Yih-Dar
c4c6b4dbda
Add PyTorch 1.11 to past CI (#18302) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-07-26 15:47:23 +02:00
Yih-Dar
2c5747edfe
Update notification service (#17921) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-07-21 15:03:50 +02:00
Yih-Dar
f681437203
Enable Past CI (#17919) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-07-05 18:08:36 +02:00
NielsRogge
e113c5cb64
Remove render tags (#17897) 
Co-authored-by: Niels Rogge <nielsrogge@Nielss-MacBook-Pro.local>
 2022-06-29 06:06:42 -04:00
Santiago Castro
90415475bb
Fix the Conda package build (#16737) 
* Fix the Conda package build

* Update build.sh

* Update release-conda.yml
 2022-06-29 06:03:16 -04:00
Zachary Mueller
893ab12452
Auto-build Docker images before on-merge if setup.py was changed (#17573) 
* Auto-build on setup modification

* Modify push-caller

* Make adjustments based on code review
 2022-06-23 16:51:33 -04:00
Yih-Dar
8d634b70e0
Fix push CI artifact path (#17788) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-06-23 12:31:22 +02:00
Joao Gante
0d0c392c45
CLI: use hub's create_commit (#17755) 
* use create_commit

* better commit message and description

* touch setup.py to trigger cache update

* add hub version gating
 2022-06-22 16:50:21 +01:00
Yih-Dar
6589e510fa
Attempt to change Push CI to workflow_run (#17753) 
* Use workflow_run event for push CI

* change to workflow_run

* Add comments

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-06-18 08:35:03 +02:00
Yih-Dar
ca169dbdf1
Enable PyTorch nightly build CI (#17335) 
* nightly build pytorch CI

* fix working dir

* change time and event name

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-06-17 16:42:27 +02:00
Yih-Dar
50415b84d6
Revert "Change push CI to run on workflow_run event (#17692)" (#17717) 
This reverts commit b76290f44c.
 2022-06-15 18:42:43 +02:00
Yih-Dar
b76290f44c
Change push CI to run on workflow_run event (#17692) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-06-15 17:43:31 +02:00
Yih-Dar
da0bed5f4a
Pre-build DeepSpeed (#17607) 
* pre-build deepspeed

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-06-09 23:02:33 +02:00
Yih-Dar
9aa230aa2f
Use latest stable PyTorch/DeepSpeed for Push & Scheduled CI (#17417) 
* update versions

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-06-07 11:53:05 +02:00
Yih-Dar
659b27fd26
Print more library versions in CI (#17384) 
* print more lib. versions and just befor test runs

* update print_env_pt.py

* rename to print_env

* Disable warning + better job name

* print python version

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-06-02 10:24:16 +02:00
Yih-Dar
0932adb3e8
Split push CI into 2 workflows (#17369) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-06-02 10:19:26 +02:00
Patrick von Platen
52e7c92920
Add HF.co for PRs / Issues regarding specific model checkpoints (#17485) 
* Add HF.co for PRs / Issues regarding specific model checkpoints

* Update .github/ISSUE_TEMPLATE/config.yml

Co-authored-by: Julien Chaumond <julien@huggingface.co>

Co-authored-by: Julien Chaumond <julien@huggingface.co>
 2022-05-31 15:58:39 +02:00
Martina Fumanelli
dfc38463b8
Setup for Italian translation and add quicktour.mdx translation (#17472) 
* Setup for Italian translation and add first document

- Add 'it' folder for files translated into Italian
- Add _config.py and _toctree.yml files
- Add translation of quicktour.mdx

* Fix style issue of italian documentation files

* Add 'it' to the languages section in the .github/workflows

* Remove - installation from _toctree for Italian

* Translation for index file

- Add index to _toctree.yml
- Add translation of index.mdx

* Fix typo in docs/source/it/index.mdx

* Translate code comments in docs/source/it/_config.py

Co-authored-by: Martina Fumanelli <martinafumanelli@Martinas-MBP.homenet.telecomitalia.it>
 2022-05-31 09:57:43 -04:00
Yih-Dar
400b30936a
Docker image build in parallel (#17434) 
* docker image build in parallel

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-31 15:39:03 +02:00
Yih-Dar
1ac2b8fa7f
Fix missing job action button in CI report (#17270) 
* use matrix.machine_type

* fix job names used in job_link

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-17 08:31:06 +02:00
Yih-Dar
50d1867cf8
Add PR title to push CI report (#17246) 
* add PR title to push CI report

* add link

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-13 21:50:40 +02:00
Yih-Dar
506899d147
Fix push CI channel (#17242) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-13 20:59:56 +02:00
Yih-Dar
993553b2f1
fix --gpus option for docker (#17235) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-13 17:26:26 +02:00
Yih-Dar
38043d8453
Update self-push workflow (#17177) 
* update push ci

* install git-python

* update comment

* update deepspeed jobs

* fix report

* skip 2 more tests that require fairscale

* Fix changes in test_fetcher.py (to deal with `setup.py` is changed)

* set RUN_PT_TF_CROSS_TESTS=1 and final clean-up

* remove SIGOPT_API_TOKEN

* remove echo "$matrix_folders"

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-13 16:28:00 +02:00
Rafael Zimmer
85fc455972
Added translation of installation.mdx to Portuguese Issue #16824 (#16979) 
* Added translation of installation.mdx to Portuguese, as well
as default templates of _toctree.yml and _config.py

* [ build_documentation.yml ] - Updated doc_builder to build
documentation in Portuguese.
[ pipeline_tutorial.mdx ] - Created translation for the pipeline_tutorial.mdx.

* [ build_pr_documentation.yml ] - Added pt language to pr_documentation builder.

[ pipeline_tutorial.mdx ] - Grammar changes.

* [ accelerate.mdx ] - Translated to Portuguese the acceleration tutorial.

* [ multilingual.mdx ] - Added portuguese translation for multilingual tutorial.

[ training.mdx ] - Added portuguese translation for training tutorial.

* [ preprocessing.mdx ] - WIP

* Update _toctree.yml

* Adding Pré-processamento to _toctree.yml

* Update accelerate.mdx

* Nits and eliminate preprocessing file while it is ready

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>
 2022-05-13 07:55:44 -04:00
fxmarty
3f936df662
Fix typo in bug report template (#17178) 
* Fix typo

* Force rerun workflows

Co-authored-by: Felix Marty <felix@huggingface.co>
 2022-05-12 16:31:12 -04:00
Yih-Dar
3212afa614
split single_gpu and multi_gpu (#17083) 
* split single_gpu and multi_gpu

* update needs in send_result

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-09 07:13:07 -04:00
Yih-Dar
351cdbdfdc
Fix self-push CI report path in cat (#17111) 
* fix report cat path

* fix report cat path

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-06 07:45:17 -07:00
Sylvain Gugger
dd16a113a4
Remove torchhub test (#17097) 2022-05-05 10:02:47 -04:00
Sylvain Gugger
bb8d40529e
Deprecate model templates (#17062) 
* Deprecate model templates

* Address review comments
 2022-05-04 09:36:38 -04:00
Sylvain Gugger
dd739f7045 Remove fetch in model templates test 2022-05-03 10:49:12 -04:00
Sylvain Gugger
6e17ba6aa5
Remove Python and use v2 action (#17059) 2022-05-03 10:12:17 -04:00
Yih-Dar
19420fd99e
Move test model folders (#17034) 
* move test model folders (TODO: fix imports and others)

* fix (potentially partially) imports (in model test modules)

* fix (potentially partially) imports (in tokenization test modules)

* fix (potentially partially) imports (in feature extraction test modules)

* fix import utils.test_modeling_tf_core

* fix path ../fixtures/

* fix imports about generation.test_generation_flax_utils

* fix more imports

* fix fixture path

* fix get_test_dir

* update module_to_test_file

* fix get_tests_dir from wrong transformers.utils

* update config.yml (CircleCI)

* fix style

* remove missing imports

* update new model script

* update check_repo

* update SPECIAL_MODULE_TO_TEST_MAP

* fix style

* add __init__

* update self-scheduled

* fix add_new_model scripts

* check one way to get location back

* python setup.py build install

* fix import in test auto

* update self-scheduled.yml

* update slack notification script

* Add comments about artifact names

* fix for yolos

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-05-03 14:42:02 +02:00
Sylvain Gugger
4be8b95a9f
Disable Flax GPU tests on push (#17042) 2022-05-02 10:25:53 -04:00
Yih-Dar
992996e9ca
Add -e flag to some GH workflow yml files (#16959) 
* Add -e flag

* add check

* create new keys

* run python setup.py build install

* add comments

* change to develop

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-04-27 21:44:21 +02:00
Lysandre Debut
c11a49573f
Refactor issues with yaml (#16772) 
* Refactor issues with yaml

* Update .github/ISSUE_TEMPLATE/bug-report.yml

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

* Update .github/ISSUE_TEMPLATE/bug-report.yml

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>

* Update .github/ISSUE_TEMPLATE/feature-request.yml

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* Update .github/ISSUE_TEMPLATE/bug-report.yml

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update .github/ISSUE_TEMPLATE/bug-report.yml

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Address review comments

Co-authored-by: Anton Lozhkov <aglozhkov@gmail.com>
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
 2022-04-18 16:43:21 -04:00
Sylvain Gugger
3785f4665a Fix syntax error in TorchHub workflow 2022-04-18 07:54:00 -04:00
Joao Gante
6984848ed0
Create empty venv on cache miss (#16816) 2022-04-18 07:49:31 -04:00
Joao Gante
02de7a8e7f
CI: non-remote GH Actions now use a python venv (#16789) 2022-04-18 09:47:38 +01:00
Stas Bekman
ce2fef2ad2
[trainer / deepspeed] fix hyperparameter_search (#16740) 
* [trainer / deepspeed] fix hyperparameter_search

* require optuna

* style

* oops

* add dep in the right place

* create deepspeed-testing dep group

* Trigger CI
 2022-04-14 17:24:38 -07:00
Stas Bekman
3a16ab25c8
[self-scheduled ci] explain where dependencies are (#16757) 2022-04-13 12:28:02 -04:00
Joao Gante
0b8f697219
CI: setup-dependent pip cache (#16751) 
* Setup-dependent pip cache

* Do not restore from old versions
 2022-04-13 16:19:14 +01:00
Sylvain Gugger
b9a768b3ff
Enable doc in Spanish (#16518) 
* Reorganize doc for multilingual support

* Fix style

* Style

* Toc trees

* Adapt templates
 2022-04-04 10:25:46 -04:00
Lysandre Debut
eca77f4719
Updates the default branch from master to main (#16326) 
* Updates the default branch from master to main

* Links from `master` to `main`

* Typo

* Update examples/flax/README.md

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-03-23 03:46:59 -04:00
Patrick von Platen
4f6c938342
[Bug template] Shift responsibilities for long-range (#16344) 2022-03-22 21:55:22 +01:00
Patrick von Platen
c1af180dfe
Add Slack notification support for doc tests (#16253) 
* up

* up

* up

* fix

* yeh

* ups

* Empty test commit

* correct quicktour

* correct

* correct

* up

* up

* uP

* uP

* up

* up

* uP

* up

* up

* up

* up

* up

* up

* up

* up

* up

* up

* Update src/transformers/models/van/modeling_van.py

* finish

* apply suggestions

* remove folder

* revert to daily testing
 2022-03-21 11:33:18 +01:00
Yih-Dar
09013efdf1
Update step name (#16189) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-03-16 11:19:38 -04:00
Sylvain Gugger
8bfd2fb8f0
Use templates (#16142) 
* Use tempaltes for all doc building jobs

* Add this branch to the doc build

* Switch to main branch
 2022-03-15 08:07:56 -04:00
Patrick von Platen
5664d27622
Shift responsibilities a bit (#16154) 2022-03-15 11:07:17 +01:00
Lysandre Debut
a01fe4cd32
Rebuild deepspeed (#16081) 
* Rebuild deepspeed

* Apply suggestions from code review

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
 2022-03-11 14:35:48 -05:00
Patrick von Platen
322c8533d7
Run daily test without time-out at least once (#16077) 2022-03-11 18:04:17 +01:00
Sylvain Gugger
10591399d6
Build the doc in a seperate folder then move it (#16020) 
* Build the doc in a seperate folder then move it

* Allow job

* Is this it?

* Dislike comments?

* Copy instead of move

* Removing version built

* Typos

* No variable

* Take _versions.yml into account

* Finish main job and add dev job

* Forgot the run

* Fix syntax error

* Execute builder from the repo

* Typo
 2022-03-10 07:44:29 -05:00
Lysandre Debut
e493a3a5e2
Fix github actions comment (#16009) 
* Add issue number

* Dev
 2022-03-09 08:39:03 -05:00
Patrick von Platen
c1aaa43935
[Doctests] Move doctests to new GPU & Fix bugs (#15969) 
* test

* up

* up

* Empty test commit

* up

* update tests

* up

* fix some vision models

* correct

* correct docs

* Trigger notification

* finalize

* check

* correct quicktour

* Apply suggestions from code review

* improve doctests

* Trigger Build

* next try

* next try

* and again

* Output current clone information

* Output current clone information

* Correct path

* add tf round again

* revert to daily job

Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
 2022-03-09 13:09:56 +01:00
Sylvain Gugger
f5a080dd10
Do a pull in case docs were updated during build (#15922) 2022-03-08 07:19:41 -05:00
Sylvain Gugger
3d2242869d
Update delete-dev-doc job to match build-dev-doc (#15891) 
* Update delete-dev-doc job to match build-dev-doc

* More debug info

* More debug info

* Stash if needed

* Remove the comment update

* Fix paths

* Wtf is going on..

* Fix git status test

* Try another way

* I don't understand what's happening

* Bash shell

* What's happening now...

* What's happening now...

* Try like this

* Back to trying to use bash

* And like that?

* Refine tests

* Stash after adding new files

* Stash after adding new files

* Proper commit sha and PR number

* Address review comments
 2022-03-02 16:18:54 -05:00
Lysandre Debut
d1a29078c0
Remove stash for now (#15882) 2022-03-01 22:36:19 -05:00
Lysandre Debut
26426923b7
No self-hosted runner for dev documentation (#15710) 2022-03-01 14:05:54 -05:00
Mishig Davaadorj
00eaffc81f
Bump up doc node version to 16 (#15874) 2022-03-01 18:37:57 +01:00
Suraj Patil
afca0d5192
use python 3.7 for flax self-push tests (#15865) 
* set python 3.7 for flax tests

* setup-python@v2

* python-dev

* install -y

* python3-dev

* install kenlm from source

* install cython

* cd to kenlm

* kenlm install

* don't install kenlm

* change flax pretrained to run flax tests

* cleanup

* remove python-dev
 2022-03-01 18:26:30 +01:00
Lysandre Debut
54f0db4066
Add PT + TF automatic builds (#15860) 
* Add PT + TF automatic builds

* Apply suggestions from code review

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Wrap up

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
 2022-03-01 08:55:11 -05:00
Lysandre Debut
b7e292aebd
Fix the push run (#15807) 2022-02-24 19:30:17 +01:00
Lysandre Debut
bb7949b35a
Fix model templates (#15806) 
* Fix model templates

* Update paths
 2022-02-23 18:27:29 -05:00
Lysandre
309e87e25e Docker images should only run on a daily basis 2022-02-23 18:01:44 -05:00
Lysandre
c475f3ce2d Scheduled tests should only run on a daily basis 2022-02-23 17:52:22 -05:00
Eliott C
6336017c15
Fix build_documentation CI (#15803) 2022-02-23 21:53:51 +01:00
Lysandre Debut
a0e3480699
[Test refactor 5/5] Build docker images (#15729) 2022-02-23 15:48:19 -05:00
Lysandre Debut
4c737f0e40
[Test refactor 4/5] Improve the scheduled tests (#15728) 2022-02-23 15:48:05 -05:00
Lysandre Debut
29c10a41d0
[Test refactor 1/5] Per-folder tests reorganization (#15725) 
* Per-folder tests reorganization

Co-authored-by: sgugger <sylvain.gugger@gmail.com>
Co-authored-by: Stas Bekman <stas@stason.org>
 2022-02-23 15:46:28 -05:00
Eliott C
86636f52a9
Fix indent in doc-builder CI (#15798) 2022-02-23 20:01:33 +01:00
Eliott C
a1efc82362
HTML dev docs (#15678) 
Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>
 2022-02-23 19:43:22 +01:00
Eliott C
bc3379e12c
🔥 Remove build_doc_test github action (#15680) 2022-02-16 14:06:26 +01:00
Lysandre Debut
7029240927
Upgrade click version (#15579) 2022-02-09 10:28:43 -05:00
Patrick von Platen
e2b6e73fa2
[Flax tests] Disable scheduled GPU tests (#15503) 2022-02-03 17:12:14 +01:00
Ayush Chaurasia
c74f3d4c48
Add W&B backend for hyperparameter sweep (#14582) 
# Add support for W&B hyperparameter sweep
This PR:
* allows using wandb for running hyperparameter search.
* The runs are visualized on W&B sweeps dashboard
* This supports runnning sweeps on parallel devices, all reporting to the same central dashboard.

### Usage
**To run new a hyperparameter search:**
```
trainer.hyperparameter_search(
    backend="wandb", 
    project="transformers_sweep", # name of the project
    n_trials=5,
    metric="eval/loss", # metric to be optimized, default 'eval/loss'. A warning is raised if the passed metric is not found
)
```
This outputs a sweep id. Eg. `my_project/sweep_id`

**To run sweeps on parallel devices:**
Just pass sweep id which you want to run parallel
```
trainer.hyperparameter_search(
    backend="wandb", 
    sweep_id = "my_project/sweep_id"
)
```
 2022-02-02 14:06:14 -05:00