Arthur
673440d073
update ruff version ( #30932 )
...
* update ruff version
* fix research projects
* Empty
* Fix errors
---------
Co-authored-by: Lysandre <lysandre@huggingface.co>
2024-05-22 06:40:15 +02:00
amyeroberts
57c965a8f1
Remove deprecated logic and warnings ( #30743 )
...
* Remove deprecated logic and warnings
* Add back some code that seems to be important...
* Let's just add all he nllb stuff back; removing it is a bit more involved
* Remove kwargs
* Remove more kwargs
2024-05-17 12:15:59 +01:00
Hilco van der Wilk
b6404866cd
Update legacy Repository usage in various example files ( #29085 )
...
* Update legacy Repository usage in `examples/pytorch/text-classification/run_glue_no_trainer.py`
Marked for deprecation here https://huggingface.co/docs/huggingface_hub/guides/upload#legacy-upload-files-with-git-lfs
* Fix import order
* Replace all example usage of deprecated Repository
* Fix remaining repo call and rename args variable
* Revert removing creation of gitignore files and don't change research examples
2024-03-12 13:20:49 +00:00
Klaus Hipp
39fa400969
Fix input data file extension in examples ( #28741 )
2024-01-29 10:06:31 +00:00
V.Prasanna kumar
ffbcfc0166
Broken links fixed related to datasets docs ( #27569 )
...
fixed the broken links belogs to dataset library of transformers
2023-11-17 13:44:09 -08:00
Lucain
66b088faf0
Provide alternative when warning on use_auth_token ( #27105 )
2023-10-27 14:32:54 +02:00
Tom Aarsen
40ea9ab2a1
Add many missing spaces in adjacent strings ( #26751 )
...
Add missing spaces in adjacent strings
2023-10-12 10:28:40 +02:00
Phuc Van Phan
910faa3e1f
feat: adding num_proc to load_dataset ( #26326 )
...
* feat: adding num_proc to load_dataset
* feat: add add_num_proc for run_mlm_flax
* feat: add num_proc for bart and t5
* chorse: remove
2023-09-22 19:22:47 +02:00
Phuc Van Phan
5af2c62696
docs: add space to docs ( #26067 )
...
* docs: add space to docs
* docs: remove reduntant space
2023-09-11 22:03:26 +01:00
Phuc Van Phan
9cebae64ad
docs: update link huggingface map ( #26077 )
2023-09-11 12:57:04 +01:00
Yih-Dar
149cb0cce2
Add token arugment in example scripts ( #25172 )
...
* fix
* fix
* fix
* fix
* fix
* fix
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-02 11:17:31 +02:00
Yih-Dar
d53b8ad780
Update use_auth_token -> token in example scripts ( #25167 )
...
* pytorch examples
* tensorflow examples
* flax examples
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-07-28 15:33:45 +02:00
Lucain
6232c380f2
Fix .push_to_hub and cleanup get_full_repo_name usage ( #25120 )
...
* Fix .push_to_hub and cleanup get_full_repo_name usage
* Do not rely on Python bool conversion magic
* request changes
2023-07-28 11:40:08 +02:00
Alex Punnen
805db1fe13
num_noise_spans should be <= num_items #22246 ( #22938 )
2023-05-02 13:07:30 -04:00
Aaron Gokaslan
5e8c8eb5ba
Apply ruff flake8-comprehensions ( #21694 )
2023-02-22 09:14:54 +01:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting ( #21480 )
...
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
2023-02-06 18:10:56 -05:00
Sylvain Gugger
05e72aa0c4
Adapt repository creation to latest hf_hub ( #21158 )
...
* Adapt repository creation to latest hf_hub
* Update all examples
* Fix other tests, add Flax examples
* Address review comments
2023-01-18 11:14:00 -05:00
Duong A. Nguyen
4212bb0d60
[Re-submit] Compute true loss Flax examples ( #19504 )
...
* Compute true loss
* fixup
* final
* final
* final
* Update examples/flax/language-modeling/run_bart_dlm_flax.py
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
* jax.tree_map => jax.tree_util.tree_map
* Compute true loss
* final
* fixup
* final
* final
* Update examples/flax/language-modeling/run_bart_dlm_flax.py
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
* jax.tree_map => jax.tree_util.tree_map
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
2022-10-13 11:33:36 +01:00
Kaiyu Yang
e150c4e2fe
Fix the error message in run_t5_mlm_flax.py ( #19282 )
2022-10-10 14:51:11 +01:00
Sanchit Gandhi
e6f221c8d4
[JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* ( #18361 )
...
* [JAX] Replace all jax.tree_* calls with jax.tree_util.tree_*
* fix double tree_util
2022-09-09 15:18:56 +02:00
Julien Chaumond
9129fd0377
transformers-cli login => huggingface-cli login (#18490 )
...
* zero chance anyone's using that constant no?
* `transformers-cli login` => `huggingface-cli login`
* `transformers-cli repo create` => `huggingface-cli repo create`
* `make style`
2022-08-06 09:42:55 +02:00
Duong A. Nguyen
3909d7f139
Add Flax BART pretraining script ( #18297 )
...
* add bart pretraining flax script
* fixup
* add bart pretraining flax script
* add BART to README
* add BART to README
* add BART to README
* add BART to README
* add BART to README
* add bos eos document
* Update README.md
* Update README.md
* Update examples/flax/language-modeling/run_bart_dlm_flax.py
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
* final
* final
* final
* remove use_auth_token ing from_config
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
2022-08-01 12:06:30 -04:00
Sanchit Gandhi
7490a97cac
[Flax] Fix incomplete batches in example scripts ( #17863 )
...
* [Flax] Fix incomplete batches in example scripts
* fix dataloader batching
* convert jnp batch idxs to np array
* add missing `pad_shard_unpad` to final prediction generate step
* only `pad_shard_unpad` at inference time
* merge conflicts
* remove incomplete batch step from eval
* fix run_qa.py
* add `pad_shard_unpad` to run_flax_ner.py
* add `pad_shard_unpad` to run_flax_glue.py
* add `pad_shard_unpad` to run_image_classification.py
* make style
* fix mlm flax eval batches
* remove redundant imports
2022-07-27 15:50:47 +01:00
Duong A. Nguyen
170fcaa604
Generalize decay_mask_fn to apply mask to all LayerNorm params ( #18273 )
...
* generalize decay_mask_fn to find all layernorm params
* fixup
* generalising decay_mask_fn
2022-07-27 12:23:57 +01:00
Duong A. Nguyen
4bea6584e3
Remove use_auth_token from the from_config method ( #18192 )
...
* remove use_auth_token from from_config
* restore use_auth_token from_pretrained run_t5_mlm_flax
2022-07-19 08:13:20 +02:00
Duong A. Nguyen
1e8140caad
Fix RESOURCE_EXHAUSTED error when dealing with large datasets in Flax example scripts ( #18069 )
...
* Fix RESOURCE_EXHAUSTED error for large datasets on Flax example scripts
* using np.permutation for creating batch_idx
* train_samples_idx -> training_samples_idx
* fix type hints
2022-07-11 15:59:08 +02:00
Sylvain Gugger
3cab90279f
Add examples telemetry ( #17552 )
...
* Add examples telemetry
* Alternative approach
* Add to all other examples
* Add to templates as well
* Put framework separately
* Same for TensorFlow
2022-06-07 11:57:52 -04:00
Sylvain Gugger
afe5d42d8d
Black preview ( #17217 )
...
* Black preview
* Fixup too!
* Fix check copies
* Use the same version as the CI
* Bump black
2022-05-12 16:25:55 -04:00
Ahmed Elnaggar
5e68675755
Fix t5 shard on TPU Pods ( #16527 )
...
* Fix t5 shard on TPU Pods
The current script doesn't work properly on a TPU pod because the global batch is not divided correctly per host.
This pull request fixes this issue by dividing the global batch to each host before it is shared on each host.
* fix style
Co-authored-by: ahmed-elnaggar <ahmed.elnaggar@allianz.com>
2022-04-11 16:45:20 +02:00
Karim Foda
24a85cca61
Add use_auth to load_datasets for private datasets to PT and TF examples ( #16521 )
...
* fix formatting and remove use_auth
* Add use_auth_token to Flax examples
2022-04-04 10:27:45 -04:00
Yongrae Jo
8049dfa427
Update run_t5_mlm_flax.py ( #16421 )
...
Fix typo in comment: proprocessed -> preprocessed
2022-03-28 06:00:53 -04:00
Sylvain Gugger
4975002df5
Reorganize file utils ( #16264 )
...
* Split file_utils in several submodules
* Fixes
* Add back more objects
* More fixes
* Who exactly decided to import that from there?
* Second suggestion to code with code review
* Revert wront move
* Fix imports
* Adapt all imports
* Adapt all imports everywhere
* Revert this import, will fix in a separate commit
2022-03-23 10:26:33 -04:00
Yeb Havinga
91fb62d01c
Speedup training by using numpy instead of jnp for batch shuffling ( #15963 )
...
Speedup training by using numpy instead of jnp for batch shuffling
Co-authored-by: Yeb Havinga <y.t.havinga@mgrid.net>
2022-03-08 12:18:38 +01:00
Patrick von Platen
10b76987fc
[FlaxT5 Example] fix flax t5 example pretraining ( #15835 )
2022-03-04 17:04:43 +01:00
Stas Bekman
762416ffa8
[examples/flax/language-modeling] set loglevel ( #15129 )
2022-01-13 15:17:28 +01:00
Suraj Patil
6a025487a6
[Flax examples] remove dependancy on pytorch training args ( #14636 )
...
* use custom training arguments
* update tests
2021-12-12 09:19:12 +05:30
Suraj Patil
c5bd732ac6
Add Flax example tests ( #14599 )
...
* add test for glue
* add tests for clm
* fix clm test
* add summrization tests
* more tests
* fix few tests
* add test for t5 mlm
* fix t5 mlm test
* fix tests for multi device
* cleanup
* ci job
* fix metric file name
* make t5 more robust
2021-12-06 10:48:58 +05:30
Rahul Nadkarni
8332327dca
Fix sentinel token IDs in data collator for Flax T5 pretraining script ( #14477 )
2021-11-29 17:30:17 +01:00
Nicholas Broad
69e16abf98
Switch from using sum for flattening lists of lists in group_texts ( #14472 )
...
* remove sum for list flattening
* change to chain(*)
* make chain object a list
* delete empty lines
per sgugger's suggestions
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Nicholas Broad <nicholas@nmbroad.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2021-11-22 16:17:26 -05:00
Suraj Patil
7db2a79b38
[examples/flax] use Repository API for push_to_hub ( #13672 )
...
* use Repository for push_to_hub
* update readme
* update other flax scripts
* update readme
* update qa example
* fix push_to_hub call
* fix typo
* fix more typos
* update readme
* use abosolute path to get repo name
* fix glue script
2021-09-30 16:38:07 +05:30
Patrick von Platen
2e4082364e
[Flax T5] Speed up t5 training ( #13012 )
...
* fix_torch_device_generate_test
* remove @
* update
* up
* fix
* remove f-stings
* correct readme
* up
Co-authored-by: Patrick von Platen <patrick@huggingface.co>
2021-08-06 11:21:37 +02:00
fgaim
66197adc98
Flax MLM: Allow validation split when loading dataset from local file ( #12689 )
...
* Allow validation split when loading dataset from local file
* Flax clm & t5, enable validation split for datasets loaded from local file
2021-07-20 13:38:25 +02:00
Nick Doiron
5803a2a7ac
Add ByT5 option to example run_t5_mlm_flax.py ( #12634 )
...
* Allow ByT5 type in Flax T5 script
* use T5TokenizerFast
* change up tokenizer config
* model_args
* reorder imports
* Update run_t5_mlm_flax.py
2021-07-13 13:39:57 +01:00
Patrick von Platen
deecdd4939
[Flax] Fix cur step flax examples ( #12608 )
...
* fix_torch_device_generate_test
* remove @
* fix save problem
2021-07-09 13:51:28 +01:00
Sylvain Gugger
6f1adc4334
Fix group_lengths for short datasets ( #12558 )
2021-07-08 07:23:41 -04:00
Ibraheem Moosa
122d7dc34f
Remove logging of GPU count etc logging. ( #12569 )
...
Successfully logging this requires Pytorch. For the purposes of this script we are not using Pytorch.
2021-07-07 23:05:47 +01:00
Patrick von Platen
7d321b7689
[Flax] Allow retraining from save checkpoint ( #12559 )
...
* fix_torch_device_generate_test
* remove @
* finish
2021-07-07 19:13:43 +05:30
Suraj Patil
2d42915abe
[examples/flax] add adafactor optimizer ( #12544 )
...
* add adafactor
* Update examples/flax/language-modeling/run_mlm_flax.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2021-07-07 11:50:30 +05:30
Patrick von Platen
208df208bf
[Flax] Adapt examples to be able to use eval_steps and save_steps ( #12543 )
...
* fix_torch_device_generate_test
* remove @
* up
* up
* correct
* upload
Co-authored-by: Patrick von Platen <patrick@huggingface.co>
2021-07-06 19:41:51 +01:00
Patrick von Platen
4605b2b8ec
[Flax] Fix another bug in logging steps ( #12516 )
...
* fix_torch_device_generate_test
* remove @
* up
2021-07-05 18:35:22 +01:00
