zspo
d98591a12b
[docs] fix some bugs about parameter description ( #28806 )
...
Co-authored-by: p_spozzhang <p_spozzhang@tencent.com>
2024-02-01 16:59:29 +00:00
Alex Hedges
95091e1582
Set cache_dir
for evaluate.load()
in example scripts ( #28422 )
...
While using `run_clm.py`,[^1] I noticed that some files were being added
to my global cache, not the local cache. I set the `cache_dir` parameter
for the one call to `evaluate.load()`, which partially solved the
problem. I figured that while I was fixing the one script upstream, I
might as well fix the problem in all other example scripts that I could.
There are still some files being added to my global cache, but this
appears to be a bug in `evaluate` itself. This commit at least moves
some of the files into the local cache, which is better than before.
To create this PR, I made the following regex-based transformation:
`evaluate\.load\((.*?)\)` -> `evaluate\.load\($1,
cache_dir=model_args.cache_dir\)`. After using that, I manually fixed
all modified files with `ruff` serving as useful guidance. During the
process, I removed one existing usage of the `cache_dir` parameter in a
script that did not have a corresponding `--cache-dir` argument
declared.
[^1]: I specifically used `pytorch/language-modeling/run_clm.py` from
v4.34.1 of the library. For the original code, see the following URL:
acc394c4f5/examples/pytorch/language-modeling/run_clm.py
.
2024-01-11 15:38:44 +01:00
V.Prasanna kumar
ffbcfc0166
Broken links fixed related to datasets docs ( #27569 )
...
fixed the broken links belogs to dataset library of transformers
2023-11-17 13:44:09 -08:00
Matt
2e72bbab2c
Incorrect setting for num_beams in translation and summarization examples ( #27519 )
...
* Remove the torch main_process_first context manager from TF examples
* Correctly set num_beams=1 in our examples, and add a guard in GenerationConfig.validate()
* Update src/transformers/generation/configuration_utils.py
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
2023-11-15 18:18:54 +00:00
Lucain
66b088faf0
Provide alternative when warning on use_auth_token ( #27105 )
2023-10-27 14:32:54 +02:00
Tom Aarsen
40ea9ab2a1
Add many missing spaces in adjacent strings ( #26751 )
...
Add missing spaces in adjacent strings
2023-10-12 10:28:40 +02:00
Roy Hvaara
fc63914399
[JAX] Replace uses of jnp.array
in types with jnp.ndarray
. ( #26703 )
...
`jnp.array` is a function, not a type:
https://jax.readthedocs.io/en/latest/_autosummary/jax.numpy.array.html
so it never makes sense to use `jnp.array` in a type annotation. Presumably the intent was to write `jnp.ndarray` aka `jax.Array`.
Co-authored-by: Peter Hawkins <phawkins@google.com>
2023-10-10 21:35:16 +02:00
Phuc Van Phan
5af2c62696
docs: add space to docs ( #26067 )
...
* docs: add space to docs
* docs: remove reduntant space
2023-09-11 22:03:26 +01:00
Jackmin801
145109382a
Allow trust_remote_code
in example scripts ( #25248 )
...
* pytorch examples
* pytorch mim no trainer
* cookiecutter
* flax examples
* missed line in pytorch run_glue
* tensorflow examples
* tensorflow run_clip
* tensorflow run_mlm
* tensorflow run_ner
* tensorflow run_clm
* pytorch example from_configs
* pytorch no trainer examples
* Revert "tensorflow run_clip"
This reverts commit 261f86ac1f
.
* fix: duplicated argument
2023-08-07 16:32:25 +02:00
Yih-Dar
149cb0cce2
Add token
arugment in example scripts ( #25172 )
...
* fix
* fix
* fix
* fix
* fix
* fix
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-02 11:17:31 +02:00
Yih-Dar
d53b8ad780
Update use_auth_token
-> token
in example scripts ( #25167 )
...
* pytorch examples
* tensorflow examples
* flax examples
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-07-28 15:33:45 +02:00
Lucain
6232c380f2
Fix .push_to_hub
and cleanup get_full_repo_name
usage ( #25120 )
...
* Fix .push_to_hub and cleanup get_full_repo_name usage
* Do not rely on Python bool conversion magic
* request changes
2023-07-28 11:40:08 +02:00
Connor Henderson
8e6c34b390
fix: Allow only test_file in pytorch and flax summarization ( #22293 )
...
allow only test_file in pytorch and flax summarization
2023-03-22 10:46:56 +00:00
Aaron Gokaslan
5e8c8eb5ba
Apply ruff flake8-comprehensions ( #21694 )
2023-02-22 09:14:54 +01:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting ( #21480 )
...
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
2023-02-06 18:10:56 -05:00
Sylvain Gugger
05e72aa0c4
Adapt repository creation to latest hf_hub ( #21158 )
...
* Adapt repository creation to latest hf_hub
* Update all examples
* Fix other tests, add Flax examples
* Address review comments
2023-01-18 11:14:00 -05:00
Duong A. Nguyen
4212bb0d60
[Re-submit] Compute true loss Flax examples ( #19504 )
...
* Compute true loss
* fixup
* final
* final
* final
* Update examples/flax/language-modeling/run_bart_dlm_flax.py
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
* jax.tree_map => jax.tree_util.tree_map
* Compute true loss
* final
* fixup
* final
* final
* Update examples/flax/language-modeling/run_bart_dlm_flax.py
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
* jax.tree_map => jax.tree_util.tree_map
Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>
2022-10-13 11:33:36 +01:00
Sanchit Gandhi
e6f221c8d4
[JAX] Replace all jax.tree_* calls with jax.tree_util.tree_* ( #18361 )
...
* [JAX] Replace all jax.tree_* calls with jax.tree_util.tree_*
* fix double tree_util
2022-09-09 15:18:56 +02:00
Karim Foda
d6eeb87170
Flax Remat for LongT5 ( #17994 )
...
* [Flax] Add remat (gradient checkpointing)
* fix variable naming in test
* flip: checkpoint using a method
* fix naming
* fix class naming
* apply PVP's suggestions from code review
* add gradient_checkpointing to examples
* Add gradient_checkpointing to run_mlm_flax
* Add remat to longt5
* Add gradient checkpointing test longt5
* Fix args errors
* Fix remaining tests
* Make fixup & quality fixes
* replace kwargs
* remove unecessary kwargs
* Make fixup changes
* revert long_t5_flax changes
* Remove return_dict and copy to LongT5
* Remove test_gradient_checkpointing
Co-authored-by: sanchit-gandhi <sanchit@huggingface.co>
2022-08-14 16:27:13 +01:00
Julien Chaumond
9129fd0377
transformers-cli login
=> huggingface-cli login
(#18490 )
...
* zero chance anyone's using that constant no?
* `transformers-cli login` => `huggingface-cli login`
* `transformers-cli repo create` => `huggingface-cli repo create`
* `make style`
2022-08-06 09:42:55 +02:00
Sylvain Gugger
941d233153
Fix ROUGE add example check and update README ( #18398 )
...
* Fix ROUGE add example check and update README
* Stay consistent in values
2022-08-01 11:14:49 -04:00
Sylvain Gugger
986526a0e4
Replace as_target
context managers by direct calls ( #18325 )
...
* Preliminary work on tokenizers
* Quality + fix tests
* Treat processors
* Fix pad
* Remove all uses of in tests, docs and examples
* Replace all as_target_tokenizer
* Fix tests
* Fix quality
* Update examples/flax/image-captioning/run_image_captioning_flax.py
Co-authored-by: amyeroberts <amy@huggingface.co>
* Style
Co-authored-by: amyeroberts <amy@huggingface.co>
2022-07-29 08:09:09 -04:00
Vijay S Kalmath
da503ea02f
Migrate metrics used in flax examples to Evaluate ( #18348 )
...
Currently, tensorflow examples use the `load_metric` function from
Datasets library, commit migrates function call to `load` function
from Evaluate library.
2022-07-28 15:06:23 -04:00
Sanchit Gandhi
7490a97cac
[Flax] Fix incomplete batches in example scripts ( #17863 )
...
* [Flax] Fix incomplete batches in example scripts
* fix dataloader batching
* convert jnp batch idxs to np array
* add missing `pad_shard_unpad` to final prediction generate step
* only `pad_shard_unpad` at inference time
* merge conflicts
* remove incomplete batch step from eval
* fix run_qa.py
* add `pad_shard_unpad` to run_flax_ner.py
* add `pad_shard_unpad` to run_flax_glue.py
* add `pad_shard_unpad` to run_image_classification.py
* make style
* fix mlm flax eval batches
* remove redundant imports
2022-07-27 15:50:47 +01:00
Duong A. Nguyen
170fcaa604
Generalize decay_mask_fn to apply mask to all LayerNorm params ( #18273 )
...
* generalize decay_mask_fn to find all layernorm params
* fixup
* generalising decay_mask_fn
2022-07-27 12:23:57 +01:00
Duong A. Nguyen
4bea6584e3
Remove use_auth_token from the from_config method ( #18192 )
...
* remove use_auth_token from from_config
* restore use_auth_token from_pretrained run_t5_mlm_flax
2022-07-19 08:13:20 +02:00
Sylvain Gugger
3cab90279f
Add examples telemetry ( #17552 )
...
* Add examples telemetry
* Alternative approach
* Add to all other examples
* Add to templates as well
* Put framework separately
* Same for TensorFlow
2022-06-07 11:57:52 -04:00
Sylvain Gugger
afe5d42d8d
Black preview ( #17217 )
...
* Black preview
* Fixup too!
* Fix check copies
* Use the same version as the CI
* Bump black
2022-05-12 16:25:55 -04:00
Karim Foda
24a85cca61
Add use_auth to load_datasets for private datasets to PT and TF examples ( #16521 )
...
* fix formatting and remove use_auth
* Add use_auth_token to Flax examples
2022-04-04 10:27:45 -04:00
Stas Bekman
a73281e3e4
[examples] max samples can't be bigger than the len of dataset ( #16501 )
...
* [examples] max samples can't be bigger than then len of dataset
* do tf and flax
2022-03-30 12:33:16 -07:00
Sylvain Gugger
4975002df5
Reorganize file utils ( #16264 )
...
* Split file_utils in several submodules
* Fixes
* Add back more objects
* More fixes
* Who exactly decided to import that from there?
* Second suggestion to code with code review
* Revert wront move
* Fix imports
* Adapt all imports
* Adapt all imports everywhere
* Revert this import, will fix in a separate commit
2022-03-23 10:26:33 -04:00
Yih-Dar
a94105f95f
Fix preprocess_function in run_summarization_flax.py ( #14769 )
...
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2021-12-15 11:36:28 +01:00
Suraj Patil
6a025487a6
[Flax examples] remove dependancy on pytorch training args ( #14636 )
...
* use custom training arguments
* update tests
2021-12-12 09:19:12 +05:30
Suraj Patil
c5bd732ac6
Add Flax example tests ( #14599 )
...
* add test for glue
* add tests for clm
* fix clm test
* add summrization tests
* more tests
* fix few tests
* add test for t5 mlm
* fix t5 mlm test
* fix tests for multi device
* cleanup
* ci job
* fix metric file name
* make t5 more robust
2021-12-06 10:48:58 +05:30
Yih-Dar
a6ea244f99
Fix: save checkpoint after each epoch and push checkpoint to the hub ( #13872 )
...
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2021-10-05 16:30:13 +05:30
Suraj Patil
7db2a79b38
[examples/flax] use Repository API for push_to_hub ( #13672 )
...
* use Repository for push_to_hub
* update readme
* update other flax scripts
* update readme
* update qa example
* fix push_to_hub call
* fix typo
* fix more typos
* update readme
* use abosolute path to get repo name
* fix glue script
2021-09-30 16:38:07 +05:30
Bhadresh Savani
de23ecea36
added test file ( #12630 )
2021-07-12 12:15:14 +05:30
Patrick von Platen
813328682e
[Flax] Example scripts - correct weight decay ( #12409 )
...
* fix_torch_device_generate_test
* remove @
* finish
* finish
* correct style
2021-06-29 12:01:08 +01:00
Patrick von Platen
2d70c91206
[Flax] Adapt flax examples to include push_to_hub
( #12391 )
...
* fix_torch_device_generate_test
* remove @
* finish
* correct summary writer
* correct push to hub
* fix indent
* finish
* finish
* finish
* finish
* finish
Co-authored-by: Patrick von Platen <patrick@huggingface.co>
2021-06-28 19:23:35 +01:00
Stas Bekman
4a872caef4
remove extra white space from log format ( #12360 )
2021-06-25 13:20:14 -07:00
Suraj Patil
c0fe3c9a7a
Flax summarization script ( #12230 )
...
* add summrization script
* fix arguments, preprocessing, metrics
* add generation and metrics
* auto model, prediction loop
* prettify
* label smoothing
* adress Sylvain and Patricks suggestions
* dynamically import shift_tokens_right
* fix shift_tokens_right_fn call
2021-06-23 15:49:30 +05:30