Amy Roberts
b2748a6efd
v4.38.dev.0
2024-01-19 10:43:28 +00:00
Alex Hedges
95091e1582
Set cache_dir for evaluate.load() in example scripts ( #28422 )
...
While using `run_clm.py`,[^1] I noticed that some files were being added
to my global cache, not the local cache. I set the `cache_dir` parameter
for the one call to `evaluate.load()`, which partially solved the
problem. I figured that while I was fixing the one script upstream, I
might as well fix the problem in all other example scripts that I could.
There are still some files being added to my global cache, but this
appears to be a bug in `evaluate` itself. This commit at least moves
some of the files into the local cache, which is better than before.
To create this PR, I made the following regex-based transformation:
`evaluate\.load\((.*?)\)` -> `evaluate\.load\($1,
cache_dir=model_args.cache_dir\)`. After using that, I manually fixed
all modified files with `ruff` serving as useful guidance. During the
process, I removed one existing usage of the `cache_dir` parameter in a
script that did not have a corresponding `--cache-dir` argument
declared.
[^1]: I specifically used `pytorch/language-modeling/run_clm.py` from
v4.34.1 of the library. For the original code, see the following URL:
acc394c4f5/examples/pytorch/language-modeling/run_clm.py
2024-01-11 15:38:44 +01:00
Lysandre
3ed3e3190c
Dev version
2023-12-13 18:29:31 +01:00
Lysandre
bc78fd1274
Dev version
2023-11-02 18:15:36 +01:00
Dong-geon Lee
25e6e9418c
Unify warning styles for better readability ( #27184 )
2023-10-31 18:12:14 +00:00
Lucain
66b088faf0
Provide alternative when warning on use_auth_token ( #27105 )
2023-10-27 14:32:54 +02:00
Tom Aarsen
40ea9ab2a1
Add many missing spaces in adjacent strings ( #26751 )
...
Add missing spaces in adjacent strings
2023-10-12 10:28:40 +02:00
Zach Mueller
1d6a84749b
Fix checkpoint path in no_trainer scripts ( #26733 )
...
checkpoint path
2023-10-11 16:16:27 +02:00
Lysandre
bd6205919a
v4.35.0.dev0
2023-10-03 16:54:37 +02:00
Phuc Van Phan
4fb64e285a
chore: correct update_step and correct gradient_accumulation_steps ( #26068 )
2023-09-12 18:31:23 +01:00
Phuc Van Phan
5af2c62696
docs: add space to docs ( #26067 )
...
* docs: add space to docs
* docs: remove reduntant space
2023-09-11 22:03:26 +01:00
Lysandre
d8e13b3e04
v4.34.dev.0
2023-09-04 15:12:11 -04:00
Sylvain Gugger
5c67682b16
v4.33.0.dev0
2023-08-21 07:07:04 -04:00
Zach Mueller
01ab39b65f
Load state in else ( #25318 )
...
* Load else
* New approach
* Propagate
2023-08-08 05:41:00 -04:00
Jackmin801
145109382a
Allow trust_remote_code in example scripts ( #25248 )
...
* pytorch examples
* pytorch mim no trainer
* cookiecutter
* flax examples
* missed line in pytorch run_glue
* tensorflow examples
* tensorflow run_clip
* tensorflow run_mlm
* tensorflow run_ner
* tensorflow run_clm
* pytorch example from_configs
* pytorch no trainer examples
* Revert "tensorflow run_clip"
This reverts commit 261f86ac1f
2023-08-07 16:32:25 +02:00
Yih-Dar
149cb0cce2
Add token arugment in example scripts ( #25172 )
...
* fix
* fix
* fix
* fix
* fix
* fix
* fix
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-08-02 11:17:31 +02:00
Yih-Dar
d53b8ad780
Update use_auth_token -> token in example scripts ( #25167 )
...
* pytorch examples
* tensorflow examples
* flax examples
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
2023-07-28 15:33:45 +02:00
Lucain
6232c380f2
Fix .push_to_hub and cleanup get_full_repo_name usage ( #25120 )
...
* Fix .push_to_hub and cleanup get_full_repo_name usage
* Do not rely on Python bool conversion magic
* request changes
2023-07-28 11:40:08 +02:00
Zach Mueller
aa1b09c5d1
Change logic for logging in the examples ( #24956 )
...
Change logic
2023-07-20 12:30:10 -04:00
Sylvain Gugger
e9ad51306f
4.32.0.dev0
2023-07-17 13:30:44 -04:00
Ethan
f7d80cb3d2
Fix steps bugs in no trainer examples ( #24197 )
...
Fix step bugs in no trainer + load checkpoint + grad acc
2023-06-12 11:49:55 -04:00
Sylvain Gugger
ba695c1efd
v4.31.0.dev0
2023-06-07 16:49:00 -04:00
Zachary Mueller
072188d638
Act on deprecations in Accelerate no_trainer examples ( #24053 )
...
Act on deprecation
2023-06-06 13:04:38 -04:00
Zachary Mueller
b191d7db44
Update all no_trainer with skip_first_batches ( #23664 )
2023-05-22 14:49:31 -04:00
Hari
5d02e6bd20
Convert numpy arrays to lists before saving the evaluation metrics as json ( #23268 )
...
* convert numpy array to list before writing to json
per_category_iou and per_category_accuracy are ndarray in the eval_metrics
* code reformatted with make style
2023-05-11 08:54:23 -04:00
Sylvain Gugger
a0c0a78233
v4.30.0.dev0
2023-05-09 14:59:38 -04:00
Sylvain Gugger
888c4a2ae0
v4.29.0.dev0
2023-04-12 20:04:29 -04:00
Sylvain Gugger
ebdb185bef
v4.28.0.dev0
2023-03-14 13:49:10 -04:00
Sylvain Gugger
b19d64d852
Respect documentation on passive log level ( #21700 )
...
* Respect documentation on passive log level
* Fix test and set log level in examples
* Add doc
2023-02-22 09:39:18 +01:00
Aaron Gokaslan
5e8c8eb5ba
Apply ruff flake8-comprehensions ( #21694 )
2023-02-22 09:14:54 +01:00
Jeroen Van Der Donckt
bbe98ea9c3
🖊️ fix typo in pytorch semantic segmentation readme ( #21492 )
2023-02-07 09:39:24 -05:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting ( #21480 )
...
* Result of black 23.1
* Update target to Python 3.7
* Switch flake8 to ruff
* Configure isort
* Configure isort
* Apply isort with line limit
* Put the right black version
* adapt black in check copies
* Fix copies
2023-02-06 18:10:56 -05:00
Sylvain Gugger
7119bb052a
v4.27.0.dev0
2023-01-23 16:52:35 -05:00
amyeroberts
4bc18e7a83
Update examples with image processors ( #21155 )
...
* Update examples to use image processors
* Small fixes
* Resolve conflicts
2023-01-19 15:14:58 +00:00
Sylvain Gugger
05e72aa0c4
Adapt repository creation to latest hf_hub ( #21158 )
...
* Adapt repository creation to latest hf_hub
* Update all examples
* Fix other tests, add Flax examples
* Address review comments
2023-01-18 11:14:00 -05:00
Sylvain Gugger
60d1f31bb0
v4.26.0.dev0
2022-12-01 16:19:33 -05:00
Jiahao Li
9681f052a1
Fix result saving errors of pytorch examples ( #20276 )
2022-11-16 09:51:04 -05:00
Muhammad Sakib Khan Inan
777b1bfe62
New logging support to "Trainer" Class (ClearML Logger) ( #20184 )
...
* Init Update
* ClearML Callbacks integration
* update corrections
* args reporting updated
* {'tensorboard': False, 'pytorch': False}
* ClearML Tests added
* add clearml
* output_uri=True in Task.init
* reformatted integrations.py
* reformatted and fixed
* IF-ELSE statement issue on "has_clearml" resolved
* Add clearml in main callback docs
* Add additional clearml documentation
* Update src/transformers/integrations.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Accept suggestion
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Accept suggestion
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Small change in comments
* Make style clearml
* Accept suggestion
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Victor Sonck <victor.sonck@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2022-11-15 10:08:59 -05:00
amyeroberts
a6b7759880
Add Image Processors ( #19796 )
...
* Add CLIP image processor
* Crop size as dict too
* Update warning
* Actually use logger this time
* Normalize doesn't change dtype of input
* Add perceiver image processor
* Tidy up
* Add DPT image processor
* Add Vilt image processor
* Tidy up
* Add poolformer image processor
* Tidy up
* Add LayoutLM v2 and v3 imsge processors
* Tidy up
* Add Flava image processor
* Tidy up
* Add deit image processor
* Tidy up
* Add ConvNext image processor
* Tidy up
* Add levit image processor
* Add segformer image processor
* Add in post processing
* Fix up
* Add ImageGPT image processor
* Fixup
* Add mobilevit image processor
* Tidy up
* Add postprocessing
* Fixup
* Add VideoMAE image processor
* Tidy up
* Add ImageGPT image processor
* Fixup
* Add ViT image processor
* Tidy up
* Add beit image processor
* Add mobilevit image processor
* Tidy up
* Add postprocessing
* Fixup
* Fix up
* Fix flava and remove tree module
* Fix image classification pipeline failing tests
* Update feature extractor in trainer scripts
* Update pad_if_smaller to accept tuple and int size
* Update for image segmentation pipeline
* Update src/transformers/models/perceiver/image_processing_perceiver.py
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
* Update src/transformers/image_processing_utils.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* Update src/transformers/models/beit/image_processing_beit.py
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
* PR comments - docstrings; remove accidentally added resize; var names
* Update docstrings
* Add exception if size is not in the right format
* Fix exception check
* Fix up
* Use shortest_edge in tuple in script
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
2022-11-02 11:57:36 +00:00
Sylvain Gugger
c3a93d8d82
v4.25.0.dev0
2022-10-31 21:48:40 -04:00
Lysandre
10100979ed
Dev version
2022-10-10 17:25:40 -04:00
Sylvain Gugger
c20b2c7e18
Use repo_type instead of deprecated datasets repo IDs ( #19202 )
...
* Use repo_type instead of deprecated datasets repo IDs
* Add missing one in doc
2022-09-26 09:50:48 -04:00
Lysandre
16913b3c92
Dev version
2022-09-14 14:58:20 -04:00
Nicholas Broad
4f299b2446
Accelerator end training ( #18910 )
...
* add accelerator.end_training()
Some trackers need this to end their runs.
* fixup and quality
* add space
* add space again ?!?
2022-09-07 07:46:26 -04:00
Rahul A R
e9442440fc
streamlining 'checkpointing_steps' parsing ( #18755 )
2022-08-25 11:00:38 -04:00
Atharva Ingle
d90a36d192
remove check for main process for trackers initialization ( #18706 )
2022-08-22 11:16:27 -04:00
Zachary Mueller
358fc18613
Add evaluate to examples requirements ( #18666 )
2022-08-18 10:57:39 -04:00
Rasmus Arpe Fogh Jensen
a765b68aa6
Update no_trainer.py scripts to include accelerate gradient accumulation wrapper ( #18473 )
...
* Added accelerate gradient accumulation wrapper to run_image_classification_no_trainer.py example script
* make fixup changes
* PR comments
* changed input to Acceletor based on PR comment, ran make fixup
* Added comment explaining the sync_gradients statement
* Fixed lr scheduler max steps
* Changed run_clm_no_trainer.py script to use accelerate gradient accum wrapper
* Fixed all scripts except wav2vec2 pretraining to use accelerate gradient accum wrapper
* Added accelerate gradient accum wrapper for wav2vec2_pretraining_no_trainer.py script
* make fixup and lr_scheduler step inserted back into run_qa_beam_search_no_trainer.py
* removed changes to run_wav2vec2_pretraining_no_trainer.py script and fixed using wrong constant in qa_beam_search_no_trainer.py script
2022-08-08 15:52:47 -04:00
Julien Chaumond
9129fd0377
transformers-cli login => huggingface-cli login (#18490 )
...
* zero chance anyone's using that constant no?
* `transformers-cli login` => `huggingface-cli login`
* `transformers-cli repo create` => `huggingface-cli repo create`
* `make style`
2022-08-06 09:42:55 +02:00
Kian Sierra McGettigan
0bf1e1aca4
Update no trainer examples for QA and Semantic Segmentation ( #18474 )
...
* swag_no_trainer updated for with gather_metrics
* Removed unused variable samples_seen
* updated examples with gather_for_metrics
2022-08-04 13:22:19 -04:00
