riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-08 07:10:06 +06:00
Code Issues Actions 25 Packages Projects Releases Wiki Activity
19,383 Commits 859 Branches 225 Tags 2.1 GiB
remove-script-datasets-in-tests-test-datasets-main
Commit Graph

163 Commits

Author SHA1 Message Date
Yih-Dar
d53b8ad780
Update use_auth_token -> token in example scripts (#25167) 
* pytorch examples

* tensorflow examples

* flax examples

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2023-07-28 15:33:45 +02:00
Lucain
6232c380f2
Fix .push_to_hub and cleanup get_full_repo_name usage (#25120) 
* Fix .push_to_hub and cleanup get_full_repo_name usage

* Do not rely on Python bool conversion magic

* request changes
 2023-07-28 11:40:08 +02:00
Zach Mueller
aa1b09c5d1
Change logic for logging in the examples (#24956) 
Change logic
 2023-07-20 12:30:10 -04:00
Sylvain Gugger
e9ad51306f
4.32.0.dev0 2023-07-17 13:30:44 -04:00
Ethan
f7d80cb3d2
Fix steps bugs in no trainer examples (#24197) 
Fix step bugs in no trainer + load checkpoint + grad acc
 2023-06-12 11:49:55 -04:00
Sylvain Gugger
ba695c1efd
v4.31.0.dev0 2023-06-07 16:49:00 -04:00
Zachary Mueller
072188d638
Act on deprecations in Accelerate no_trainer examples (#24053) 
Act on deprecation
 2023-06-06 13:04:38 -04:00
Zachary Mueller
b191d7db44
Update all no_trainer with skip_first_batches (#23664) 2023-05-22 14:49:31 -04:00
Sylvain Gugger
a0c0a78233
v4.30.0.dev0 2023-05-09 14:59:38 -04:00
Sylvain Gugger
888c4a2ae0
v4.29.0.dev0 2023-04-12 20:04:29 -04:00
Sylvain Gugger
1b1867d86b
Replace -100s in predictions by the pad token (#22693) 
* Replace -100s in predictions by the pad token

* Style

* Try to catch them all
 2023-04-11 09:32:20 -04:00
Sylvain Gugger
ebdb185bef
v4.28.0.dev0 2023-03-14 13:49:10 -04:00
Sylvain Gugger
b19d64d852
Respect documentation on passive log level (#21700) 
* Respect documentation on passive log level

* Fix test and set log level in examples

* Add doc
 2023-02-22 09:39:18 +01:00
Sylvain Gugger
6f79d26442
Update quality tooling for formatting (#21480) 
* Result of black 23.1

* Update target to Python 3.7

* Switch flake8 to ruff

* Configure isort

* Configure isort

* Apply isort with line limit

* Put the right black version

* adapt black in check copies

* Fix copies
 2023-02-06 18:10:56 -05:00
Sylvain Gugger
7119bb052a
v4.27.0.dev0 2023-01-23 16:52:35 -05:00
Sylvain Gugger
05e72aa0c4
Adapt repository creation to latest hf_hub (#21158) 
* Adapt repository creation to latest hf_hub

* Update all examples

* Fix other tests, add Flax examples

* Address review comments
 2023-01-18 11:14:00 -05:00
Sylvain Gugger
60d1f31bb0
v4.26.0.dev0 2022-12-01 16:19:33 -05:00
Zachary Mueller
822ae69c1b
Update reqs to include min gather_for_metrics Accelerate version (#20242) 
* Update reqs to include min gather_for_metrics Accelerate version

* Other reqs
 2022-11-15 13:28:00 -05:00
Muhammad Sakib Khan Inan
777b1bfe62
New logging support to "Trainer" Class (ClearML Logger) (#20184) 
* Init Update

* ClearML Callbacks integration

* update corrections

* args reporting updated

* {'tensorboard': False, 'pytorch': False}

* ClearML Tests added

* add clearml

* output_uri=True in Task.init

* reformatted integrations.py

* reformatted and fixed

* IF-ELSE statement issue on "has_clearml" resolved

* Add clearml in main callback docs

* Add additional clearml documentation

* Update src/transformers/integrations.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Accept suggestion

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Accept suggestion

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Small change in comments

* Make style clearml

* Accept suggestion

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Victor Sonck <victor.sonck@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-11-15 10:08:59 -05:00
Ming Liu
36b063ed4f
Update README.md (#20188) 
There is typo in the original hyperlink.

Below is the original version:
Based on the script [`run_translation_no_trainer.py`](https://github.com/huggingface/transformers/blob/main/examples/pytorch/translation/**run_translationn_no_trainer.py**).
 2022-11-14 12:53:02 -05:00
Sylvain Gugger
06886d5a68
Only resize embeddings when necessary (#20043) 
* Only resize embeddings when necessary

* Add comment
 2022-11-03 12:05:04 -04:00
Sylvain Gugger
c3a93d8d82
v4.25.0.dev0 2022-10-31 21:48:40 -04:00
Lysandre
10100979ed Dev version 2022-10-10 17:25:40 -04:00
Lysandre
16913b3c92 Dev version 2022-09-14 14:58:20 -04:00
Rahul A R
00fc9217d1
Fixed bug which caused overwrite_cache to always be True (#19000) 
* fixed bug which caused overwrite_cache to always be True (#18967).

* reformatting changes
 2022-09-13 11:29:48 -04:00
Nicholas Broad
4f299b2446
Accelerator end training (#18910) 
* add accelerator.end_training()

Some trackers need this to end their runs.

* fixup and quality

* add space

* add space again ?!?
 2022-09-07 07:46:26 -04:00
Rahul A R
e9442440fc
streamlining 'checkpointing_steps' parsing (#18755) 2022-08-25 11:00:38 -04:00
Zachary Mueller
358fc18613
Add evaluate to examples requirements (#18666) 2022-08-18 10:57:39 -04:00
zhoutang776
25e651a2de
Update run_translation_no_trainer.py (#18637) 
* Update run_translation_no_trainer.py

found an error in selecting `no_decay` parameters and some small modifications when the user continues to train from a checkpoint

* fixs `no_decay` and `resume_step` issue

1. change `no_decay` list
2. if use continue to train their model from provided checkpoint, the `resume_step` will not be initialized properly if `args.gradient_accumulation_steps != 1`
 2022-08-16 13:25:57 -04:00
Julien Chaumond
9129fd0377
transformers-cli login => huggingface-cli login (#18490) 
* zero chance anyone's using that constant no?

* `transformers-cli login` => `huggingface-cli login`

* `transformers-cli repo create` => `huggingface-cli repo create`

* `make style`
 2022-08-06 09:42:55 +02:00
Sylvain Gugger
941d233153
Fix ROUGE add example check and update README (#18398) 
* Fix ROUGE add example check and update README

* Stay consistent in values
 2022-08-01 11:14:49 -04:00
Ogundepo Odunayo
679d68a11b
Correct the spelling of bleu metric (#18375) 2022-08-01 07:51:27 -04:00
atturaioe
1f84399171
Migrate metric to Evaluate in Pytorch examples (#18369) 
* Migrate metric to Evaluate in pytorch examples

* Remove unused imports
 2022-08-01 07:40:25 -04:00
Sylvain Gugger
986526a0e4
Replace as_target context managers by direct calls (#18325) 
* Preliminary work on tokenizers

* Quality + fix tests

* Treat processors

* Fix pad

* Remove all uses of  in tests, docs and examples

* Replace all as_target_tokenizer

* Fix tests

* Fix quality

* Update examples/flax/image-captioning/run_image_captioning_flax.py

Co-authored-by: amyeroberts <amy@huggingface.co>

* Style

Co-authored-by: amyeroberts <amy@huggingface.co>
 2022-07-29 08:09:09 -04:00
Lysandre
c89a592e87 Dev version 2022-07-27 17:13:57 +02:00
Zachary Mueller
75259b44bf
Properly calculate the total train iterations and recalculate num epochs in no_trainer scripts (#17856) 2022-06-23 15:46:01 -04:00
Sylvain Gugger
7c6ec195ad v4.21.0.dev0 2022-06-16 12:20:53 -04:00
Sylvain Gugger
3cab90279f
Add examples telemetry (#17552) 
* Add examples telemetry

* Alternative approach

* Add to all other examples

* Add to templates as well

* Put framework separately

* Same for TensorFlow
 2022-06-07 11:57:52 -04:00
Sourab Mangrulkar
d156898f3b
Improve notrainer examples (#17449) 
* improve no-trainer examples

* Trigger CI

* adding comment to clarify tracker init on main process

* Trigger CI

* Trigger CI

* Trigger CI
 2022-05-28 00:06:31 +05:30
Zachary Mueller
1762ded30a
Fix metric calculation in examples and setup tests to run on multi-gpu for no_trainer scripts (#17331) 
* Fix length in no_trainer examples

* Add setup and teardown

* Use new accelerator config generator to automatically make tests able to run based on environment
 2022-05-18 14:17:40 -04:00
Sylvain Gugger
afe5d42d8d
Black preview (#17217) 
* Black preview

* Fixup too!

* Fix check copies

* Use the same version as the CI

* Bump black
 2022-05-12 16:25:55 -04:00
Lysandre Debut
5294fa12ee Dev version 2022-05-12 11:04:23 -04:00
Zachary Mueller
d719bcd46a
Fix all docs for accelerate install directions (#17145) 2022-05-09 15:45:18 -04:00
Zachary Mueller
f275e593bf
Fix no_trainer examples to properly calculate the number of samples (#17046) 
* Update all examples to properly calculate progress bar
 2022-05-02 11:56:25 -04:00
Zachary Mueller
35d48db881
Update no_trainer examples to use new logger (#17044) 
* Propagate and fix imports
 2022-05-02 11:56:15 -04:00
Zachary Mueller
3486a92a57
Fix savedir for by epoch (#16996) 2022-04-28 13:49:45 -04:00
Zachary Mueller
60e1d883f1
Fixup no_trainer save logic (#16968) 
* Fixup all examples
 2022-04-27 14:46:49 -04:00
Sylvain Gugger
c79bbc3ba5
Fix multiple deletions of the same files in save_pretrained (#16947) 
* Fix multiple deletions of the same files in save_pretrained

* Add is_main_process argument
 2022-04-27 12:28:42 -04:00
Zachary Mueller
705d65368f
Fix multiproc metrics in no_trainer examples (#16865) 2022-04-20 17:26:27 -04:00
Zachary Mueller
be752d12f8
Fixup no_trainer examples scripts and add more tests (#16765) 
* Change tracking to store_true

* Remove step param and use it in the log dictionary directly

* use vars(args) when passing args to init_trackers

* Include tracking tests since tensorboard is already a dep
 2022-04-13 14:40:48 -04:00
Heerak Son
db3edd050b
Update run_translation_no_trainer.py (#16652) 
args.model_name_or_path -> args.config_name
fix it
 2022-04-12 08:55:12 -04:00
Zachary Mueller
d4b3e359aa
Don't push checkpoints to hub in no_trainer scripts (#16703) 
Adds checkpoint prefixes to the gitignore if `push_to_hub` is used along with `checkpointint_steps`
 2022-04-11 12:42:45 -04:00
Zachary Mueller
d57da99237
Add tests for no_trainer and fix existing examples (#16656) 
* Fixed some bugs involving saving during epochs
* Added tests mimicking the existing examples tests
* Added in json exporting to all `no_trainer` examples for consistency
 2022-04-08 10:03:56 -04:00
Zachary Mueller
febe42b5da
Update no_trainer scripts with new Accelerate functionalities (#16617) 
Adds logging and save/loading to the Accelerate scripts

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-04-06 15:29:32 -04:00
Lysandre Debut
a180efe7fd Dev version 2022-04-06 11:08:12 -04:00
Karim Foda
24a85cca61
Add use_auth to load_datasets for private datasets to PT and TF examples (#16521) 
* fix formatting and remove use_auth

* Add use_auth_token to Flax examples
 2022-04-04 10:27:45 -04:00
Stas Bekman
a73281e3e4
[examples] max samples can't be bigger than the len of dataset (#16501) 
* [examples] max samples can't be bigger than then len of dataset

* do tf and flax
 2022-03-30 12:33:16 -07:00
Sylvain Gugger
4975002df5
Reorganize file utils (#16264) 
* Split file_utils in several submodules

* Fixes

* Add back more objects

* More fixes

* Who exactly decided to import that from there?

* Second suggestion to code with code review

* Revert wront move

* Fix imports

* Adapt all imports

* Adapt all imports everywhere

* Revert this import, will fix in a separate commit
 2022-03-23 10:26:33 -04:00
Lysandre Debut
eca77f4719
Updates the default branch from master to main (#16326) 
* Updates the default branch from master to main

* Links from `master` to `main`

* Typo

* Update examples/flax/README.md

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-03-23 03:46:59 -04:00
Sylvain Gugger
79d28e80b6 v4.18.0.dev.0 2022-03-03 10:19:58 -05:00
Suraj Patil
bf1fe32824
[examples/summarization and translation] fix readme (#15833) 2022-02-25 17:28:16 +01:00
Lysandre
eab338104d Docs for version v4.16.0 2022-01-27 13:11:51 -05:00
Lysandre
f87db5e412 Release: v4.16.0 2022-01-27 13:06:33 -05:00
Patrick von Platen
fa39ff9fc4 Docs for v4.16.0dev0 2021-12-22 20:39:44 +01:00
Patrick von Platen
05fa1a7ac1 Release: v4.15.0 2021-12-22 18:43:15 +01:00
Lysandre
7c9c41f43c Docs for v4.14.0 2021-12-15 18:29:53 +01:00
Lysandre
960d8cb41d Release: v4.14.0 2021-12-15 18:20:35 +01:00
Lysandre
ab31b3e41b Docs for v4.14.0dev0 2021-12-09 17:09:23 +01:00
Lysandre
4da3a696e4 Release: v4.13.0 2021-12-09 16:55:21 +01:00
Gaurang Tandon
4ea19de80c
fix: verify jsonlines file in run_translation (#14660) (#14661) 
* fix: verify jsonl in run_translation (#14660)

* fix(run_translation.py): json/jsonl validation

Both json and jsonl are to be accepted as valid jsonlines file extension

* fix(run_translation.py): make black happy

* Ran make style
 2021-12-08 13:25:30 -05:00
Sylvain Gugger
08a5f57567
Add new LFS prune API (#14294) 2021-11-05 18:58:51 -04:00
Lysandre
b8fad022a0 v4.13.0.dev0 2021-10-28 12:56:46 -04:00
Lysandre
62bf536631 Release v4.12.0 2021-10-28 12:09:49 -04:00
Patrick von Platen
44eb8bdeea
map only on one process (#13810) 2021-09-30 18:52:53 +02:00
Lysandre
11c69b8045 Docs for version v4.11.0 2021-09-27 14:19:38 -04:00
Lysandre
dc193c906d Release: v4.11.0 2021-09-27 14:14:09 -04:00
Gunjan Chhablani
38580455de
Add model card creation snippet to example scripts (#13730) 
* Update run_glue.py

* Update run_glue.py

* Add model creation snippet to other scripts

* Fix style
 2021-09-24 15:51:46 +02:00
Sylvain Gugger
b7d264be0d
Add push_to_hub to no_trainer examples (#13659) 
* Add push_to_hub to no_trainer examples

* Quality

* Document integration

* Roll out to other examples
 2021-09-21 13:13:30 -04:00
Aleksander Smywiński-Pohl
008c2d0b7a
Fix typo in documentation (#13494) 
* Fix typo in deepspeed documentation

* Add missing import in deepspeed configuration

* Fix path in translation examples
 2021-09-09 08:00:05 -04:00
Lysandre
5ee67a4412 Docs for v4.10.0 2021-08-31 16:02:31 +02:00
Lysandre
d12bbe4942 Release: v4.10.0 2021-08-31 15:53:10 +02:00
Sylvain Gugger
c76de1053e
Add generate kwargs to Seq2SeqTrainingArguments (#13339) 
* Add generate kwargs to Seq2SeqTrainingArguments

* typo

* Address review comments + doc

* Style
 2021-08-31 08:42:00 -04:00
Allan Lin
91ff480e26
Update namespaces inside torch.utils.data to the latest. (#13167) 
* Update torch.utils.data namespaces to the latest.

* Format

* Update Dataloader.

* Style
 2021-08-19 14:29:51 +02:00
Sylvain Gugger
303989de0e
Add accelerate to examples requirements (#12888) 2021-07-26 09:57:34 -04:00
Lysandre
40de2d5a4f Docs for v4.10.0dev0 2021-07-22 12:52:25 +02:00
Lysandre
72aee83ced Release: v4.9.0 2021-07-22 12:11:55 +02:00
Bhadresh Savani
539ee456d4
[Examples] Replicates the new --log_level feature to all trainer-based pytorch (#12359) 
* added log_level

* fix comment

* fixed log_level

* Trigger CI

* Unfied logging

* simplified args for log_level
 2021-06-25 14:58:42 -07:00
Stas Bekman
64e6098094
[trainer] add main_process_first context manager (#12351) 
* main_process_first context manager

* handle multi-node, add context description

* sync desc
 2021-06-25 14:58:03 -07:00
Stas Bekman
4a872caef4
remove extra white space from log format (#12360) 2021-06-25 13:20:14 -07:00
Sylvain Gugger
2150dfed31 v4.9.0.dev0 2021-06-23 13:31:19 -04:00
Sylvain Gugger
9252a5127f Release: v4.8.0 2021-06-23 13:25:56 -04:00
Stas Bekman
ebe5413589
[trainer] 2 bug fixes and a rename (#12309) 
* bug fixes and a rename

* add extended DDP test
 2021-06-22 11:13:23 -07:00
Stas Bekman
dad414d5f9
[trainer + examples] set log level from CLI (#12276) 
* set log level from CLI

* add log_level_replica + test + extended docs

* cleanup

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* rename datasets objects to allow datasets module

* improve the doc

* style

* doc improve

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-06-21 19:30:50 -07:00
Bhavitvya Malik
e43e11260f
update desc for map in all examples (#12226) 
* update desc for map in all examples

* added plm

* suggestions
 2021-06-17 15:37:31 -04:00
Lysandre
0daadc1919 Docs for v4.8.0 2021-06-17 18:17:42 +02:00
Lysandre
7a6c9fab8e Release: v4.7.0 2021-06-17 17:57:42 +02:00
Sylvain Gugger
7d7ceca396
Model card defaults (#12122) 
* [WIP] Model card defaults

* finetuned_from default value

* Add all mappings to the mapping file

* Be more defensive on finetuned_from arg

* Add default task tag

* Separate tags from tasks

* Edge case for dataset

* Apply suggestions from code review

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
 2021-06-15 16:01:37 -04:00
Sylvain Gugger
b8ab541340
Don't log anything before logging is setup in examples (#12121) 
* Don't log anything before logging is setup in examples

* Last example
 2021-06-14 08:03:33 -04:00
Philip May
cfca638acb
Add MT5ForConditionalGeneration as supported arch. to summarization README (#11961) 
* Add MT5ForConditionalGeneration as supported arch.

* Update README.md
 2021-05-31 21:24:33 +05:30
Sylvain Gugger
f086652b16
Add option to log only once in multinode training (#11819) 
* Add option to long only once in multinode training

* Use an alternate property
 2021-05-25 08:03:43 -04:00