riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-20 21:18:21 +06:00
Code Issues Actions 7 Packages Projects Releases Wiki Activity
6,161 Commits 876 Branches 227 Tags 2 GiB
f38c4ad302
Commit Graph

42 Commits

Author SHA1 Message Date
Stas Bekman
f38c4ad302
better logging and help (#9203) 2020-12-20 10:28:28 -08:00
Stas Bekman
c19d04623e
[finetune_trainer] enhancements and fixes (#9042) 
* trainer and finetune_trainer enhancements and fixes

* add fallback default

* move the fixing of incorrect keys back into finetune trainer

* s/eval/val/ to match the split

* trainer can now use a different prefix than eval_ for metrics

* document new arg

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* use 'eval' as the default for metric_key_prefix

* complete adjust var names + disambiguate

* fix logger

* add clarifying comment

* add clarifying comment

* style

* Apply suggestions from code review

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* Update src/transformers/trainer.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* complete removal of optional for metric_key_prefix

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
 2020-12-14 17:45:33 -08:00
Sylvain Gugger
783d7d2629
Reorganize examples (#9010) 
* Reorganize example folder

* Continue reorganization

* Change requirements for tests

* Final cleanup

* Finish regroup with tests all passing

* Copyright

* Requirements and readme

* Make a full link for the documentation

* Address review comments

* Apply suggestions from code review

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* Add symlink

* Reorg again

* Apply suggestions from code review

Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>

* Adapt title

* Update to new strucutre

* Remove test

* Update READMEs

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
 2020-12-11 10:07:02 -05:00
Sylvain Gugger
c89bdfbe72
Reorganize repo (#8580) 
* Put models in subfolders

* Styling

* Fix imports in tests

* More fixes in test imports

* Sneaky hidden imports

* Fix imports in doc files

* More sneaky imports

* Finish fixing tests

* Fix examples

* Fix path for copies

* More fixes for examples

* Fix dummy files

* More fixes for example

* More model import fixes

* Is this why you're unhappy GitHub?

* Fix imports in conver command
 2020-11-16 21:43:42 -05:00
Sam Shleifer
7abc1d96d1
no warn (#8329) 2020-11-05 11:42:24 -05:00
Stas Bekman
5423f2a9d4
[testing] port test_trainer_distributed to distributed pytest + TestCasePlus enhancements (#8107) 
* move the helper code into testing_utils

* port test_trainer_distributed to work with pytest

* improve docs

* simplify notes

* doc

* doc

* style

* doc

* further improvements

* torch might not be available

* real fix

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2020-10-28 11:51:32 -04:00
Stas Bekman
023f0f3708
[s2s trainer] tests to use distributed on multi-gpu machine (#7965) 2020-10-22 17:26:22 -04:00
Stas Bekman
8b38173398
[seq2seq testing] multigpu test run via subprocess (#7281) 
Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
 2020-10-21 17:20:53 -04:00
Stas Bekman
2255c2c7a0
[seq2seq] get_git_info fails gracefully (#7843) 
Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
 2020-10-16 00:22:43 -04:00
Suraj Patil
99cb924bfb
[s2s] add config params like Dropout in Seq2SeqTrainingArguments (#7532) 2020-10-04 12:42:30 -04:00
Suraj Patil
72d363d979
[examples/s2s] clean up finetune_trainer (#7509) 2020-10-01 12:19:29 -04:00
Sam Shleifer
48f23f92a8
[s2sTrainer] test + code cleanup (#7467) 2020-10-01 00:33:01 -04:00
Amanpreet Singh
c031d01023
Seq2SeqDataset: avoid passing src_lang everywhere (#7470) 
Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
 2020-09-30 13:27:48 -04:00
Sam Shleifer
7296fea1d6
[s2s] rougeLSum expects \n between sentences (#7410) 
Co-authored-by: Swetha Mandava <smandava@nvidia.com>
 2020-09-27 16:27:19 -04:00
Sam Shleifer
d9d0f1140b
[s2s] distributed eval allows num_return_sequences > 1 (#7254) 2020-09-24 17:30:09 -04:00
Sam Shleifer
656c27c3a3
[s2s] save hostname with repo info (#7301) 
* save hostname
 2020-09-21 17:26:24 -04:00
Sam Shleifer
a5638b2b3a
[s2s] dynamic batch size with --max_tokens_per_batch (#7030) 2020-09-17 15:19:34 -04:00
Sam Shleifer
0203ad43bc
[s2s] distributed eval cleanup (#7186) 2020-09-16 15:38:37 -04:00
Stas Bekman
fdaf8ab349
[s2s run_eval] new features (#7109) 
Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
 2020-09-16 13:59:57 -04:00
Sam Shleifer
33d479d2b2
[s2s] distributed eval in one command (#7124) 2020-09-14 15:57:56 -04:00
Sam Shleifer
de9e297964
[s2s] distributed eval cleanup (#7110) 2020-09-13 23:40:38 -04:00
Sam Shleifer
e7f8d2ab64
[s2s] two stage run_distributed_eval.py (#7105) 2020-09-13 17:28:18 -04:00
Sam Shleifer
77950c485a
[wip/s2s] DistributedSortishSampler (#7056) 2020-09-10 15:23:44 -04:00
Sam Shleifer
a4fc0c80b1
[s2s] run_eval.py parses generate_kwargs (#6948) 2020-09-04 14:19:31 -04:00
Sam Shleifer
9336086ab5
prepare_seq2seq_batch makes labels/ decoder_input_ids made later. (#6654) 
* broken test

* batch parity

* tests pass

* boom boom

* boom boom

* split out bart tokenizer tests

* fix tests

* boom boom

* Fixed dataset bug

* Fix marian

* Undo extra

* Get marian working

* Fix t5 tok tests

* Test passing

* Cleanup

* better assert msg

* require torch

* Fix mbart tests

* undo extra decoder_attn_mask change

* Fix import

* pegasus tokenizer can ignore src_lang kwargs

* unused kwarg test cov

* boom boom

* add todo for pegasus issue

* cover one word translation edge case

* Cleanup

* doc
 2020-08-28 11:15:17 -04:00
Lysandre
a75c64d80c Black 20 release 2020-08-26 17:20:22 +02:00
Sam Shleifer
0344428f79
[s2s] round bleu, rouge to 4 digits (#6704) 2020-08-25 00:33:11 -04:00
Sam Shleifer
e92efcf728
Mult rouge by 100: standard units (#6359) 2020-08-13 12:15:54 -04:00
Sam Shleifer
be1520d3a3
rename prepare_translation_batch -> prepare_seq2seq_batch (#6103) 2020-08-11 15:57:07 -04:00
Suraj Patil
9bed355449
[s2s] fix label_smoothed_nll_loss (#6344) 2020-08-08 04:21:12 -04:00
Sam Shleifer
2804fff839
[s2s]Use prepare_translation_batch for Marian finetuning (#6293) 
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2020-08-06 14:58:38 -04:00
Sam Shleifer
dafa296c95
[s2s] Delete useless method, log tokens_per_batch (#6081) 2020-07-28 11:24:23 -04:00
Sam Shleifer
3c7fbf35a6
MBART: support summarization tasks where max_src_len > max_tgt_len (#6003) 
* MBART: support summarization tasks

* fix test

* Style

* add tokenizer test
 2020-07-28 08:18:11 -04:00
Sam Shleifer
5b193b39b0
[examples/seq2seq]: add --label_smoothing option (#5919) 2020-07-21 16:51:39 -04:00
Sam Shleifer
09a2f40684
Seq2SeqDataset uses linecache to save memory by @Pradhy729 (#5792) 
Co-authored-by: Pradhy729 <49659913+Pradhy729@users.noreply.github.com>
 2020-07-18 13:57:33 -04:00
Sam Shleifer
e238e3d55a
[seq2seq] Don't copy self.source in sortishsampler (#5818) 2020-07-17 01:53:25 -04:00
Sam Shleifer
1a647abf0b
[fix] check code quality (#5772) 2020-07-15 14:59:38 -04:00
Sam Shleifer
d0486c8bc2
[cleanup] T5 test, warnings (#5761) 2020-07-15 08:23:22 -04:00
Sam Shleifer
353b8f1e7a
Add mbart-large-cc25, support translation finetuning (#5129) 
improve unittests for finetuning, especially w.r.t testing frozen parameters
fix freeze_embeds for T5
add streamlit setup.cfg
 2020-07-07 13:23:01 -04:00
Sam Shleifer
393b8dc09a
examples/seq2seq/run_eval.py fixes and docs (#5322) 2020-06-26 19:20:43 -04:00
Thomas Wolf
601d4d699c
[tokenizers] Updates data processors, docstring, examples and model cards to the new API (#5308) 
* remove references to old API in docstring - update data processors

* style

* fix tests - better type checking error messages

* better type checking

* include awesome fix by @LysandreJik for #5310

* updated doc and examples
 2020-06-26 19:48:14 +02:00
Sam Shleifer
40457bcebb
examples/seq2seq supports translation (#5202) 2020-06-24 23:58:11 -04:00