riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity
6,996 Commits 887 Branches 232 Tags 2.1 GiB
aaaed56ffc
Commit Graph

6996 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sylvain Gugger
aaaed56ffc
Trainer iterable dataset (#11254) 
* IterableDatasetShard

* Test and integration in Trainer

* Update src/transformers/trainer_pt_utils.py

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* Style

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
 2021-04-14 17:02:26 -04:00
Stas Bekman
83206ca6a8
[deepspeed] test on one node 2 gpus max (#11237) 
* test on one node 2 gpus max

* fix the other place

* refactor

* fix

* cleanup

* more exact version
 2021-04-14 11:06:59 -07:00
Sylvain Gugger
25e1af36e0
Fix #10128 (#11248) 2021-04-14 11:47:54 -04:00
Stas Bekman
63ca402380
[troubleshooting] add 2 points of reference to the offline mode (#11236) 
* add 2 points of reference to the offline mode

* link the new doc

* add error message

* Update src/transformers/modeling_utils.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* style

* rename

* Trigger CI

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-04-14 08:39:23 -07:00
Yusuke Mori
075e821d1d
Add prefix to examples in model_doc rst (#11226) 
* Add prefix to examples in model_doc rst

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-04-14 10:58:55 -04:00
Thomas Wood
4670b57ce9
Fix dimention misspellings. (#11238) 
* Update modeling_gpt_neo.py

dimention -> dimension

* Update configuration_speech_to_text.py

dimention -> dimension
 2021-04-14 10:39:37 -04:00
Sudharsan S T
f25444cb22
Close open files to suppress ResourceWarning (#11240) 
Co-authored-by: Sudharsan Thirumalai <sudharsan.t@sprinklr.com>
 2021-04-14 10:31:04 -04:00
Lysandre Debut
7fe5aaa8b0
Stale bot updated (#10562) 
* Updated stale bot

* Specify issue number

* Remove particular handling of assignees

* Unleash the stalebot

* Remove debug branch
 2021-04-14 10:24:31 -04:00
Joel Stremmel
9337c6c668
make embeddings plural in warning message (#11228) 2021-04-14 10:13:25 -04:00
Nithin Holla
653076ca30
Save the Wav2Vec2 processor before training starts (#10910) 
Co-authored-by: nithin19 <nithin@amberscript.com>
 2021-04-14 14:52:06 +03:00
Stas Bekman
3d339ee659
[Deepspeed] zero3 tests band aid (#11235) 
* temp band-aid

* style
 2021-04-13 17:58:09 -04:00
Lysandre Debut
1ad7b0398c
Run CI on deepspeed and fairscale (#11172) 
* Run CI on deepspeed and fairscale

* Test it on this branch :)

* Rename

* Update the CI image
 2021-04-13 15:47:06 -04:00
Sylvain Gugger
f38cd4373f
Indent code block in the documentation (#11233) 
* Indent code block

* Indent code blocks version 2

* Quality
 2021-04-13 15:36:36 -04:00
Sylvain Gugger
9d8e8a8703
Avoid using no_sync on SageMaker DP (#11229) 2021-04-13 15:34:00 -04:00
Philipp Schmid
9fa2995993
added cache_dir=model_args.cache_dir to all example with cache_dir arg (#11220) 2021-04-13 18:35:18 +02:00
Sylvain Gugger
3312e96bfb
Doc check: a bit of clean up (#11224) 2021-04-13 12:14:25 -04:00
Suraj Patil
edca520d0f
Refactor GPT2 (#11225) 
* refactor GPT2

* fix mlp and head pruning

* address Sylvains comments

* apply suggestion from code review

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
 2021-04-13 21:15:24 +05:30
Sylvain Gugger
893e51a53f Document v4.5.1 2021-04-13 11:28:17 -04:00
Sylvain Gugger
81009b7a5c
Replace error by warning when loading an architecture in another (#11207) 
* Replace error by warning when loading an architecture in another

* Style

* Style again

* Add a test

* Adapt old test
 2021-04-13 10:33:52 -04:00
Yusuke Mori
22fa0a6004
Add documentation for BertJapanese (#11219) 
* Start writing BERT-Japanese doc

* Fix typo, Update toctree

* Modify model file to use comment for document, Add examples

* Clean bert_japanese by make style

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Split a big code block into two

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Add prefix >>> to all lines in code blocks

* Clean bert_japanese by make fixup

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-04-13 09:49:15 -04:00
Suraj Patil
896d7be974
fix docstrings (#11221) 2021-04-13 08:58:08 -04:00
Lysandre Debut
823df93955
Fix GPT-2 warnings (#11213) 
* Fix GPT-2 warnings

* Update src/transformers/models/gpt2/modeling_gpt2.py

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>

Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
 2021-04-13 08:53:03 -04:00
Lysandre Debut
0cd89d8c83
Add Matt as the TensorFlow reference (#11212) 2021-04-13 08:52:30 -04:00
Ceyda Cinarel
7c205bf40c
wav2vec2 converter: create the proper vocab.json while converting fairseq wav2vec2 finetuned model (#11041) 
* add vocab while converting wav2vec2 original finetuned model

* check save directory exists

* return_attention_mask fix

* quality
 2021-04-13 15:54:33 +05:30
calpt
d49d3cf6d6
Use MSELoss in (M)BartForSequenceClassification (#11178) 2021-04-13 15:24:46 +05:30
Philipp Schmid
f243a5ec0d
Sagemaker test docs update for framework upgrade (#11206) 
* increased train_runtime for model parallelism

* added documentation for framework upgrade
 2021-04-12 19:08:33 -04:00
Lysandre Debut
74d7c24d8d
Import torch.utils.checkpoint in ProphetNet (#11214) 2021-04-12 18:56:17 -04:00
cronoik
38a10c6b52
Replaced which with who (#11183) 2021-04-12 18:08:28 -04:00
NielsRogge
9f1260971f
Add DeiT (PyTorch) (#11056) 
* First draft of deit

* More improvements

* Remove DeiTTokenizerFast from init

* Conversion script works

* Add DeiT to ViT conversion script

* Add tests, add head model, add support for deit in vit conversion script

* Update model checkpoint names

* Update image_mean and image_std, set resample to bicubic

* Improve docs

* Docs improvements

* Add DeiTForImageClassificationWithTeacher to init

* Address comments by @sgugger

* Improve feature extractors

* Make fix-copies

* Minor fixes

* Address comments by @patil-suraj

* All models uploaded

* Fix tests

* Remove labels argument from DeiTForImageClassificationWithTeacher

* Fix-copies, style and quality

* Fix tests

* Fix typo

* Multiple docs improvements

* More docs fixes
 2021-04-12 18:07:10 -04:00
Takuya Makino
cb251ba619
Fix typo (#11188) 2021-04-12 17:35:32 -04:00
fghuman
0c6fcd3034
Added documentation for data collator. (#10941) 
* Added documentation for data collator.

* Update docs/source/data_collator.rst

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Added documentation for data collator.

* Added documentation for the data collator.

* Merge branch 'doc_DataCollator' of C:\Users\mahii\PycharmProjects\transformers with conflicts.

* Update documentation for the data collator.

* Update documentation for the data collator.

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Amna <A.A.Ahmad@student.tudelft.nl>
 2021-04-12 11:59:46 -04:00
Masatoshi TSUCHIYA
ef102c4886
model_path should be ignored as the checkpoint path (#11157) 
* model_path is refered as the path of the trainer, and should be ignored as the checkpoint path.

* Improved according to Sgugger's comment.
 2021-04-12 09:06:41 -04:00
Sylvain Gugger
623cd6aef9 Fix style 2021-04-12 08:14:29 -04:00
cronoik
a99f7f5c75
Minor typos fixed (#11182) 2021-04-12 07:55:40 -04:00
Sylvain Gugger
26212c14e5 Reactivate Megatron tests an use less workers 2021-04-09 18:09:53 -04:00
Lysandre
716120cbd6 Fix Typo 2021-04-09 17:46:52 -04:00
Philipp Schmid
6f90c29eaa
added json dump and extraction of train run time (#11167) 
* added json dump and extraction of train run time

* make style happy
 2021-04-09 15:18:00 -04:00
Stas Bekman
07f0bb691d
[examples run_clm] fix _LazyModule hasher error (#11168) 
* fix _LazyModule hasher error

* reword
 2021-04-09 11:39:12 -07:00
Suraj Patil
c161dd56df
[examples/translation] support mBART-50 and M2M100 fine-tuning (#11170) 
* keep a list of multilingual tokenizers

* add forced_bos_token argument
 2021-04-09 23:58:42 +05:30
Kevin Canwen Xu
fb41f9f50c
Add a special tokenizer for CPM model (#11068) 
* Add a special tokenizer for CPM model

* make style

* fix

* Add docs

* styles

* cpm doc

* fix ci

* fix the overview

* add test

* make style

* typo

* Custom tokenizer flag

* Add REAMDE.md

Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
 2021-04-10 02:07:47 +08:00
Sylvain Gugger
45fc8c7951
Make get_special_tokens_mask consider all tokens (#11163) 2021-04-09 11:57:44 -04:00
Saviour Owolabi
6060746570
Update README.md (#11161) 
Corrected a typo ('Downlowd' to 'Download')
 2021-04-09 11:52:21 -04:00
Keisuke Hirota
b9b60c1630
Fix LogitsProcessor documentation (#11130) 
* Change duplicated LogitsProcessor to LogitsWarper in LogitsProcessorList document

* Write more detailed information about LogitsProcessor's scores argument

* apply suggestion from review

* style

Co-authored-by: Suraj Patil <surajp815@gmail.com>
 2021-04-09 12:39:44 +05:30
Niklas Muennighoff
8b78a32be1
[Community notebooks] Add Wav2Vec notebook for creating captions for YT Clips (#11142) 
* Add Wav2Vec Inference notebook

* Update docs/source/community.md

Co-authored-by: Suraj Patil <surajp815@gmail.com>
 2021-04-09 12:10:37 +05:30
Stas Bekman
0311ba2153
typo (#11152) 
* typo

* style
 2021-04-08 19:47:31 -07:00
Sylvain Gugger
269c9638df Merge branch 'master' of github.com:huggingface/transformers 2021-04-08 21:14:56 -04:00
Sylvain Gugger
d31c7b104e Skip Megatron tests for now 2021-04-08 21:14:43 -04:00
Stas Bekman
c2e0fd5283
[setup] make fairscale and deepspeed setup extras (#11151) 
* make fairscale and deepspeed setup extras

* fix default

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* no reason not to ask for the good version

* update the CIs

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-04-08 15:46:54 -07:00
Sylvain Gugger
ba8b1f4754
Add support for multiple models for one config in auto classes (#11150) 
* Add support for multiple models for one config in auto classes

* Use get_values everywhere

* Prettier doc
 2021-04-08 18:41:36 -04:00
Stas Bekman
97ccf67bb3
[setup] extras[docs] must include 'all' (#11148) 
* extras[doc] must include 'all'

* fix

* better

* regroup
 2021-04-08 18:10:44 -04:00