riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity
9,466 Commits 887 Branches 232 Tags 2.1 GiB
7ccacdf10f
Commit Graph

9466 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Patrick von Platen
7ccacdf10f
[Doctests] Correct filenaming (#16599) 
* [Doctests] Correct filenaming

* improve quicktour

* make style
 2022-04-05 14:15:02 +02:00
Suraj Patil
21decb7731
handle torch_dtype in low cpu mem usage (#16580) 2022-04-05 12:26:03 +02:00
Francesco Saverio Zuppichini
8bf6d28c10
made _load_pretrained_model_low_mem static + bug fix (#16548) 2022-04-05 11:56:36 +02:00
SaulLu
02214cb3cc
add a template to add missing tokenization test (#16553) 
* add a template to add missing tokenization test

* add cookiecutter setting

* improve doc

* Update templates/adding_a_missing_tokenization_test/README.md

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-04-05 10:50:22 +02:00
Yih-Dar
765bafb8e4
Fix CI: test_inference_for_pretraining in ViTMAEModelTest (#16591) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-04-05 10:00:03 +02:00
Sylvain Gugger
104c065277 Trigger doc build 2022-04-04 14:06:49 -04:00
Andres Codas
1cd2e21d1b
initialize the default rank set on TrainerState (#16530) 
* initialize the default rank set on TrainerState

* fix style
 2022-04-04 12:20:26 -04:00
Sanchit Gandhi
6f9d8dc156
[SpeechEncoderDecoderModel] Correct Encoder Last Hidden State Output (#16586) 2022-04-04 17:50:56 +02:00
Joao Gante
dad5ca83b2
TF: Finalize unpack_inputs-related changes (#16499) 
* Add unpack_inputs to remaining models

* removed kwargs to `call()` in TF models

* fix TF T5 tests
 2022-04-04 16:37:33 +01:00
SaulLu
be9474bd35
add a test checking the format of convert_tokens_to_string's output (#16540) 
* add new tests

* add comment to overridden tests
 2022-04-04 16:57:24 +02:00
Karim Foda
24a85cca61
Add use_auth to load_datasets for private datasets to PT and TF examples (#16521) 
* fix formatting and remove use_auth

* Add use_auth_token to Flax examples
 2022-04-04 10:27:45 -04:00
Sylvain Gugger
b9a768b3ff
Enable doc in Spanish (#16518) 
* Reorganize doc for multilingual support

* Fix style

* Style

* Toc trees

* Adapt templates
 2022-04-04 10:25:46 -04:00
Sylvain Gugger
3951b9f390
Add utility to find model labels (#16526) 
* Add utility to find model labels

* Use it in the Trainer

* Update src/transformers/utils/generic.py

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>

* Quality

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
 2022-04-04 10:06:57 -04:00
Daniel Stancl
ec4da72fe9
Fix flax import in __init__.py: modeling_xglm -> modeling_flax_xglm (#16556) 2022-04-04 14:54:25 +02:00
Nicolas Patry
013a7dbe3d
Making the impossible to connect error actually report the right URL. (#16446) 2022-04-04 14:26:23 +02:00
Patrick von Platen
ad0cba08ea
[FlaxSpeechEncoderDecoder] Fix dtype bug (#16581) 
* [FlaxSpeechEncoderDecoder] Fix dtype bug

* more fixes
 2022-04-04 13:53:54 +02:00
Yih-Dar
60d27b1f15
Add code samples for TF speech models (#16494) 
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-04-01 17:54:01 +02:00
Lysandre Debut
53a4d6b115
Pin tokenizers version <0.13 (#16539) 
* Pin tokenizers version <0.13

* Style
 2022-04-01 11:53:18 -04:00
NielsRogge
61ee26a892
Improve code example (#16450) 
Co-authored-by: Niels Rogge <nielsrogge@nielss-mbp.home>
 2022-04-01 17:19:36 +02:00
Yih-Dar
2199382dfd
Use random_attention_mask for TF tests (#16517) 
* use random_attention_mask for TF tests

* Fix for TFCLIP test (for now).

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-04-01 16:53:07 +02:00
Gunjan Chhablani
823dbf8a41
Remove MBart subclass of XLMRoberta in tokenzier docs (#16546) 
* Remove MBart subclass of XLMRoberta in tokenzier

* Fix style

* Copy docs from MBart50 tokenizer
 2022-04-01 16:39:28 +02:00
Rishav Chandra Varma
5fe06b9bdd
Adding missing type hints for mBART model (PyTorch) (#16429) 
* added type hints for mbart tensorflow tf implementation

* Adding missing type hints for mBART model 

Tensorflow Implementation model added with missing type hints

* Missing Type hints - correction

For TF model

* Code fixup using make quality tests

* Hint types - typo error

* make fix-copies and make fixup

* type hints

* updated files

* type hints update

* making dependent modesls coherent

Co-authored-by: matt <rocketknight1@gmail.com>
 2022-04-01 15:21:26 +01:00
Gunjan Chhablani
9947dd077c
Add VisualBert type hints (#16544) 2022-04-01 15:02:58 +01:00
Gunjan Chhablani
59a9c83e40
Fix Bart type hints (#16297) 
* Add type hints to PLBart PyTorch

* Remove pending merge conflicts

* Fix PLBart Type Hints

* Add changes from review
 2022-04-01 14:50:22 +01:00
Dahlbomii
afc5a1ea3a
Type hints added (#16529) 2022-04-01 14:27:41 +01:00
Ferdinand Schlatt
483a9450a0
call on_train_end when trial is pruned (#16536) 2022-04-01 08:50:47 -04:00
Jim Rohrer
9de70f213e
Add ONNX export for BeiT (#16498) 
* Add beit onnx conversion support

* Updated docs

* Added cross reference to ViT ONNX config
 2022-04-01 10:52:42 +02:00
Cathy
bfeff6cc6a
Fixed a typo in legacy seq2seq_trainer.py (#16531) 2022-04-01 09:17:31 +02:00
Anton Lozhkov
5807054bd3
[research] link to the XTREME-S paper (#16519) 
* [research] link to the XTREME-S paper

* Update examples/research_projects/xtreme-s/README.md

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>

Co-authored-by: Lysandre Debut <lysandre.debut@reseau.eseo.fr>
 2022-03-31 23:26:50 +04:00
Sylvain Gugger
e4b234834a
Fix syntax error in generate docstrings (#16516) 2022-03-31 08:45:47 -04:00
Mowaninuola Osifeso
b808d8a596
added type hints to xglm pytorch (#16500) 
* added type hints to xglm pytorch

* Update src/transformers/models/xglm/modeling_xglm.py

* Update src/transformers/models/xglm/modeling_xglm.py

Co-authored-by: Matt <Rocketknight1@users.noreply.github.com>
 2022-03-31 13:43:04 +01:00
Bhadresh Savani
05b4c32908
fixed a typo (#16508) 2022-03-31 07:49:02 -04:00
Santiago Gómez
6a4dbba1a3
Translate accelerate.mdx from english to spanish (#16176) 
* Translate accelerate.mdx from english to spanish

* Update docs/source_es/accelerate.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Apply suggestions from code review

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Apply suggestions from code review

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Fix nits and finish translation

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>
 2022-03-31 07:45:18 -04:00
Liliana Badillo
c551addeb0
Translate installation.mdx to Spanish (#16229) 
* Translate installation.mdx to Spanish

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/installation.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Fix nits and finish translation

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>
 2022-03-31 07:44:47 -04:00
Juanjo do Olmo
98939e6aee
Spanish translation of the file multilingual.mdx (#16329) 
* Duplication of the source eng file

* Spanish translation of the file multilingual.mdx

* Update docs/source_es/multilingual.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/multilingual.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/multilingual.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/multilingual.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/multilingual.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/multilingual.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Update docs/source_es/multilingual.mdx

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>

* Fix nits and finish translation

Co-authored-by: Omar U. Espejel <espejelomar@gmail.com>
 2022-03-31 07:43:31 -04:00
chenbohua3
99a01423b9
make tuple annotation more specific to avoid failures during symbolic_trace (#16490) 
* make tuple annotation more specific to avoid failures during symbolic_trace

* make tuple annotation more specific to avoid failures during symbolic_trace
 2022-03-31 12:39:46 +01:00
Francesco Saverio Zuppichini
a8b6443e06
Refactor Modeling Outputs (#16341) 
* first proposal

* replace model outputs in various models

* conflicts

* docstring

* update poolformer

* minor change in docstring

* CI

* removed poolformer specific outputs from doc

* removed convnext specific outputs from doc

* CI

* weird char in segformer

* conversations

* reverted docstring for BaseModelOutputWithPooling

* update outputs

* changed docstring in BaseModelOutput

* updated docstring in modeling outputs

* typos :)

* fixed typo after copy & paste it all around

* CI

* Apply suggestions from code review

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>

* segformer

Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>
 2022-03-31 09:32:33 +02:00
Manuel R. Ciosici
857eb87cc4
Support reduce_bucket_size=auto for deepspeed stages <3 (#16496) 2022-03-30 14:12:29 -07:00
Lai Wei
81ac45f85c
update smddp api to v1.4.0 (#16371) 
* update smddp api to v1.4.0

* Update src/transformers/trainer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Update src/transformers/trainer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* address comments

* fix style

* remove unused import

* fix indent

* disable style check for import

* fix space

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-03-30 16:28:35 -04:00
Stas Bekman
a73281e3e4
[examples] max samples can't be bigger than the len of dataset (#16501) 
* [examples] max samples can't be bigger than then len of dataset

* do tf and flax
 2022-03-30 12:33:16 -07:00
Francesco Saverio Zuppichini
c4deb7b3ae
Feature Extractor accepts segmentation_maps (#15964) 
* feature extractor accepts

* resolved conversations

* added examples in test for ADE20K

* num_classes -> num_labels

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* resolving conversations

* resolving conversations

* removed ADE

* CI

* minor changes in conversion script

* reduce_labels in feature extractor

* minor changes

* correct preprocess for instace segmentation maps

* minor changes

* minor changes

* CI

* debugging

* better padding

* going to update labels inside the model

* going to update labels inside the model

* minor changes

* tests

* removed changes in feature_extractor_utils

* conversation

* conversation

* example in feature extractor

* more docstring in modeling

* test

* make style

* doc

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-03-30 18:46:51 +02:00
Joao Gante
c2f8eaf6bc
TF: unpack inputs on Convbert, GPTJ, LED, and templates (#16491) 
* Add unpack_inputs to remaining models

* remove stray use of inputs in the templates; fix tf.debugging of attn masks
 2022-03-30 17:12:27 +01:00
tomerip
ae189ef991
Add support for exporting GPT-J to ONNX-TRT (#16492) 
Add support for exporting GPT-J to ONNX-TRT

Co-authored-by: Tomer Stav <stavt@amazon.com>
 2022-03-30 17:56:03 +02:00
dctelus
d04adc3521
Add length to PreTrainedTokenizer train_new_from_iterator (#16493) 2022-03-30 11:41:04 -04:00
Aditya Kane
147c816685
Nit: MCSCOCO -> MS COCO (#16481) 2022-03-30 10:06:32 -04:00
Dahlbomii
ffd19ee1de
TF GPT-J Type hints and TF decorator (#16488) 
* Type hints and TF decorator added

* Type hints and TF decorator added

* make style

Co-authored-by: matt <rocketknight1@gmail.com>
 2022-03-30 14:03:54 +01:00
Antoni Baum
277d49a590
Do not initialize torch.distributed process group if one is already initailized (#16487) 
* Do not initialize torch process group twice

* Apply suggestions from code review
 2022-03-29 19:07:31 -04:00
Yih-Dar
2b483230a1
Raise diff tolerance value for TFViTMAEModelTest (#16483) 
* Raise diff tolerance value

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
 2022-03-29 22:12:27 +02:00
Christopher Akiki
ee18d4d2a9
TF GPT2: clearer model variable naming with @unpack_inputs (#16311) 
* add unpack_inputs decorator to Main Layer

* add unpack_inputs decorator to Model

* add unpack_inputs decorator to LMHead Model

* add unpack_inputs decorator to Double Head Model

* add unpack_inputs decorator to Sequence Classification Model

* run fixup recipe

* make unpack_inputs the first decorator
 2022-03-29 20:35:25 +01:00
Sander Land
d7c8ce57d4
Avoid accessing .dataset of a DataLoader in Trainer (#16451) 
* Avoid accessing .dataset of a dataloader

* style

* fix

* cleaning up, reverting some misunderstandings

* black

* add train_dataset argument to get_train_dataloader, and fix other instances of length checks

* flake8

* address comments

* fix bug

* cleanup

* add test

* Update tests/trainer/test_trainer.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* under torch

* merge

* stylistic suggestion

Co-authored-by: Sander Land <sander@chatdesk.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2022-03-29 15:00:18 -04:00