Julien Plu
83d803ba02
Making TF BART-like models XLA and AMP compliant ( #10191 )
...
* Update BART
* Update Blenderbot
* Update BlenderbotSmall
* Update Marian
* Update MBart
* Update MBart
* Update Pegasus
* Update template
* Fix Marian and Pegasus
* Apply style
* Default initializer
* Default initializer
* Default initializer
* Remove int32 casts
* Fix template
* Remove more cast
2021-02-17 17:48:56 +01:00
Julien Plu
31b0560ab4
Add AMP for Albert ( #10141 )
2021-02-15 17:18:33 +01:00
Julien Plu
570218878a
Fix TF template ( #10189 )
...
* Fix template
* Update Seq2Seq tests
2021-02-15 09:21:57 -05:00
Patrick von Platen
8e13b73593
Update README.md
2021-02-11 18:35:27 +03:00
Patrick von Platen
d6b4f48ecb
Update ADD_BIG_BIRD.md
2021-02-11 18:34:17 +03:00
Patrick von Platen
4cda2d73ef
Update ADD_BIG_BIRD.md
2021-02-09 19:58:35 +03:00
Julien Plu
b82fe7d258
Replace strided slice with tf.expand_dims ( #10078 )
...
* Replace tf.newaxis -> tf.expand_dims
* Fix tests
* Fix tests
* Use reshape when a tensors needs a double expand
* Fix GPT2
* Fix GPT2
2021-02-09 11:48:28 -05:00
Lysandre Debut
c9df1b1d53
Model templates ( #10072 )
2021-02-08 09:07:02 -05:00
Julien Plu
cdd8659231
Fix TF template ( #10069 )
...
* Fix template
* Fix template
2021-02-08 08:10:50 -05:00
Julien Plu
31563e056d
Restore TF embeddings and attention layers to their previous version ( #9890 )
...
* Refacto BERT
* Restore all the concerned models
* Remove print
* Update template
* Apply Sylvain's and Morgan's comments
* Fix cast
* Put the cast inside call
* Remove cond in ebds
* Fix funnel
* Restore previous dot product (attention_scores) computation
* Add ConvBERT and BART
* Make all the S2S models ONNX compliant
* Fix test
* Fix check copies
2021-02-08 14:36:30 +03:00
Lysandre Debut
ae37ceacbd
Fix typo ( #10064 )
2021-02-08 06:02:05 -05:00
Patrick von Platen
89be094e29
[Templates] Add template "call-for-model" markdown and "call-for-big-bird" markdown ( #9921 )
...
* add big bird
* change teacher to mentor
* add proposal template
* adapt template
* delete old template
* correct some links
* finish template
* create big bird from template
* add big bird
* improve boxes
* finish boxes
* add pointers for BigBird
* finish big bird
* up
* up
* up
* up
* apply lysandres and sylvains suggestions
* delete bogus file
* correct markdown
* try different style
* try different style
* finalize
2021-02-05 15:47:54 +03:00
Lysandre Debut
e89c959af9
Fix model templates ( #9999 )
2021-02-04 07:47:26 -05:00
demSd
00031785a8
BartForCausalLM analogs to ProphetNetForCausalLM ( #9128 )
...
* initiliaze bart4causalLM
* create BartDecoderWrapper, setters/getters
* delete spaces
* forward and additional methods
* update cache function, loss function, remove ngram* params in data class.
* add bartcausallm, bartdecoder testing
* correct bart for causal lm
* remove at
* add mbart as well
* up
* fix typo
* up
* correct
* add pegasusforcausallm
* add blenderbotforcausallm
* add blenderbotsmallforcausallm
* add marianforcausallm
* add test for MarianForCausalLM
* add Pegasus test
* add BlenderbotSmall test
* add blenderbot test
* fix a fail
* fix an import fail
* a fix
* fix
* Update modeling_pegasus.py
* fix models
* fix inputs_embeds setting getter
* adapt tests
* correct repo utils check
* finish test improvement
* fix tf models as well
* make style
* make fix-copies
* fix copies
* run all tests
* last changes
* fix all tests
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2021-02-04 11:56:12 +03:00
Patrick von Platen
0f4dc5d864
fix typo in naming ( #9944 )
2021-02-02 12:22:42 +03:00
Patrick von Platen
538b3b4607
[Tokenizer Utils Base] Make pad function more flexible ( #9928 )
...
* change tokenizer requirement
* split line
* Correct typo from list to str
* improve style
* make other function pretty as well
* add comment
* correct typo
* add new test
* pass tests for tok without padding token
* Apply suggestions from code review
2021-02-02 10:35:27 +03:00
Patrick von Platen
0e3be1ac8f
Add new model docs ( #9667 )
...
* add new model logic
* fix docs
* change structure
* improve add_new_model
* push new changes
* up
* up
* correct spelling
* improve docstring
* correct line length
* update readme
* correct links
* correct typos
* only add rst file for now
* Apply suggestions from code review 1
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Bram Vanroy <Bram.Vanroy@UGent.be>
* Apply suggestions from code review
Co-authored-by: Bram Vanroy <Bram.Vanroy@UGent.be>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Stefan Schweter <stefan@schweter.it>
Co-authored-by: Bram Vanroy <Bram.Vanroy@UGent.be>
* Apply suggestions from code review
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>
* finish adding all suggestions
* make style
* apply Niels feedback
* Apply suggestions from code review
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* apply sylvains suggestions
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Bram Vanroy <Bram.Vanroy@UGent.be>
Co-authored-by: Stefan Schweter <stefan@schweter.it>
Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2021-02-01 17:55:10 +03:00
Sylvain Gugger
d85691ac75
Doc title in the template ( #9910 )
2021-02-01 03:05:31 -05:00
Funtowicz Morgan
2ee9f9b69e
Fix computation of attention_probs when head_mask is provided. ( #9853 )
...
* Fix computation of attention_probs when head_mask is provided.
Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>
* Apply changes to the template
Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
2021-01-28 06:11:52 -05:00
Lysandre Debut
763ece2fea
Fix model templates ( #9842 )
2021-01-27 08:20:58 -05:00
Julien Plu
bd701ab1a0
Fix template ( #9840 )
2021-01-27 07:40:30 -05:00
Julien Plu
4adbdce5ee
Clean TF Bert ( #9788 )
...
* Start cleaning BERT
* Clean BERT and all those depends of it
* Fix attribute name
* Apply style
* Apply Sylvain's comments
* Apply Lysandre's comments
* remove unused import
2021-01-27 11:28:11 +01:00
Julien Plu
a1720694a5
Remove a TF usage warning and rework the documentation ( #9756 )
...
* Rework documentation
* Update the template
* Trigger CI
* Restore the warning but with the TF logger
* Update convbert doc
2021-01-27 10:45:42 +01:00
Lysandre
897a24c869
Fix head_mask for model templates
2021-01-26 11:02:48 +01:00
Julien Plu
a7dabfb3d1
Fix TF s2s models ( #9478 )
...
* Fix Seq2Seq models for serving
* Apply style
* Fix lonfgormer
* Fix mBart/Pegasus/Blenderbot
* Apply style
* Add a main intermediate layer
* Apply style
* Remove import
* Apply tf.function to Longformer
* Fix utils check_copy
* Update S2S template
* Fix BART + Blenderbot
* Fix BlenderbotSmall
* Fix BlenderbotSmall
* Fix BlenderbotSmall
* Fix MBart
* Fix Marian
* Fix Pegasus + template
* Apply style
* Fix common attributes test
* Forgot to fix the LED test
* Apply Patrick's comment on LED Decoder
2021-01-21 17:03:29 +01:00
Julien Plu
3f290e6c84
Fix mixed precision in TF models ( #9163 )
...
* Fix Gelu precision
* Fix gelu_fast
* Naming
* Fix usage and apply style
* add TF gelu approximate version
* add TF gelu approximate version
* add TF gelu approximate version
* Apply style
* Fix albert
* Remove the usage of the Activation layer
2021-01-21 07:00:11 -05:00
Julien Plu
7251a4736d
Fix template ( #9697 )
2021-01-20 09:04:53 -05:00
Julien Plu
14042d560f
New TF embeddings (cleaner and faster) ( #9418 )
...
* Create new embeddings + add to BERT
* Add Albert
* Add DistilBert
* Add Albert + Electra + Funnel
* Add Longformer + Lxmert
* Add last models
* Apply style
* Update the template
* Remove unused imports
* Rename attribute
* Import embeddings in their own model file
* Replace word_embeddings per weight
* fix naming
* Fix Albert
* Fix Albert
* Fix Longformer
* Fix Lxmert Mobilebert and MPNet
* Fix copy
* Fix template
* Update the get weights function
* Update src/transformers/modeling_tf_utils.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Update src/transformers/models/electra/modeling_tf_electra.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* address Sylvain's comments
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2021-01-20 12:08:12 +01:00
Sylvain Gugger
7e662e6a3b
Fix model templates and use less than 119 chars ( #9684 )
...
* Fix model templates and use less than 119 chars
* Missing new line
2021-01-19 17:11:22 -05:00
Yusuke Mori
b020a736c3
Update past_key_values in GPT-2 ( #9596 )
...
* Update past_key_values in gpt2 (#9391 )
* Update generation_utils, and rename some items
* Update modeling_gpt2 to avoid an error in gradient_checkpointing
* Remove 'reorder_cache' from util and add variations to XLNet, TransfoXL, GPT-2
* Change the location of '_reorder_cache' in modeling files
* Add '_reorder_cache' in modeling_ctrl
* Fix a bug of my last commit in CTRL
* Add '_reorder_cache' to GPT2DoubleHeadsModel
* Manage 'use_cache' in config of test_modeling_gpt2
* Clean up the doc string
* Update src/transformers/models/gpt2/modeling_gpt2.py
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
* Fix the doc string (GPT-2, CTRL)
* improve gradient_checkpointing_behavior
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
2021-01-19 16:00:15 +01:00
Patrick von Platen
7f28613213
[TFBart] Split TF-Bart ( #9497 )
...
* make templates ready
* make add_new_model_command_ready
* finish tf bart
* prepare tf mbart
* finish tf bart
* add tf mbart
* add marian
* prep pegasus
* add tf pegasus
* push blenderbot tf
* add blenderbot
* add blenderbot small
* clean-up
* make fix copy
* define blend bot tok
* fix
* up
* make style
* add to docs
* add copy statements
* overwrite changes
* improve
* fix docs
* finish
* fix last slow test
* fix missing git conflict line
* fix blenderbot
* up
* fix blenderbot small
* load changes
* finish copied from
* upload fix
2021-01-12 02:06:32 +01:00
Julien Plu
1e3c362235
Fix template ( #9512 )
2021-01-11 08:03:28 -05:00
Julien Plu
1243ee7d0c
Full rework of the TF input/output embeddings and bias resizing ( #9193 )
...
* Start rework resizing
* Rework bias/decoder resizing
* Full resizing rework
* Full resizing rework
* Start to update the models with the new approach
* Finish to update the models
* Update all the tests
* Update the template
* Fix tests
* Fix tests
* Test a new approach
* Refactoring
* Refactoring
* Refactoring
* New rework
* Rework BART
* Rework bert+blenderbot
* Rework CTRL
* Rework Distilbert
* Rework DPR
* Rework Electra
* Rework Flaubert
* Rework Funnel
* Rework GPT2
* Rework Longformer
* Rework Lxmert
* Rework marian+mbart
* Rework mobilebert
* Rework mpnet
* Rework openai
* Rework pegasus
* Rework Roberta
* Rework T5
* Rework xlm+xlnet
* Rework template
* Fix TFT5EncoderOnly + DPRs
* Restore previous methods
* Fix Funnel
* Fix CTRL and TransforXL
* Apply style
* Apply Sylvain's comments
* Restore a test in DPR
* Address the comments
* Fix bug
* Apply style
* remove unused import
* Fix test
* Forgot a method
* missing test
* Trigger CI
* naming update
* Rebase
* Trigger CI
2021-01-11 06:27:28 -05:00
Julien Plu
cf416764f4
Fix template ( #9504 )
2021-01-11 05:21:25 -05:00
Julien Plu
4f7022d68d
Reformat ( #9482 )
2021-01-10 15:10:15 +01:00
Sylvain Gugger
1bdf42409c
Fast imports part 3 ( #9474 )
...
* New intermediate inits
* Update template
* Avoid importing torch/tf/flax in tokenization unless necessary
* Styling
* Shutup flake8
* Better python version check
2021-01-08 07:40:59 -05:00
Sylvain Gugger
758ed3332b
Transformers fast import part 2 ( #9446 )
...
* Main init work
* Add version
* Change from absolute to relative imports
* Fix imports
* One more typo
* More typos
* Styling
* Make quality script pass
* Add necessary replace in template
* Fix typos
* Spaces are ignored in replace for some reason
* Forgot one models.
* Fixes for import
Co-authored-by: LysandreJik <lysandre.debut@reseau.eseo.fr>
* Add documentation
* Styling
Co-authored-by: LysandreJik <lysandre.debut@reseau.eseo.fr>
2021-01-07 09:36:14 -05:00
Julien Plu
812045adcc
New serving ( #9419 )
...
* Add a serving method
* Add albert
* Add serving for BERT and BART
* Add more models
* Finish the serving addition
* Temp fix
* Restore DPR
* Fix funnel attribute
* Fix attributes GPT2
* Fix OpenAIGPT attribute
* Fix T5 attributes
* Fix Bart attributes
* Fix TransfoXL attributes
* Add versioning
* better test
* Update template
* Fix Flaubert
* Fix T5
* Apply style
* Remove unused imports
* Deactivate extra parameters
* Remove too long test + saved_model default to False
* Ignore the saved model test for some models
* Fix some inputs
* Fix mpnet serving
* Trigger CI
* Address all comments
2021-01-07 11:48:49 +01:00
Patrick von Platen
b8462b5b2a
[GenerationOutputs] Fix GenerationOutputs Tests ( #9443 )
...
* fix generation models
* fix led
* fix docs
* add is_decoder
* fix last docstrings
* make style
* fix t5 cross attentions
* correct t5
2021-01-06 19:37:02 +01:00
Patrick von Platen
eef66035a2
[PyTorch Bart] Split Bart into different models ( #9343 )
...
* first try
* remove old template
* finish bart
* finish mbart
* delete unnecessary line
* init pegasus
* save intermediate
* correct pegasus
* finish pegasus
* remove cookie cutter leftover
* add marian
* finish blenderbot
* replace in file
* correctly split blenderbot
* delete "old" folder
* correct "add statement"
* adapt config for tf comp
* correct configs for tf
* remove ipdb
* fix more stuff
* fix mbart
* push pegasus fix
* fix mbart
* more fixes
* fix research projects code
* finish docs for bart, mbart, and marian
* delete unnecessary file
* correct attn typo
* correct configs
* remove pegasus for seq class
* correct peg docs
* correct peg docs
* finish configs
* further improve docs
* add copied from statements to mbart
* fix copied from in mbart
* add copy statements to marian
* add copied from to marian
* add pegasus copied from
* finish pegasus
* finish copied from
* Apply suggestions from code review
* make style
* backward comp blenderbot
* apply lysandres and sylvains suggestions
* apply suggestions
* push last fixes
* fix docs
* fix tok tests
* fix imports code style
* fix doc
2021-01-05 22:00:05 +01:00
Patrick von Platen
912f6881d2
add import math ( #9346 )
2020-12-29 19:35:06 +01:00
Patrick von Platen
785e52cd30
improve templates ( #9342 )
2020-12-29 16:48:44 +01:00
Patrick von Platen
83fdd252f6
[Seq2Seq Templates] Correct some TF-serving errors and add gradient checkpointing to PT by default. ( #9334 )
...
* correct tests
* correct shape and get_tf_activation
* more correction tf
* add gradient checkpointing to templates
* correct typo
2020-12-28 17:51:04 +01:00
Patrick von Platen
6c091abef2
[Templates] Adapt Bert ( #9284 )
...
* adapt templates
* adapt config
* add test as well
* fix output type
* fix cache false naming
* finish tests
* last fix
2020-12-24 01:44:33 +01:00
Patrick von Platen
d5db6c37d4
[Seq2Seq Templates] Fix check_repo.py templates file ( #9277 )
...
* add enc dec pt model to check repo
* fix indent
2020-12-23 11:40:20 +01:00
Patrick von Platen
cbe63949d7
Model Templates for Seq2Seq ( #9251 )
...
* adapt cookie cutter
* fix copy past statement
* delete copy statements for now
* remove unused import from template
* make doc rst
* correct config docstring
* correct training
* correct inputs processing tf enc dec
* make style
* adapt templates
* clean tabs
* correct tensor -> Tensor naming
* correct indent
* correct templates
* fix the test
* break lines to avoid > 119
* Apply suggestions from code review
2020-12-22 23:41:20 +01:00
Julien Plu
161a6461db
Fix TF template ( #9234 )
2020-12-21 13:52:16 +01:00
Julien Plu
5a8a4eb187
Improve BERT-like models performance with better self attention ( #9124 )
...
* Improve BERT-like models attention layers
* Apply style
* Put back error raising instead of assert
* Update template
* Fix copies
* Apply raising valueerror in MPNet
* Restore the copy check for the Intermediate layer in Longformer
* Update longformer
2020-12-21 13:10:15 +01:00
Lysandre Debut
6587cf9f84
Patch *ForCausalLM model ( #9092 )
2020-12-14 00:39:55 -05:00
Julien Plu
51d9c569fa
Fix embeddings resizing in TF models ( #8657 )
...
* Resize the biases in same time than the embeddings
* Trigger CI
* Biases are not reset anymore
* Remove get_output_embeddings + better LM model detection in generation utils
* Apply style
* First test on BERT
* Update docstring + new name
* Apply the new resizing logic to all the models
* fix tests
* Apply style
* Update the template
* Fix naming
* Fix naming
* Apply style
* Apply style
* Remove unused import
* Revert get_output_embeddings
* Trigger CI
* Update num parameters
* Restore get_output_embeddings in TFPretrainedModel and add comments
* Style
* Add decoder resizing
* Style
* Fix tests
* Separate bias and decoder resize
* Fix tests
* Fix tests
* Apply style
* Add bias resizing in MPNet
* Trigger CI
* Apply style
2020-12-13 23:05:24 -05:00
