Michael Benayoun
0fe17f375a
FX tracing improvement ( #14321 )
...
* Change the way tracing happens, enabling dynamic axes out of the box
* Update the tests and modeling xlnet
* Add the non recoding of leaf modules to avoid recording more values for the methods to record than what will be seen at tracing time (which would otherwise desynchronize the recorded values and the values that need to be given to the proxies during tracing, causing errors).
* Comments and making tracing work for gpt-j and xlnet
* Refactore things related to num_choices (and batch_size, sequence_length)
* Update fx to work on PyTorch 1.10
* Postpone autowrap_function feature usage for later
* Add copyrights
* Remove unnecessary file
* Fix issue with add_new_model_like
* Apply suggestions
2022-02-07 22:25:33 +01:00
tucan9389
0f3f045ebd
Add GPTJForQuestionAnswering ( #14503 )
...
* Add GPTJForQuestionAnswering
* Reformat for GPTJForQuestionAnswering
* Fix isort error
* make style for GPTJForQA
* Add _keys_to_ignore_on_load_missing
* Change the sequence of qa and classification
Co-authored-by: Suraj Patil <surajp815@gmail.com>
2021-12-06 11:44:10 -05:00
Suraj Patil
4c0dd199c8
FlaxGPTJ ( #14396 )
...
* add flax gptj
* no bias in attention dense
* no wpe
* fix rotary embeddings
* fix rotary embeds
* fix rotray embeds
* quality
* doc and quality
* fix equivalence tests
2021-12-01 10:57:39 +05:30
Suraj Patil
ce91bf9a34
[GPTJ] enable common tests and few fixes ( #14190 )
...
* enable common tests, small fixes
* don't tie word embeds
* don't ignore lm_head
2021-11-01 22:38:52 +05:30
Thomas Wang
5b45422b58
Remove n_ctx from configs ( #14165 )
...
* Remove n_ctx from configs
* Fix GPTJ and OpenAIGPT, both are acceptable breaking changes as there are no configs such that it breaks
* Remove unecessary n_positions from TFOpenAIGPT
2021-10-29 11:50:25 +02:00
Suraj Patil
8bbb53e20b
skip gptj slow generate tests for now ( #13809 )
2021-09-30 15:44:33 -04:00
Anton Lozhkov
7c7d2ec952
[GPT-J] Use the float16 checkpoints in integration tests ( #13676 )
...
* Use fp16 checkpoints
* Style
* Fix outputs and disable OOM tests
* Correct another output
* Use a random smaller model for generation tests
* repo quickfix
* fix gradient checkpointing
2021-09-22 23:17:57 +03:00
Sylvain Gugger
27d4639779
Make gradient_checkpointing a training argument ( #13657 )
...
* Make gradient_checkpointing a training argument
* Update src/transformers/modeling_utils.py
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Update src/transformers/configuration_utils.py
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
* Fix tests
* Style
* document Gradient Checkpointing as a performance feature
* Small rename
* PoC for not using the config
* Adapt BC to new PoC
* Forgot to save
* Rollout changes to all other models
* Fix typo
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Stas Bekman <stas@stason.org>
2021-09-22 07:51:38 -04:00
Stella Biderman
c02cd95c56
GPT-J-6B ( #13022 )
...
* Test GPTJ implementation
* Fixed conflicts
* Update __init__.py
* Update __init__.py
* change GPT_J to GPTJ
* fix missing imports and typos
* use einops for now
(need to change to torch ops later)
* Use torch ops instead of einsum
* remove einops deps
* Update configuration_auto.py
* Added GPT J
* Update gptj.rst
* Update __init__.py
* Update test_modeling_gptj.py
* Added GPT J
* Changed configs to match GPT2 instead of GPT Neo
* Removed non-existent sequence model
* Update configuration_auto.py
* Update configuration_auto.py
* Update configuration_auto.py
* Update modeling_gptj.py
* Update modeling_gptj.py
* Progress on updating configs to agree with GPT2
* Update modeling_gptj.py
* num_layers -> n_layer
* layer_norm_eps -> layer_norm_epsilon
* attention_layers -> num_hidden_layers
* Update modeling_gptj.py
* attention_pdrop -> attn_pdrop
* hidden_act -> activation_function
* Update configuration_gptj.py
* Update configuration_gptj.py
* Update configuration_gptj.py
* Update configuration_gptj.py
* Update configuration_gptj.py
* Update modeling_gptj.py
* Update modeling_gptj.py
* Update modeling_gptj.py
* Update modeling_gptj.py
* Update modeling_gptj.py
* Update modeling_gptj.py
* fix layernorm and lm_head size
delete attn_type
* Update docs/source/model_doc/gptj.rst
Co-authored-by: Suraj Patil <surajp815@gmail.com>
* removed claim that GPT J uses local attention
* Removed GPTJForSequenceClassification
* Update src/transformers/models/gptj/configuration_gptj.py
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* Removed unsupported boilerplate
* Update tests/test_modeling_gptj.py
Co-authored-by: Suraj Patil <surajp815@gmail.com>
* Update src/transformers/models/gptj/modeling_gptj.py
Co-authored-by: Suraj Patil <surajp815@gmail.com>
* Update src/transformers/models/gptj/modeling_gptj.py
Co-authored-by: Suraj Patil <surajp815@gmail.com>
* Update src/transformers/models/gptj/modeling_gptj.py
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* Update tests/test_modeling_gptj.py
Co-authored-by: Eric Hallahan <eric@hallahans.name>
* Update tests/test_modeling_gptj.py
Co-authored-by: Eric Hallahan <eric@hallahans.name>
* Update tests/test_modeling_gptj.py
Co-authored-by: Eric Hallahan <eric@hallahans.name>
* Update src/transformers/models/gptj/modeling_gptj.py
Co-authored-by: Suraj Patil <surajp815@gmail.com>
* Update __init__.py
* Update configuration_gptj.py
* Update modeling_gptj.py
* Corrected indentation
* Remove stray backslash
* Delete .DS_Store
* Delete .DS_Store
* Delete .DS_Store
* Delete .DS_Store
* Delete .DS_Store
* Update docs to match
* Remove tf loading
* Remove config.jax
* Remove stray `else:` statement
* Remove references to `load_tf_weights_in_gptj`
* Adapt tests to match output from GPT-J 6B
* Apply suggestions from code review
Co-authored-by: Suraj Patil <surajp815@gmail.com>
* Default `activation_function` to `gelu_new`
- Specify the approximate formulation of GELU to ensure parity with the default setting of `jax.nn.gelu()`
* Fix part of the config documentation
* Revert "Update configuration_auto.py"
This reverts commit e9860e9c04cfaaae4c4d687788954f194d024ea8
2021-08-31 17:53:02 +02:00
