riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity
7,637 Commits 887 Branches 232 Tags 2.1 GiB
795c1444e9
Commit Graph

7637 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Nicolas Patry
795c1444e9
Improving pipeline tests (#12784) 
* Proposal

* Testing pipelines slightly better.

- Overall same design
- Metaclass to get proper different tests instead of subTest (not well
supported by Pytest)
- Added ANY meta object to make output checking more readable.
- Skipping architectures either without tiny_config or without
architecture.

* Small fix.

* Fixing the tests in case of None value.

* Oups.

* Rebased with more architectures.

* Fixing reformer tests (no override anymore).

* Adding more options for model tester config.

Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr>
 2021-07-22 15:19:35 +02:00
Lysandre
40de2d5a4f Docs for v4.10.0dev0 2021-07-22 12:52:25 +02:00
Lysandre
72aee83ced Release: v4.9.0 2021-07-22 12:11:55 +02:00
Maxwell Forbes
fcf83011df
Fix type of max_seq_length arg in run_swag.py (#12832) 2021-07-22 02:14:14 -04:00
Stas Bekman
27a8c9e4f1
[parallelism doc] document Deepspeed-Inference and parallelformers (#12836) 
* document Deepspeed-Inference and parallelformers

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-07-21 15:11:02 -07:00
Stas Bekman
807b6bd160
[Deepspeed] warmup_ratio docs (#12830) 
* [Deepspeed] warmup_ratio docs

* Update docs/source/main_classes/deepspeed.rst

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* style

* Update docs/source/main_classes/deepspeed.rst

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* style

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-07-21 10:49:29 -07:00
Sylvain Gugger
8c2384d8e2
Raise warning in HP search when hp is not in args (#12831) 2021-07-21 12:44:41 -04:00
Stas Bekman
cf0755aa6e
[debug] DebugUnderflowOverflow doesn't work with DP (#12816) 2021-07-21 09:36:02 -07:00
Lysandre Debut
ac3cb660ca
Add _CHECKPOINT_FOR_DOC to all models (#12811) 
* Add _CHECKPOINT_FOR_DOC

* Update src/transformers/models/funnel/modeling_funnel.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-07-21 08:29:43 -04:00
Sylvain Gugger
786ced3639
Add versioning system to fast tokenizer files (#12713) 
* Add versioning system to fast tokenizer files

* Deal with offline mode

* Use staging env in tests

* Style

* Apply suggestions from code review

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* Style

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
 2021-07-21 08:24:36 -04:00
Masatoshi TSUCHIYA
037bdf82d3
Refer warmup_ratio when setting warmup_num_steps. (#12818) 
* Refer warmup_ratio when setting warmup_num_steps.

* Add a method to get number of warmup steps to TrainerArguments class.

* Fix.

* Fix.
 2021-07-21 06:37:49 -04:00
Philip May
15d19ecfda
fix convert_tokens_to_string calls (#11716) 2021-07-21 04:28:30 -04:00
Lysandre Debut
c3d9ac7607
Expose get_config() on ModelTesters (#12812) 
* Expose get_config() on ModelTesters

* Typo
 2021-07-21 04:13:11 -04:00
Stas Bekman
cabcc75171
[trainer] sanity checks for save_steps=0|None and logging_steps=0 (#12796) 
* [trainer] fix % 0

* sanity checks

* fix logging_strategy

* correction

* Update src/transformers/training_args.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-07-20 09:05:26 -07:00
Patrick von Platen
acdd78db08
Update README.md 2021-07-20 16:48:37 +02:00
Suraj Patil
b5b4e54920
add and fix examples (#12810) 2021-07-20 09:28:50 -04:00
Patrick von Platen
31d06729f4
Update README.md 2021-07-20 14:19:37 +02:00
Patrick von Platen
2955d50e0c
[Longformer] Correct longformer docs (#12809) 
* fix_torch_device_generate_test

* remove @

* correct longformer docs

Co-authored-by: Patrick von Platen <patrick@huggingface.co>
 2021-07-20 14:17:21 +02:00
Patrick von Platen
13fefdf340
Update README.md 
cc @patil-suraj
 2021-07-20 13:51:15 +02:00
fgaim
66197adc98
Flax MLM: Allow validation split when loading dataset from local file (#12689) 
* Allow validation split when loading dataset from local file

* Flax clm & t5, enable validation split for datasets loaded from local file
 2021-07-20 13:38:25 +02:00
Will Rice
6f8e367ae9
Fix Padded Batch Error 12282 (#12487) 
This fixes the padded batch [issue](https://github.com/huggingface/transformers/issues/12282). The error was generated due to the maximum sequence length of the attention mask not matching the padded sequence length of the hidden_states. `np.allclose` now passes with a 1e-2 absolute tolerance.

This change fixes
 2021-07-20 13:36:47 +02:00
Stas Bekman
7fae535052
add troubleshooting docs (#12791) 2021-07-20 03:32:02 -04:00
Sylvain Gugger
0118ef89ee
Enforce eval and save strategies are compatible when --load_best_model_at_end (#12786) 
* Enforce eval and save strategies are compatible when --load_best_model_at_end

* Update doc

* Fix typos

* Fix tests
 2021-07-19 19:50:47 +02:00
Lysandre Debut
546dc24e08
Longer timeout for slow tests (#12779) 2021-07-19 04:55:40 -04:00
Antoni Baum
cab3b86892
[ray] Fix datasets_modules ImportError with Ray Tune (#12749) 
* Fix dynamic_modules ImportError with Ray Tune

* Nit
 2021-07-19 04:32:40 -04:00
Patrick von Platen
534f6eb9f1
Create README.md 2021-07-17 19:22:37 +02:00
Patrick von Platen
c6b9095cb2
Update README.md 2021-07-17 19:22:26 +02:00
Sylvain Gugger
da72ac6e26
Fix push_to_hub docstring and make it appear in doc (#12770) 2021-07-17 15:52:33 +02:00
Tomohiro Endo
08d609bfb8
Add tokenizers class mismatch detection between cls and checkpoint (#12619) 
* Detect mismatch by analyzing config

* Fix comment

* Fix import

* Update src/transformers/tokenization_utils_base.py

Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>

* Revise based on reviews

* remove kwargs

* Fix exception

* Fix handling exception again

* Disable mismatch test in PreTrainedTokenizerFast

Co-authored-by: SaulLu <55560583+SaulLu@users.noreply.github.com>
 2021-07-17 15:52:21 +02:00
Patrick von Platen
b4b562d834
[Wav2Vec2] Padded vectors should not allowed to be sampled (#12764) 
* fix_torch_device_generate_test

* remove @

* finish

* correct script

* correct script
 2021-07-16 19:07:08 +02:00
SaulLu
6e87010060
Preserve list type of additional_special_tokens in special_token_map (#12759) 
* preserve type of `additional_special_tokens` in `special_token_map`

* format

* Update src/transformers/tokenization_utils_base.py

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-07-16 18:26:54 +02:00
Funtowicz Morgan
fbf1397bf8
Turn on eval mode when exporting to ONNX (#12758) 
* Set model in eval mode when exporting to ONNX.

* Disable t5 for now.

* Disable T5 with past too.

* Style.
 2021-07-16 15:09:15 +02:00
Suraj Patil
8ef3f36561
fix typos (#12757) 2021-07-16 16:44:59 +05:30
Nathan Zhou
c07334c12e
add intel-tensorflow-avx512 to the candidates (#12751) 2021-07-16 05:54:49 -04:00
Stas Bekman
6989264963
[doc] testing: how to trigger a self-push workflow (#12724) 
* [testing] details of how to start self-push workflow

* style

* fix

* Apply suggestions from code review

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
 2021-07-15 16:18:56 -07:00
Patrick von Platen
a76dd7ee82
Update README.md 2021-07-16 00:16:30 +01:00
Patrick von Platen
2e9fb13fb1
[Wav2Vec2] Correctly pad mask indices for PreTraining (#12748) 
* fix_torch_device_generate_test

* remove @

* start adding tests

* correct wav2vec2 pretraining

* up

* up

Co-authored-by: Patrick von Platen <patrick@huggingface.co>
 2021-07-15 21:40:25 +01:00
SaulLu
5f2791c7c1
Replace specific tokenizer in log message by AutoTokenizer (#12745) 2021-07-15 12:59:48 -04:00
Stas Bekman
31cfcbd3e2
[doc] performance: batch sizes (#12725) 2021-07-15 09:39:34 -07:00
Stas Bekman
68605e9db1
[doc] parallelism: Which Strategy To Use When (#12712) 2021-07-15 09:38:51 -07:00
Lysandre Debut
eb4d7ef97b
Remove framework mention (#12731) 2021-07-15 11:49:02 -04:00
Lysandre Debut
959d448b3f
Fix led torchscript (#12735) 
* Don't test LED on torchscript

* Typo
 2021-07-15 11:48:50 -04:00
Lysandre Debut
f03580fb02
Fix DETR integration test (#12734) 2021-07-15 11:48:37 -04:00
Lysandre Debut
f42d9dcc0e
Patch T5 device test (#12742) 2021-07-15 16:40:17 +01:00
Lysandre Debut
370be9cc38
Fix MBart failing test (#12737) 2021-07-15 16:39:35 +01:00
qqaatw
2349ac58c4
Translate README.md to Traditional Chinese (#12701) 
* Add README_zh-tw.md

* Add links to each README.

* Fix a mismatched term.

* Minor improvements.

* Rename language code to be more inclusive.

* Polish terms to make them fluent.

* Remove redundant spaces.

* Fix typo.
 2021-07-15 23:35:39 +08:00
Lysandre Debut
eb2e006b35
Skip test while the model is not available (#12740) 2021-07-15 09:14:12 -04:00
Lysandre Debut
8c7bd1b97b
Skip test while the model is not available (#12739) 2021-07-15 09:06:47 -04:00
Lysandre Debut
3290315a2a
Fix AutoModel tests (#12733) 2021-07-15 09:06:12 -04:00
Lysandre Debut
01cb2f25e3
LXMERT integration test typo (#12736) 2021-07-15 08:29:49 -04:00