Stas Bekman
563485bf95
[tests] fix typos in inputs ( #6818 )
2020-08-30 18:19:57 +08:00
Sam Shleifer
3cac867fac
t5 model should make decoder_attention_mask ( #6800 )
2020-08-28 15:22:33 -04:00
Sam Shleifer
9336086ab5
prepare_seq2seq_batch makes labels/ decoder_input_ids made later. ( #6654 )
...
* broken test
* batch parity
* tests pass
* boom boom
* boom boom
* split out bart tokenizer tests
* fix tests
* boom boom
* Fixed dataset bug
* Fix marian
* Undo extra
* Get marian working
* Fix t5 tok tests
* Test passing
* Cleanup
* better assert msg
* require torch
* Fix mbart tests
* undo extra decoder_attn_mask change
* Fix import
* pegasus tokenizer can ignore src_lang kwargs
* unused kwarg test cov
* boom boom
* add todo for pegasus issue
* cover one word translation edge case
* Cleanup
* doc
2020-08-28 11:15:17 -04:00
Lysandre
a75c64d80c
Black 20 release
2020-08-26 17:20:22 +02:00
Sam Shleifer
624495706c
T5Tokenizer adds EOS token if not already added ( #5866 )
...
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
2020-08-25 14:56:08 -04:00
Suraj Patil
407da12ef1
[T5Tokenizer] add prepare_seq2seq_batch method ( #6122 )
...
* tests
2020-08-17 13:57:19 -04:00
Sam Shleifer
07dd7c2fd8
[cleanup] test_tokenization_common.py ( #4390 )
2020-05-19 10:46:55 -04:00
Julien Chaumond
83a41d39b3
💄 super
2020-01-15 18:33:50 -05:00
alberduris
81d6841b4b
GPU text generation: mMoved the encoded_prompt to correct device
2020-01-06 15:11:12 +01:00
alberduris
dd4df80f0b
Moved the encoded_prompts to correct device
2020-01-06 15:11:12 +01:00
Aymeric Augustin
c824d15aa1
Remove __future__ imports.
2019-12-22 17:47:54 +01:00
Aymeric Augustin
00204f2b4c
Replace CommonTestCases for tokenizers with a mixin.
...
This is the same change as for (TF)CommonTestCases for modeling.
2019-12-22 15:35:25 +01:00
Aymeric Augustin
a3c5883f2c
Rename file for consistency.
2019-12-22 15:35:25 +01:00
Aymeric Augustin
7e98e211f0
Remove unittest.main() in test modules.
...
This construct isn't used anymore these days.
Running python tests/test_foo.py puts the tests/ directory on
PYTHONPATH, which isn't representative of how we run tests.
Use python -m unittest tests/test_foo.py instead.
2019-12-22 14:42:03 +01:00
Aymeric Augustin
ced0a94204
Switch test files to the standard test_*.py scheme.
2019-12-22 14:15:13 +01:00
