Julien Chaumond
|
4d6c93e923
|
Kill __main__
|
2019-12-27 22:55:22 -05:00 |
|
Julien Chaumond
|
9b2badf3c9
|
[cli] Update doc
|
2019-12-27 22:54:29 -05:00 |
|
Julien Chaumond
|
f78ebc22ad
|
[cli] Add ability to delete remote object
|
2019-12-27 22:53:49 -05:00 |
|
Anthony MOI
|
bfe870be65
|
Hotfix tokenizers version for sdist installs
|
2019-12-27 11:05:52 -05:00 |
|
Thomas Wolf
|
74ea432847
|
Merge pull request #2286 from adelevie/patch-2
Typo in tokenization_utils.py
|
2019-12-27 10:50:47 +01:00 |
|
Thomas Wolf
|
492bea9aa0
|
Merge pull request #2292 from patrickvonplaten/add_cached_past_for_language_generation
Add cached past for language generation
|
2019-12-27 10:33:27 +01:00 |
|
Thomas Wolf
|
e213900fa2
|
Merge pull request #2290 from patrickvonplaten/fix_typo_in_doc_for_language_generation
duplicated line for repeating_words_penalty_for_language_generation
|
2019-12-27 10:29:06 +01:00 |
|
Thomas Wolf
|
9f5f646442
|
Merge pull request #2211 from huggingface/fast-tokenizers
Fast tokenizers
|
2019-12-27 10:24:29 +01:00 |
|
Aymeric Augustin
|
9024b19994
|
Auto-format (fixes previous commit).
|
2019-12-27 10:13:52 +01:00 |
|
Aymeric Augustin
|
3233b58ad4
|
Quote square brackets in shell commands.
This ensures compatibility with zsh.
Fix #2316.
|
2019-12-27 08:50:25 +01:00 |
|
Anthony MOI
|
e6ec24fa88
|
Better added_tokens handling
|
2019-12-26 16:49:48 -05:00 |
|
Anthony MOI
|
599db139f9
|
Code style update
|
2019-12-26 15:13:30 -05:00 |
|
Anthony MOI
|
835b76a46f
|
Handle unk_token
As we discussed, this is handled here directly
cc @thomwolf
|
2019-12-26 14:42:55 -05:00 |
|
Anthony MOI
|
7ead04ce14
|
FastPreTrainedTokenizer => PreTrainedTokenizerFast
|
2019-12-26 14:39:39 -05:00 |
|
Anthony MOI
|
1f82a5d910
|
Update for changes in tokenizers API
|
2019-12-26 14:37:55 -05:00 |
|
Thomas Wolf
|
8c67b529f6
|
Merge pull request #2324 from kashif/patch-1
Typo in serving.py
|
2019-12-26 12:38:06 +01:00 |
|
Kashif Rasul
|
7211541ade
|
Typo in serving.py
|
2019-12-26 12:21:40 +01:00 |
|
patrickvonplaten
|
0f6017bee3
|
improve comments for examples
|
2019-12-26 00:35:11 +01:00 |
|
patrickvonplaten
|
87c8fca9bc
|
add example for ctrl text generation in docs
|
2019-12-26 00:29:19 +01:00 |
|
patrickvonplaten
|
88def24c45
|
merge conflicts - renamed to previous_token singular
|
2019-12-26 00:27:16 +01:00 |
|
patrickvonplaten
|
822f725a07
|
duplicated line for repeating_words_penalty_for_language_generation
|
2019-12-26 00:25:29 +01:00 |
|
patrickvonplaten
|
fc84bd5254
|
adapt style to predefined style layout
|
2019-12-25 23:32:44 +01:00 |
|
patrickvonplaten
|
deff792bb6
|
add prepare inputs for transfo_xl and xlnet
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
9398058e19
|
add easy tensor shape match test
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
90cda45e9e
|
add past re-ordering for beam search
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
6bca56fdb0
|
check for self.config.mem_len instead of self.mem_len in _do_output_past
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
365ccd0af2
|
make if statements cleaner for prepare_inputs_for_generation
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
d039c679d2
|
better naming for if statement
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
7e0c5c731a
|
changed do_output_past function to check for self.config.output_past instead of self.output_past
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
eeaa402cd4
|
rename comments
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
7bb4271291
|
remove ipdb debugging statements
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
267587c258
|
add and improve comments
|
2019-12-25 23:17:24 +01:00 |
|
patrickvonplaten
|
d891fd0ae0
|
add past hidden key states for more efficient language generation & add prepare_inputs for gpt2 and ctrl model
|
2019-12-25 23:17:24 +01:00 |
|
Thomas Wolf
|
aeef4823ab
|
Merge pull request #2303 from patrickvonplaten/fix_error_with_repetition_penalty
fix repetition penalty error in modeling_utils.py
|
2019-12-25 22:39:20 +01:00 |
|
Thomas Wolf
|
0412f3d929
|
Merge pull request #2291 from aaugustin/fix-flake8-F841
Fix F841 flake8 warning
|
2019-12-25 22:37:42 +01:00 |
|
Thomas Wolf
|
8742c95461
|
Merge pull request #2289 from patrickvonplaten/fix_effective_batch_size_lang_gen_xlm
fix bug in prepare inputs for language generation for xlm for effective batch_size > 1
|
2019-12-25 22:30:46 +01:00 |
|
Thomas Wolf
|
1240be3ed9
|
Merge pull request #2312 from vitaliyradchenko/fix_special_and_add_tokens_loading
Correct tokenization for special and added tokens
|
2019-12-25 20:52:30 +01:00 |
|
vitaliyradchenko
|
b262577d17
|
add special tokens to unique_added_tokens_encoder
|
2019-12-25 18:31:35 +02:00 |
|
vitaliyradchenko
|
83a2347952
|
fixed lack of added and special tokens
|
2019-12-25 18:03:19 +02:00 |
|
Thomas Wolf
|
cea04a2443
|
Merge pull request #2310 from ShnitzelKiller/scatter-unfix
revert erroneous fix #2276
|
2019-12-25 12:43:22 +01:00 |
|
James Noeckel
|
e1844d9a45
|
use positional arguments due to inconsistent API
|
2019-12-25 01:34:02 -08:00 |
|
James Noeckel
|
9fb7addd4d
|
revert erroneous fix
|
2019-12-24 22:26:09 -08:00 |
|
Anthony MOI
|
734d29b03d
|
tokenizers is now a real dependency
|
2019-12-24 13:32:41 -05:00 |
|
Anthony MOI
|
2818e50569
|
Add tests for fast tokenizers
|
2019-12-24 13:29:01 -05:00 |
|
Anthony MOI
|
31c56f2e0b
|
Fix style
|
2019-12-24 12:43:27 -05:00 |
|
Anthony MOI
|
951ae99bea
|
BertTokenizerFast
|
2019-12-24 12:24:24 -05:00 |
|
Anthony MOI
|
041eac2d6d
|
GPT2TokenizerFast
|
2019-12-24 12:24:14 -05:00 |
|
Anthony MOI
|
3471ff0d35
|
FastPreTrainedTokenizer
|
2019-12-24 12:23:30 -05:00 |
|
patrickvonplaten
|
18e5bdbec5
|
fix repetition penalty error in modeling_utils.py
|
2019-12-24 17:18:05 +01:00 |
|
patrickvonplaten
|
f18ac4c28e
|
fix sequence length for prepare_inputs for xlnet
|
2019-12-24 16:43:24 +01:00 |
|