Commit Graph

177 Commits

Author SHA1 Message Date
LysandreJik
3566d27919 Clarified PreTrainedModel.from_pretrained warning messages in documentation. 2019-08-08 19:04:34 -04:00
LysandreJik
fbd746bd06 Updated test architecture 2019-08-08 18:21:34 -04:00
LysandreJik
6c41a8f5dc Encode and Decode are back in the superclass. They now handle sentence pairs special tokens. 2019-08-08 18:20:32 -04:00
Julien Chaumond
e367ac469c [RoBERTa] Re-apply 39d72bcc7b
cc @lysandrejik
2019-08-08 11:26:11 -04:00
Julien Chaumond
9d0603148b [RoBERTa] RobertaForSequenceClassification + conversion 2019-08-08 11:24:54 -04:00
LysandreJik
f2b300df6b fix #976 2019-08-08 10:38:57 -04:00
LysandreJik
7df303f5ad fix #971 2019-08-08 10:36:26 -04:00
LysandreJik
d2cc6b101e Merge branch 'master' into RoBERTa 2019-08-08 09:42:05 -04:00
LysandreJik
39d72bcc7b Fixed the RoBERTa checkpoint conversion script according to the LM head refactoring. 2019-08-07 14:21:57 -04:00
LysandreJik
770043eea2 Sentence-pair tasks handling. Using common tests on RoBERTa. Forced push to fix indentation. 2019-08-07 12:53:19 -04:00
Thomas Wolf
7729ef7381
Merge pull request #955 from FeiWang96/master
Fix comment typo
2019-08-07 10:11:25 +02:00
Thomas Wolf
5c6ecf37e7
Merge pull request #958 from saket404/typo-fix
Fixed small typo
2019-08-07 10:10:20 +02:00
Thomas Wolf
822d6768eb
Merge pull request #962 from guotong1988/patch-1
Update modeling_xlnet.py
2019-08-07 10:09:20 +02:00
Thomas Wolf
7e6102ce74
Merge pull request #963 from guotong1988/patch-2
Update modeling_bert.py
2019-08-07 10:09:04 +02:00
wangfei
6ec1ee9ec2 Fix examples in docstring 2019-08-06 11:32:54 +08:00
wangfei
72622926e5 Fix examples in docstring 2019-08-06 11:32:41 +08:00
wangfei
f889e77b9c Fix examples of loading pretrained models in docstring 2019-08-06 11:30:35 +08:00
wangfei
beb03ec6c5 Fix examples of loading pretrained models in docstring 2019-08-06 11:24:46 +08:00
Thomas Wolf
d43dc48b34
Merge branch 'master' into auto_models 2019-08-05 19:17:35 +02:00
thomwolf
0b524b0848 remove derived classes for now 2019-08-05 19:08:19 +02:00
thomwolf
13936a9621 update doc and tests 2019-08-05 18:48:16 +02:00
thomwolf
ed4e542260 adding tests 2019-08-05 18:14:07 +02:00
thomwolf
3a126e73dd fix #950 2019-08-05 17:26:29 +02:00
thomwolf
b90e29d52c working on automodels 2019-08-05 16:06:34 +02:00
thomwolf
328afb7097 cleaning up tokenizer tests structure (at last) - last remaining ppb refs 2019-08-05 14:08:56 +02:00
Thomas Wolf
0e918707dc
Merge pull request #907 from dhpollack/fix_convert_to_tf
Fix convert to tf
2019-08-05 12:55:04 +02:00
Julien Chaumond
cb9db101c7 Python 2 must DIE 2019-08-04 22:04:15 -04:00
Julien Chaumond
05c083520a [RoBERTa] model conversion, inference, tests 🔥 2019-08-04 21:39:21 -04:00
雷打不动!
d7fd10568c
Update modeling_bert.py 2019-08-05 08:58:19 +08:00
雷打不动!
84eb699082
Update modeling_xlnet.py 2019-08-05 08:57:09 +08:00
thomwolf
00132b7a7a updating docs - adding few tests to tokenizers 2019-08-04 22:42:55 +02:00
thomwolf
009273dbdd big doc update [WIP] 2019-08-04 12:14:57 +02:00
Saket Khandelwal
836e513698 Fixed small typo 2019-08-04 16:05:10 +10:00
wangfei
a24f830604 Fix comment typo 2019-08-03 12:17:06 +08:00
David Pollack
c90119e543 spelling mistake 2019-07-29 16:56:02 +02:00
thomwolf
bfbe52ec39 cleaning up example docstrings 2019-07-27 20:25:39 +02:00
thomwolf
4cc1bf81ee typos 2019-07-27 12:08:21 +02:00
thomwolf
ac27548b25 fix unk_token test 2019-07-27 11:50:47 +02:00
thomwolf
c717d38573 dictionnary => dictionary 2019-07-26 23:30:48 +02:00
thomwolf
7b6e474c9a fix #901 2019-07-26 21:26:44 +02:00
thomwolf
632d711411 fix #908 2019-07-26 21:14:37 +02:00
thomwolf
27b0f86d36 clean up pretrained 2019-07-26 17:09:21 +02:00
thomwolf
57e54ec070 add unk_token to gpt2 2019-07-26 17:09:07 +02:00
thomwolf
ac42049c08 add auto models and auto tokenizer 2019-07-26 17:08:59 +02:00
David Pollack
09ecf225e9 fixed the fix. tf session madness. 2019-07-26 15:20:44 +02:00
David Pollack
edfd965ac8 fix convert_to_tf 2019-07-26 14:13:46 +02:00
Thomas Wolf
6219ad7216
Merge pull request #888 from rococode/patch-1
Update docs for parameter rename
2019-07-25 15:01:22 +02:00
Joel Grus
ae152cec09
make save_pretrained work with added tokens
right now it's dumping the *decoder* when it should be dumping the *encoder*. this fixes that.
2019-07-24 16:54:48 -07:00
rococo // Ron
66b15f73f0
Update docs for parameter rename
OpenAIGPTLMHeadModel now accepts `labels` instead of `lm_labels`
2019-07-24 11:27:08 -07:00
Thomas Wolf
067923d326
Merge pull request #873 from huggingface/identity_replacement
Add nn.Identity replacement for old PyTorch
2019-07-23 18:16:35 +02:00
Thomas Wolf
368670ac31
Merge pull request #866 from xanlsh/master
Rework how PreTrainedModel.from_pretrained handles its arguments
2019-07-23 18:05:30 +02:00
thomwolf
1383c7b87a Fix #869 2019-07-23 17:52:20 +02:00
Anish Moorthy
4fb56c7729 Remove unused *args parameter from PreTrainedConfig.from_pretrained 2019-07-23 10:43:01 -04:00
Anish Moorthy
e179c55490 Add docs for from_pretrained functions, rename return_unused_args 2019-07-23 10:43:01 -04:00
thomwolf
0740e63e49 updating schedules for state_dict saving 2019-07-23 15:57:18 +02:00
Thomas Wolf
c4bc66886d
Merge pull request #860 from Yiqing-Zhou/patch-1
read().splitlines() -> readlines()
2019-07-23 15:24:25 +02:00
Yiqing-Zhou
b1019d2a8e
token[-1] -> token.rstrip('\n') 2019-07-23 20:41:26 +08:00
thomwolf
0227b4a940 fix #827 2019-07-23 14:06:43 +02:00
Anish Moorthy
490ebbdcf7 Fix PretrainedModel.from_pretrained not passing cache_dir forward 2019-07-22 18:03:08 -04:00
Anish Moorthy
b8009cb0da Make PreTrainedModel.from_pretrained pass unused arguments to model 2019-07-22 18:03:08 -04:00
Yiqing-Zhou
bef0c629ca
fix
Remove '\n' before adding token into vocab
2019-07-22 22:30:49 +08:00
Yiqing-Zhou
897d0841be
read().splitlines() -> readlines()
splitlines() does not work as what we expect here for bert-base-chinese because there is a '\u2028' (unicode line seperator) token in vocab file. Value of '\u2028'.splitlines() is ['', ''].
Perhaps we should use readlines() instead.
2019-07-22 20:49:09 +08:00
Wei-Sheng Chin
c4e9615691 Fix a path so that test can run on Windows 2019-07-17 09:08:40 -07:00
Minho Ryu
cd8980e1f4
import sys twice 2019-07-17 18:12:01 +09:00
thomwolf
5fe0b378d8 adding missing docstring fix #793 2019-07-16 21:35:53 +02:00
thomwolf
ed7549bb1a release version 1.0 2019-07-16 16:10:58 +02:00
thomwolf
4acaa65068 model in evaluation mode by default after from_pretrained 2019-07-16 15:41:57 +02:00
thomwolf
f289e6cfe4 fix docstrings 2019-07-16 15:31:21 +02:00
thomwolf
9726b229cf model name typo 2019-07-16 15:17:45 +02:00
thomwolf
1849aa7d39 update readme and pretrained model weight files 2019-07-16 15:11:29 +02:00
thomwolf
f31154cb9d Merge branch 'xlnet' 2019-07-16 11:51:13 +02:00
thomwolf
1b35d05d4b update conversion scripts and __main__ 2019-07-16 09:41:55 +02:00
thomwolf
352e3ff998 added migration guide to readme 2019-07-16 09:03:49 +02:00
thomwolf
3b8b0e01bb update readme 2019-07-16 00:12:55 +02:00
thomwolf
e691fc0963 update QA models tests + run_generation 2019-07-15 17:45:24 +02:00
thomwolf
15d8b1266c update tokenizer - update squad example for xlnet 2019-07-15 17:30:42 +02:00
thomwolf
3b469cb422 updating squad for compatibility with XLNet 2019-07-15 15:28:37 +02:00
thomwolf
8ca767f13c clean up optimization 2019-07-15 13:49:07 +02:00
thomwolf
74a24f0fe9 clean up file_utils 2019-07-15 13:49:01 +02:00
thomwolf
ab49fafc04 update tokenization docstrings for #328 2019-07-15 12:51:23 +02:00
thomwolf
a9ab15174c fix #328 2019-07-15 12:42:12 +02:00
thomwolf
f7cd7392fd fixed tests 2019-07-15 12:32:19 +02:00
thomwolf
e28d8bde0d doc on base classes 2019-07-15 12:08:06 +02:00
thomwolf
44c985facd update doc for XLM and XLNet 2019-07-15 11:36:50 +02:00
thomwolf
0201d86015 added doc for transformer-xl 2019-07-15 10:11:09 +02:00
thomwolf
4cb489457f added doc for openai GPT 2019-07-15 09:58:01 +02:00
thomwolf
62b8eb43c1 fix add_start_docstrings on python 2 (removed) 2019-07-15 09:49:02 +02:00
thomwolf
5bc3d0cc5b added gpt2 doc 2019-07-15 09:40:05 +02:00
thomwolf
183fedfed5 fix doc on python2 2019-07-15 09:00:09 +02:00
thomwolf
2397f958f9 updating examples and doc 2019-07-14 23:20:10 +02:00
thomwolf
7d4b200e40 good quality generation example for GPT, GPT-2, Transfo-XL, XLNet 2019-07-13 15:25:03 +02:00
thomwolf
699bc7e86e fix gpt-2 unk token test 2019-07-12 11:46:57 +02:00
thomwolf
7442956361 save config file 2019-07-12 11:26:16 +02:00
Thomas Wolf
c57e9d946f
Merge branch 'xlnet' into embeddings 2019-07-12 11:10:14 +02:00
thomwolf
2918b7d2a0 updating tests 2019-07-12 10:57:58 +02:00
LysandreJik
3fbceed8d2 Fix layer reference loss + previous attempted fix 2019-07-11 22:29:55 -04:00
LysandreJik
6c2ee16c04 Test suite testing the tie_weights function as well as the resize_token_embeddings function.
Patched an issue relating to the tied weights I had introduced with the TorchScript addition.
Byte order mark management in TSV glue reading.
2019-07-11 22:09:16 -04:00
LysandreJik
e3fb4310d6 From pretrained correct initialization. Unknown token handling for gpt2. 2019-07-11 18:44:29 -04:00
thomwolf
bd404735a7 embeddings resizing + tie_weights 2019-07-12 00:02:49 +02:00
LysandreJik
50e62a4cb4 fix gpt/gpt-2 from pretrained 2019-07-11 16:50:21 -04:00