127 Commits

Author	SHA1	Message	Date
LysandreJik	22ac004a7c	Added documentation and changed parameters for special_tokens_sentences_pair.	2019-08-12 15:13:53 -04:00
Julien Chaumond	b3d83d68db	Fixup 9d0603148b	2019-08-12 12:28:55 -04:00
LysandreJik	75d5f98fd2	Roberta tokenization + fixed tests (py3 + py2).	2019-08-09 15:02:13 -04:00
LysandreJik	14e970c271	Tokenization encode/decode class-based sequence handling	2019-08-09 15:01:38 -04:00
LysandreJik	fbd746bd06	Updated test architecture	2019-08-08 18:21:34 -04:00
LysandreJik	6c41a8f5dc	Encode and Decode are back in the superclass. They now handle sentence pairs special tokens.	2019-08-08 18:20:32 -04:00
Julien Chaumond	e367ac469c	[RoBERTa] Re-apply 39d72bcc7b ... cc @lysandrejik	2019-08-08 11:26:11 -04:00
Julien Chaumond	9d0603148b	[RoBERTa] RobertaForSequenceClassification + conversion	2019-08-08 11:24:54 -04:00
LysandreJik	d2cc6b101e	Merge branch 'master' into RoBERTa	2019-08-08 09:42:05 -04:00
LysandreJik	39d72bcc7b	Fixed the RoBERTa checkpoint conversion script according to the LM head refactoring.	2019-08-07 14:21:57 -04:00
LysandreJik	770043eea2	Sentence-pair tasks handling. Using common tests on RoBERTa. Forced push to fix indentation.	2019-08-07 12:53:19 -04:00
Thomas Wolf	7729ef7381	Merge pull request #955 from FeiWang96/master ... Fix comment typo	2019-08-07 10:11:25 +02:00
Thomas Wolf	5c6ecf37e7	Merge pull request #958 from saket404/typo-fix ... Fixed small typo	2019-08-07 10:10:20 +02:00
Thomas Wolf	822d6768eb	Merge pull request #962 from guotong1988/patch-1 ... Update modeling_xlnet.py	2019-08-07 10:09:20 +02:00
Thomas Wolf	7e6102ce74	Merge pull request #963 from guotong1988/patch-2 ... Update modeling_bert.py	2019-08-07 10:09:04 +02:00
wangfei	6ec1ee9ec2	Fix examples in docstring	2019-08-06 11:32:54 +08:00
wangfei	72622926e5	Fix examples in docstring	2019-08-06 11:32:41 +08:00
wangfei	f889e77b9c	Fix examples of loading pretrained models in docstring	2019-08-06 11:30:35 +08:00
wangfei	beb03ec6c5	Fix examples of loading pretrained models in docstring	2019-08-06 11:24:46 +08:00
Thomas Wolf	d43dc48b34	Merge branch 'master' into auto_models	2019-08-05 19:17:35 +02:00
thomwolf	0b524b0848	remove derived classes for now	2019-08-05 19:08:19 +02:00
thomwolf	13936a9621	update doc and tests	2019-08-05 18:48:16 +02:00
thomwolf	ed4e542260	adding tests	2019-08-05 18:14:07 +02:00
thomwolf	3a126e73dd	fix #950	2019-08-05 17:26:29 +02:00
thomwolf	b90e29d52c	working on automodels	2019-08-05 16:06:34 +02:00
thomwolf	328afb7097	cleaning up tokenizer tests structure (at last) - last remaining ppb refs	2019-08-05 14:08:56 +02:00
Thomas Wolf	0e918707dc	Merge pull request #907 from dhpollack/fix_convert_to_tf ... Fix convert to tf	2019-08-05 12:55:04 +02:00
Julien Chaumond	cb9db101c7	Python 2 must DIE	2019-08-04 22:04:15 -04:00
Julien Chaumond	05c083520a	[RoBERTa] model conversion, inference, tests 🔥	2019-08-04 21:39:21 -04:00
雷打不动！	d7fd10568c	Update modeling_bert.py	2019-08-05 08:58:19 +08:00
雷打不动！	84eb699082	Update modeling_xlnet.py	2019-08-05 08:57:09 +08:00
thomwolf	00132b7a7a	updating docs - adding few tests to tokenizers	2019-08-04 22:42:55 +02:00
thomwolf	009273dbdd	big doc update [WIP]	2019-08-04 12:14:57 +02:00
Saket Khandelwal	836e513698	Fixed small typo	2019-08-04 16:05:10 +10:00
wangfei	a24f830604	Fix comment typo	2019-08-03 12:17:06 +08:00
David Pollack	c90119e543	spelling mistake	2019-07-29 16:56:02 +02:00
thomwolf	bfbe52ec39	cleaning up example docstrings	2019-07-27 20:25:39 +02:00
thomwolf	4cc1bf81ee	typos	2019-07-27 12:08:21 +02:00
thomwolf	ac27548b25	fix unk_token test	2019-07-27 11:50:47 +02:00
thomwolf	c717d38573	dictionnary => dictionary	2019-07-26 23:30:48 +02:00
thomwolf	7b6e474c9a	fix #901	2019-07-26 21:26:44 +02:00
thomwolf	632d711411	fix #908	2019-07-26 21:14:37 +02:00
thomwolf	27b0f86d36	clean up pretrained	2019-07-26 17:09:21 +02:00
thomwolf	57e54ec070	add unk_token to gpt2	2019-07-26 17:09:07 +02:00
thomwolf	ac42049c08	add auto models and auto tokenizer	2019-07-26 17:08:59 +02:00
David Pollack	09ecf225e9	fixed the fix. tf session madness.	2019-07-26 15:20:44 +02:00
David Pollack	edfd965ac8	fix convert_to_tf	2019-07-26 14:13:46 +02:00
Thomas Wolf	6219ad7216	Merge pull request #888 from rococode/patch-1 ... Update docs for parameter rename	2019-07-25 15:01:22 +02:00
Joel Grus	ae152cec09	make save_pretrained work with added tokens ... right now it's dumping the *decoder* when it should be dumping the *encoder*. this fixes that.	2019-07-24 16:54:48 -07:00
rococo // Ron	66b15f73f0	Update docs for parameter rename ... OpenAIGPTLMHeadModel now accepts `labels` instead of `lm_labels`	2019-07-24 11:27:08 -07:00