15 Commits

Author	SHA1	Message	Date
LysandreJik	6c41a8f5dc	Encode and Decode are back in the superclass. They now handle sentence pairs special tokens.	2019-08-08 18:20:32 -04:00
LysandreJik	d2cc6b101e	Merge branch 'master' into RoBERTa	2019-08-08 09:42:05 -04:00
LysandreJik	770043eea2	Sentence-pair tasks handling. Using common tests on RoBERTa. Forced push to fix indentation.	2019-08-07 12:53:19 -04:00
Thomas Wolf	d43dc48b34	Merge branch 'master' into auto_models	2019-08-05 19:17:35 +02:00
thomwolf	0b524b0848	remove derived classes for now	2019-08-05 19:08:19 +02:00
thomwolf	13936a9621	update doc and tests	2019-08-05 18:48:16 +02:00
thomwolf	ed4e542260	adding tests	2019-08-05 18:14:07 +02:00
thomwolf	328afb7097	cleaning up tokenizer tests structure (at last) - last remaining ppb refs	2019-08-05 14:08:56 +02:00
thomwolf	009273dbdd	big doc update [WIP]	2019-08-04 12:14:57 +02:00
thomwolf	632d711411	fix #908	2019-07-26 21:14:37 +02:00
thomwolf	ed7549bb1a	release version 1.0	2019-07-16 16:10:58 +02:00
thomwolf	ec07cf5a66	rewamp optimization	2019-07-11 14:48:22 +02:00
thomwolf	b19786985d	unified tokenizer api and serialization + tests	2019-07-09 10:25:18 +02:00
thomwolf	36bca545ff	tokenization abstract class - tests for examples	2019-07-05 15:02:59 +02:00
thomwolf	0bab55d5d5	[BIG] name change	2019-07-05 11:55:36 +02:00