10 Commits

Author	SHA1	Message	Date
Thomas Wolf	cd65c41a83	Merge branch 'master' into xlm-tokenization	2019-08-30 17:15:16 +02:00
thomwolf	69da972ace	added test and debug tokenizer configuration serialization	2019-08-30 17:09:36 +02:00
thomwolf	62df4ba59a	add dilbert tokenizer and tests	2019-08-28 12:22:56 +02:00
LysandreJik	634a3172d8	Added integration tests for sequence builders.	2019-08-12 15:14:15 -04:00
thomwolf	328afb7097	cleaning up tokenizer tests structure (at last) - last remaining ppb refs	2019-08-05 14:08:56 +02:00
thomwolf	15d8b1266c	update tokenizer - update squad example for xlnet	2019-07-15 17:30:42 +02:00
thomwolf	c079d7ddff	fix python 2 tests	2019-07-09 10:40:59 +02:00
thomwolf	b19786985d	unified tokenizer api and serialization + tests	2019-07-09 10:25:18 +02:00
thomwolf	36bca545ff	tokenization abstract class - tests for examples	2019-07-05 15:02:59 +02:00
thomwolf	0bab55d5d5	[BIG] name change	2019-07-05 11:55:36 +02:00