12 Commits

Author	SHA1	Message	Date
thomwolf	32da75486b	add tokenizer and tests	2019-06-21 11:09:51 +02:00
thomwolf	31d387604c	adding s3 model tests with --runslow	2019-04-17 11:58:27 +02:00
thomwolf	870b734bfd	added tokenizers serialization tests	2019-04-15 12:03:56 +02:00
thomwolf	2dd8f524f5	removing test for long sequences error following #337	2019-03-06 10:10:41 +01:00
thomwolf	ba9e4eb354	fix unicode in tokenization tests	2019-02-06 00:28:00 +01:00
thomwolf	448937c00d	python 2 compatibility	2019-02-06 00:07:46 +01:00
Patrick Lewis	78cf7b4ab4	added code to raise value error for bert tokenizer for covert_tokens_to_indices	2018-12-18 14:41:30 +00:00
thomwolf	757750d6f6	fix tests	2018-11-17 11:58:14 +01:00
thomwolf	1de35b624b	preparing for first release	2018-11-15 20:56:10 +01:00
Yaser Martinez Palenzuela	4d124baf8f	Add test for Chinese tokenization	2018-11-05 23:04:29 +01:00
thomwolf	3d291dea4a	clean up tests	2018-11-04 21:27:19 +01:00
thomwolf	f8276008df	update readme, file names, removing TF code, moving tests	2018-11-03 23:35:14 +01:00