6 Commits

Author	SHA1	Message	Date
Patrick Lewis	78cf7b4ab4	added code to raise value error for bert tokenizer for covert_tokens_to_indices	2018-12-18 14:41:30 +00:00
thomwolf	757750d6f6	fix tests	2018-11-17 11:58:14 +01:00
thomwolf	1de35b624b	preparing for first release	2018-11-15 20:56:10 +01:00
Yaser Martinez Palenzuela	4d124baf8f	Add test for Chinese tokenization	2018-11-05 23:04:29 +01:00
thomwolf	3d291dea4a	clean up tests	2018-11-04 21:27:19 +01:00
thomwolf	f8276008df	update readme, file names, removing TF code, moving tests	2018-11-03 23:35:14 +01:00