8 Commits

Author	SHA1	Message	Date
thomwolf	870b734bfd	added tokenizers serialization tests	2019-04-15 12:03:56 +02:00
thomwolf	3e65f255dc	add serialization semantics to tokenizers - fix transfo-xl tokenizer	2019-04-15 11:47:25 +02:00
thomwolf	5c85fc3977	fix typo - logger info	2019-03-06 10:05:21 +01:00
Catalin Voss	4a49c22584	Warn instead of raising in BERT and GPT-2 tokenizers as well, to allow for pre-caching of tokens	2019-03-05 12:31:45 -08:00
thomwolf	ab7f5d2943	simple	2019-02-18 11:33:54 +01:00
thomwolf	b450a7faf2	clean up tokenization - fix python 2 tests	2019-02-18 11:27:18 +01:00
thomwolf	d44db1145c	update readme	2019-02-18 11:12:09 +01:00
thomwolf	ffd623823d	adding gpt2	2019-02-17 23:38:51 +01:00