12 Commits

Author	SHA1	Message	Date
thomwolf	b3c6ee0ac1	tokenization updates	2019-04-15 14:24:52 +02:00
thomwolf	870b734bfd	added tokenizers serialization tests	2019-04-15 12:03:56 +02:00
thomwolf	3e65f255dc	add serialization semantics to tokenizers - fix transfo-xl tokenizer	2019-04-15 11:47:25 +02:00
thomwolf	5c85fc3977	fix typo - logger info	2019-03-06 10:05:21 +01:00
Yongbo Wang	2fdab323d1	typo	2019-02-20 21:11:06 +08:00
thomwolf	e8fe6b7140	adapting transfo tokenizer to transposed inputs	2019-02-11 13:30:04 +01:00
thomwolf	884ca81d87	transposing the inputs of Transformer-XL to have a unified interface	2019-02-11 13:19:59 +01:00
thomwolf	b514a60c36	added tests for OpenAI GPT and Transformer-XL tokenizers	2019-02-11 10:17:16 +01:00
thomwolf	448937c00d	python 2 compatibility	2019-02-06 00:07:46 +01:00
thomwolf	8831c68803	fixing various parts of model conversion, loading and weights sharing	2019-01-16 10:31:16 +01:00
thomwolf	a69ec2c722	improved corpus and tokenization conversion - added evaluation script	2019-01-15 23:17:46 +01:00
thomwolf	3a9c88377f	adding Transformer XL	2019-01-15 12:59:38 +01:00