Commit Graph

4 Commits

Author SHA1 Message Date
LysandreJik
e3fb4310d6 From pretrained correct initialization. Unknown token handling for gpt2. 2019-07-11 18:44:29 -04:00
thomwolf
b19786985d unified tokenizer api and serialization + tests 2019-07-09 10:25:18 +02:00
thomwolf
36bca545ff tokenization abstract class - tests for examples 2019-07-05 15:02:59 +02:00
thomwolf
0bab55d5d5 [BIG] name change 2019-07-05 11:55:36 +02:00