8 Commits

Author	SHA1	Message	Date
thomwolf	ed4e542260	adding tests	2019-08-05 18:14:07 +02:00
thomwolf	328afb7097	cleaning up tokenizer tests structure (at last) - last remaining ppb refs	2019-08-05 14:08:56 +02:00
thomwolf	009273dbdd	big doc update [WIP]	2019-08-04 12:14:57 +02:00
thomwolf	ed7549bb1a	release version 1.0	2019-07-16 16:10:58 +02:00
thomwolf	ec07cf5a66	rewamp optimization	2019-07-11 14:48:22 +02:00
thomwolf	b19786985d	unified tokenizer api and serialization + tests	2019-07-09 10:25:18 +02:00
thomwolf	36bca545ff	tokenization abstract class - tests for examples	2019-07-05 15:02:59 +02:00
thomwolf	0bab55d5d5	[BIG] name change	2019-07-05 11:55:36 +02:00