3 Commits

Author	SHA1	Message	Date
Arthur	f9cc333805	[ PreTrainedTokenizerFast] Keep properties from fast tokenizer (#25053) ... * draft solution * use `setdefault` * nits * add tests and fix truncation issue * fix test * test passes locally * quality * updates * update tsets	2023-07-25 18:45:01 +02:00
SaulLu	ae7bae8fe7	fix train_new_from_iterator in the case of byte-level tokenizers (#17549)	2022-06-08 15:30:41 +02:00
Lysandre Debut	29c10a41d0	[Test refactor 1/5] Per-folder tests reorganization (#15725) ... * Per-folder tests reorganization Co-authored-by: sgugger <sylvain.gugger@gmail.com> Co-authored-by: Stas Bekman <stas@stason.org>	2022-02-23 15:46:28 -05:00