CL-ModelCloud 35c155052d Fix PretrainedTokenizerFast check => Fix PretrainedTokenizerFast Save (#35835) ... * Fix the bug in tokenizer.save_pretrained when saving tokenizer_class to tokenizer_config.json * Update tokenization_utils_base.py * Update tokenization_utils_base.py * Update tokenization_utils_base.py * add tokenizer class type test * code review * code opt * fix bug * Update test_tokenization_fast.py * ruff check * make style * code opt * Update test_tokenization_fast.py --------- Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai> Co-authored-by: LRL-ModelCloud <165116337+LRL-ModelCloud@users.noreply.github.com>		2025-02-13 12:00:33 +01:00
..
__init__.py	[Test refactor 1/5] Per-folder tests reorganization (#15725)	2022-02-23 15:46:28 -05:00
test_tokenization_fast.py	Fix PretrainedTokenizerFast check => Fix PretrainedTokenizerFast Save (#35835)	2025-02-13 12:00:33 +01:00
test_tokenization_utils.py	tokenizer train from iterator without pre_tokenizers (#35396)	2025-01-09 15:34:43 +01:00