Ita Zaporozhets 3d7c2f9dea #32184 save total_vocab_size (#32240) ... * save total_vocab_size = vocab_size + user added tokens to speed up operation * updating length when added_tokens_decoder is set * add test len(tokenizer)		2024-08-05 09:22:48 +02:00
..
__init__.py	[Test refactor 1/5] Per-folder tests reorganization (#15725)	2022-02-23 15:46:28 -05:00
test_tokenization_fast.py	Skip tests properly (#31308)	2024-06-26 21:59:08 +01:00
test_tokenization_utils.py	#32184 save total_vocab_size (#32240)	2024-08-05 09:22:48 +02:00