Ita Zaporozhets 3f03c379d2 fix tiktoken convert to pass AddedToken to Tokenizer (#36566) ... * pass AddedToken to Tokenizer * ruff * handle dict for special tokens * option: test tokenizer from tiktoken same as fast * ruff * ruff		2025-03-20 11:26:49 +01:00
..
__init__.py	Move test model folders (#17034)	2022-05-03 14:42:02 +02:00
test_modeling_flax_gpt2.py	[tests] remove flax-pt equivalence and cross tests (#36283)	2025-02-19 15:13:27 +00:00
test_modeling_gpt2.py	Fix typos in tests (#36547)	2025-03-05 15:04:06 -08:00
test_modeling_tf_gpt2.py	Remove static pretrained maps from the library's internals (#29112)	2024-03-25 10:33:38 +01:00
test_tokenization_gpt2_tf.py	update ruff version (#30932)	2024-05-22 06:40:15 +02:00
test_tokenization_gpt2.py	fix tiktoken convert to pass AddedToken to Tokenizer (#36566)	2025-03-20 11:26:49 +01:00