Arthur 085ea7e56c [CodeLlamaTokenizer] Nit, update __init__ to make sure the AddedTokens are not normalized because they are special (#27359) ... * make sure tokens are properly initialized for codellama slow * add m ore pretrained models * style * test more tokenizers checkpoints		2023-11-09 10:15:10 +01:00
..
__init__.py	[CodeLlama] Add support for CodeLlama (#25740)	2023-08-25 18:57:40 +02:00
test_tokenization_code_llama.py	[CodeLlamaTokenizer] Nit, update __init__ to make sure the AddedTokens are not normalized because they are special (#27359)	2023-11-09 10:15:10 +01:00