Arthur b4d5548800 🚨🚨🚨 [SPM] Finish fix spm models 🚨🚨🚨 (#25224) ... * fix EVERYTHING * more fixes * ⚗️⚗️ Tokenizer magic ⚗️⚗️ * wrong value but test passes for the TODO * update * updat * safe protobuf import? * style * non gated repo * update * fixup * Update src/transformers/models/llama/tokenization_llama.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/llama/tokenization_llama.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update tests/models/t5/test_tokenization_t5.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * nits * fix t5 too * use assert equal * fix llama decoding * nits on t5 * fixup * only remove the prefix space, not other spaces * more deconding tests and more todos * fix CI as well * fixup * skip failing test on CI (its tf its ok) * skip test_subword_regularization_tokenizer that is also crashing on the CI for TF * update llama * revert good fixes * fixup * empty * explain why we need to encode with an additional token * better warning? * nits --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>		2023-08-17 17:08:05 +02:00
..
__init__.py	LLaMA Implementation (#21955)	2023-03-16 09:00:53 -04:00
test_modeling_llama.py	CI with num_hidden_layers=2 🚀🚀🚀 (#25266)	2023-08-02 20:22:36 +02:00
test_tokenization_llama.py	🚨🚨🚨 [SPM] Finish fix spm models 🚨🚨🚨 (#25224)	2023-08-17 17:08:05 +02:00