Arthur 8189977885 [Core Tokenization] Support a fix for spm fast models (#26678) ... * fix * last attempt * current work * fix forward compatibility * save all special tokens * current state * revert additional changes * updates * remove tokenizer.model * add a test and the fix * nit * revert one more break * fix typefield issue * quality * more tests * fix fields for FC * more nits? * new additional changes * how * some updates * the fix * where do we stand * nits * nits * revert unrelated changes * nits nits nits * styling * don't break llama just yet * revert llama changes * safe arg check * fixup * Add a test for T5 * Necessary changes * Tests passing, added tokens need to not be normalized. If the added tokens are normalized, it will the stripping which seems to be unwanted for a normal functioning * Add even more tests, when normalization is set to True (which does not work 😓 ) * Add even more tests, when normalization is set to True (which does not work 😓 ) * Update to main * nits * fmt * more and more test * comments * revert change as tests are failing * make the test more readble * nits * refactor the test * nit * updates * simplify * style * style * style convert slow * Update src/transformers/convert_slow_tokenizer.py		2024-01-18 12:31:54 +01:00
..
__init__.py	Move test model folders (#17034)	2022-05-03 14:42:02 +02:00
test_modeling_flax_t5.py	CI with num_hidden_layers=2 🚀🚀🚀 (#25266)	2023-08-02 20:22:36 +02:00
test_modeling_t5.py	fix assisted decoding assistant model inputs (#27503)	2023-11-27 14:23:54 +00:00
test_modeling_tf_t5.py	Conversation pipeline fixes (#26795)	2023-10-16 17:27:45 +01:00
test_tokenization_t5.py	[Core Tokenization] Support a fix for spm fast models (#26678)	2024-01-18 12:31:54 +01:00