Alvaro Moran 7a5659872a Mitigate a conflict when using sentencepiece (#33327) ... * test(tokenizers): add a test showing conflict with sentencepiece This is due to the fact that protobuf C implementation uses a global pool for all added descriptors, so if two different files add descriptors, they will end up conflicting. * fix(tokenizers): mitigate sentencepiece/protobuf conflict When sentencepiece is available, use that protobuf instead of the internal one. * chore(style): fix with ruff		2024-09-13 13:19:06 +02:00
..
__init__.py	[Test refactor 1/5] Per-folder tests reorganization (#15725)	2022-02-23 15:46:28 -05:00
test_tokenization_fast.py	Skip tests properly (#31308)	2024-06-26 21:59:08 +01:00
test_tokenization_utils.py	Mitigate a conflict when using sentencepiece (#33327)	2024-09-13 13:19:06 +02:00