Ita Zaporozhets 1e79eade41 SPLIT PR: add user defined symbols and control symbols (#31305) ... * PR SPLIT: moving origina changes for adding user defined symbols * adding gemma test and generalizing gemma converter * ruff * update common test * update serialization test * deberta v2 tests updates as rust version adds '.' as a user added token, so a space is not added * removing commented lines * applying feedback - user only added_tokens to add and check piece.type instead of trainer_spec for user_defined_symbols * add comment referencing sentencepiece		2024-06-21 01:48:10 -07:00
..
__init__.py
test_modeling_rembert.py	update ruff version (#30932)	2024-05-22 06:40:15 +02:00
test_modeling_tf_rembert.py	Speed up TF tests by reducing hidden layer counts (#24595)	2023-06-30 16:30:33 +01:00
test_tokenization_rembert.py	SPLIT PR: add user defined symbols and control symbols (#31305)	2024-06-21 01:48:10 -07:00