r-terada 2f53ab5745 Add sudachi and jumanpp tokenizers for bert_japanese (#19043) ... * add sudachipy and jumanpp tokenizers for bert_japanese * use ImportError instead of ModuleNotFoundError in SudachiTokenizer and JumanppTokenizer * put test cases of test_tokenization_bert_japanese in one line * add require_sudachi and require_jumanpp decorator for testing * add sudachi and pyknp(jumanpp) to dependencies * remove sudachi_dict_small and sudachi_dict_full from dependencies * empty commit for ci		2022-10-05 11:41:37 -04:00
..
__init__.py	Move test model folders (#17034)	2022-05-03 14:42:02 +02:00
test_tokenization_bert_japanese.py	Add sudachi and jumanpp tokenizers for bert_japanese (#19043)	2022-10-05 11:41:37 -04:00