Hiroshi Matsuda da20209dbc Add sudachi_projection option to BertJapaneseTokenizer (#28503) ... * add sudachi_projection option * Upgrade sudachipy>=0.6.8 * add a test case for sudachi_projection * Compatible with older versions of SudachiPy * make fixup * make style * error message for unidic download * revert jumanpp test cases * format options for sudachi_projection Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * format options for sudachi_split_mode and sudachi_dict_type * comment * add tests for full_tokenizer kwargs * pass projection arg directly * require_sudachi_projection * make style * revert upgrade sudachipy * check is_sudachi_projection_available() * revert dependency_version_table and bugfix * style format * simply raise ImportError Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * simply raise ImportError --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>		2024-02-13 04:47:20 +01:00
..
__init__.py	Move test model folders (#17034)	2022-05-03 14:42:02 +02:00
test_tokenization_bert_japanese.py	Add sudachi_projection option to BertJapaneseTokenizer (#28503)	2024-02-13 04:47:20 +01:00