Hao Wang 31565ff0fd Add sentencepiece to BertJapaneseTokenizer (#19769) ... * support sentencepiece for bertjapanesetokenizer * add test vocab file for sentencepiece, bertjapanesetokenizer * make BasicTokenizer be identical to transformers.models.bert.tokenization_bert.BasicTokenizer * fix missing of \n in comment * fix init argument missing in tests * make spm_file be optional, exclude spiece.model from tests/fixtures, and add description comments * make comment length less than 119 * apply doc style check		2022-10-21 10:04:49 -04:00
..
__init__.py	Move test model folders (#17034)	2022-05-03 14:42:02 +02:00
test_tokenization_bert_japanese.py	Add sentencepiece to BertJapaneseTokenizer (#19769)	2022-10-21 10:04:49 -04:00