SaulLu 1025a9b742

add a warning in SpmConverter for sentencepiece's model using the byte fallback feature (#16629) ...

* update proto sentencepiece model

* Revert "update proto sentencepiece model"

This reverts commit b07f671747.

* add check

* add test

* Revert "Revert "update proto sentencepiece model""

This reverts commit 46108257b8.

* test for log level

* test for log level 2

* warning at the warning level

* clean

* format

* add explanation in docstring

2022-04-11 11:06:10 +02:00

264 KiB

Raw History

View Raw