riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-19 20:48:22 +06:00
Code Issues Actions 18 Packages Projects Releases Wiki Activity

[VITS] Only trigger tokenizer warning for uroman (#25915)

Browse Source
This commit is contained in:
Sanchit Gandhi 2023-09-01 19:27:01 +01:00 committed by GitHub
parent 1fa2d89a9b
commit 3587769c08
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/models/vits/tokenization_vits.py
View File

@ -184,7 +184,7 @@ class VitsTokenizer(PreTrainedTokenizer):
filtered_text = self._preprocess_char(text)
if has_non_roman_characters(filtered_text):
if has_non_roman_characters(filtered_text) and self.is_uroman:
logger.warning(
"Text to the tokenizer contains non-Roman characters. Ensure the `uroman` Romanizer is "
"applied to the text prior to passing it to the tokenizer. See "