riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity

fix #2410

Browse Source
This commit is contained in:
Lysandre 2020-01-07 15:20:45 +01:00
parent 176d3b3079
commit 6806f8204e
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/tokenization_xlm.py
View File

@ -474,7 +474,7 @@ def replace_unicode_punct(text):
text = text.replace("！", "!")
text = text.replace("（", "(")
text = text.replace("；", ";")
text = text.replace("１", '"')
text = text.replace("１", '1')
text = text.replace("」", '"')
text = text.replace("「", '"')
text = text.replace("０", "0")