riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-02 19:21:31 +06:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

Merge pull request #2191 from huggingface/fix_sp_np

Browse Source 
Numpy compatibility for sentence piece
This commit is contained in:
Thomas Wolf 2019-12-20 22:08:08 +01:00 committed by GitHub
commit a557836a70
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
transformers/tokenization_utils.py
View File

@ -1227,6 +1227,7 @@ class PreTrainedTokenizer(object):
return self._convert_id_to_token(ids)
tokens = []
for index in ids:
index = int(index)
if skip_special_tokens and index in self.all_special_ids:
continue
if index in self.added_tokens_decoder: