riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity

Use _pad_token of pad_token_id

Browse Source 
Requesting pad_token_id would cause an error message when it is None. Use private _pad_token instead.
This commit is contained in:
Bram Vanroy 2020-01-29 09:34:37 +01:00 committed by Lysandre Debut
parent 9fde13a3ac
commit 83446a88d9
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/tokenization_utils.py
View File

@ -999,7 +999,7 @@ class PreTrainedTokenizer(object):
padded_value = value
# verify that the tokenizer has a pad_token_id
if key != "input_len" and self.pad_token_id is not None:
if key != "input_len" and self._pad_token is not None:
# Padding handle
padded_value = [
v + [self.pad_token_id if key == "input_ids" else 1] * (max_seq_len - len(v))