riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity

Fix the tokenization warning noted in #5505 (#5550)

Browse Source 
* fix warning

* style and quality
This commit is contained in:
Thomas Wolf 2020-07-06 17:15:25 +02:00 committed by GitHub
parent 1bbc28bee7
commit c473484087
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
src/transformers/tokenization_utils.py
View File

@ -588,8 +588,8 @@ class PreTrainedTokenizer(PreTrainedTokenizerBase):
first_ids,
second_ids,
add_special_tokens=add_special_tokens,
padding_strategy=PaddingStrategy.DO_NOT_PAD, # we pad in batch afterward
truncation_strategy=truncation_strategy,
padding=PaddingStrategy.DO_NOT_PAD.value, # we pad in batch afterward
truncation=truncation_strategy.value,
max_length=max_length,
stride=stride,
pad_to_multiple_of=None, # we pad in batch afterward