riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-25 07:18:58 +06:00
Code Issues Actions 32 Packages Projects Releases Wiki Activity

update function to add special tokens

Browse Source 
Since I started my PR the `add_special_token_single_sequence` function
has been deprecated for another; I replaced it with the new function.
This commit is contained in:
Rémi Louf 2019-10-31 09:48:27 +01:00 committed by Julien Chaumond
parent 5c877fe94a
commit 07f4cd73f6
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
examples/utils_summarization.py
View File

@ -139,11 +139,11 @@ def encode_for_summarization(story_lines, summary_lines, tokenizer):
sentences. sentences.
""" """
story_lines_token_ids = [ story_lines_token_ids = [
tokenizer.add_special_tokens_single_sequence(tokenizer.encode(line)) tokenizer.build_inputs_with_special_tokens(tokenizer.encode(line))
for line in story_lines for line in story_lines
] ]
summary_lines_token_ids = [ summary_lines_token_ids = [
tokenizer.add_special_tokens_single_sequence(tokenizer.encode(line)) tokenizer.build_inputs_with_special_tokens(tokenizer.encode(line))
for line in summary_lines for line in summary_lines
] ]