riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 10:12:23 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity

Fix blender bot missleading doc (#20301)

Browse Source 
* fix the doc to specify that add_prefix_space = False

* add correct expected output
This commit is contained in:
Arthur 2022-11-18 08:57:07 +01:00 committed by GitHub
parent df56c843be
commit 532e60bedf
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
src/transformers/models/blenderbot/tokenization_blenderbot.py
View File

@ -99,8 +99,9 @@ class BlenderbotTokenizer(PreTrainedTokenizer):
```
>>> from transformers import BlenderbotTokenizer
>>> tokenizer = BlenderbotTokenizer.from_pretrained("facebook/blenderbot-3B")
>>> tokenizer.add_prefix_space = False
>>> tokenizer("Hello world")['input_ids']
[6950, 1085, 2]
[47, 921, 86, 1085, 2]
>>> tokenizer(" Hello world")['input_ids']
[6950, 1085, 2]
```