riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity

[docs] Fix wrong newline in the middle of a paragraph (#6573)

Browse Source
This commit is contained in:
Romain Rigaux 2020-08-18 07:22:43 -07:00 committed by GitHub
parent 7659a8eb37
commit 5a5af22ed5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 0 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/source/preprocessing.rst
View File

@ -53,7 +53,6 @@ The tokenizer can decode a list of token ids in a proper sentence:
As you can see, the tokenizer automatically added some special tokens that the model expect. Not all model need special
tokens; for instance, if we had used` gtp2-medium` instead of `bert-base-cased` to create our tokenizer, we would have
seen the same sentence as the original one here. You can disable this behavior (which is only advised if you have added
those special tokens yourself) by passing ``add_special_tokens=False``.