riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-01 18:51:14 +06:00
Code Issues Actions 3 Packages Projects Releases Wiki Activity

Fix typo in LlamaTokenizerFast docstring example (#25018)

Browse Source
This commit is contained in:
Sören Brunk 2023-07-24 15:37:58 +02:00 committed by GitHub
parent 3b734f5042
commit 8f1f0bf50f
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/models/llama/tokenization_llama_fast.py
View File

@ -58,7 +58,7 @@ class LlamaTokenizerFast(PreTrainedTokenizerFast):
``` ```
from transformers import LlamaTokenizerFast from transformers import LlamaTokenizerFast
tokenizer = LlaTokenizerFast.from_pretrained("hf-internal-testing/llama-tokenizer") tokenizer = LlamaTokenizerFast.from_pretrained("hf-internal-testing/llama-tokenizer")
tokenizer.encode("Hello this is a test") tokenizer.encode("Hello this is a test")
>>> [1, 15043, 445, 338, 263, 1243] >>> [1, 15043, 445, 338, 263, 1243]
``` ```