riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity

Fix docstring of train_new_from_iterator

Browse Source
This commit is contained in:
Omar Sanseviero 2021-08-13 17:38:02 +02:00 committed by GitHub
parent 86a154722f
commit a13c8145bc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/tokenization_utils_fast.py
View File

@ -587,7 +587,7 @@ class PreTrainedTokenizerFast(PreTrainedTokenizerBase):
text_iterator (generator of :obj:`List[str]`):
The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts
if you have everything in memory.
vocab_size (obj:`int`):
vocab_size (:obj:`int`):
The size of the vocabulary you want for your tokenizer.
new_special_tokens (list of :obj:`str` or :obj:`AddedToken`, `optional`):
A list of new special tokens to add to the tokenizer you are training.