riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-03 03:31:05 +06:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Add tokenizer_file parameter to PreTrainedTokenizerFast docstring (#12624)

Browse Source 
Co-authored-by: Lewis Bails <Lewis.Bails@infomedia.dk>
This commit is contained in:
Lewis Bails 2021-07-12 13:51:58 +02:00 committed by GitHub
parent f8f9a679a0
commit a882b9facb
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 3 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
src/transformers/tokenization_utils_fast.py
View File

@ -59,6 +59,9 @@ INIT_TOKENIZER_DOCSTRING += """
tokenizer_object (:class:`tokenizers.Tokenizer`):
A :class:`tokenizers.Tokenizer` object from 🤗 tokenizers to instantiate from. See :doc:`Using tokenizers
from 🤗 tokenizers <../fast_tokenizers>` for more information.
tokenizer_file (:class:`str`):
A path to a local JSON file representing a previously serialized :class:`tokenizers.Tokenizer` object from
🤗 tokenizers.
"""
MODEL_TO_TRAINER_MAPPING = {