riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity

Fix token pass (#24862)

Browse Source 
* Fix how token is passed along in from_pretrained for tokenizers

* It's actually not necessary
This commit is contained in:
Sylvain Gugger 2023-07-17 13:27:11 -04:00 committed by GitHub
parent f42a35e611
commit 49eb357564
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 0 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
src/transformers/tokenization_utils_base.py
View File

@ -1720,10 +1720,6 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
)
token = use_auth_token
if token is not None:
# change to `token` in a follow-up PR
kwargs["use_auth_token"] = token
user_agent = {"file_type": "tokenizer", "from_auto_class": from_auto_class, "is_fast": "Fast" in cls.__name__}
if from_pipeline is not None:
user_agent["using_pipeline"] = from_pipeline