riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-04 05:10:06 +06:00
Code Issues Actions 20 Packages Projects Releases Wiki Activity

[docs] add the missing tokenizer when pushing models to huggingface hub (#33428)

Browse Source 
* add tokenizer

* typo
This commit is contained in:
Fanli Lin 2024-09-12 00:56:55 +08:00 committed by GitHub
parent c403441339
commit cea9ec086a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/source/en/tasks/language_modeling.md
View File

@ -253,6 +253,7 @@ At this point, only three steps remain:
... train_dataset=lm_dataset["train"], ... train_dataset=lm_dataset["train"],
... eval_dataset=lm_dataset["test"], ... eval_dataset=lm_dataset["test"],
... data_collator=data_collator, ... data_collator=data_collator,
... tokenizer=tokenizer,
... ) ... )
>>> trainer.train() >>> trainer.train()

1
docs/source/en/tasks/masked_language_modeling.md
View File

@ -245,6 +245,7 @@ At this point, only three steps remain:
... train_dataset=lm_dataset["train"], ... train_dataset=lm_dataset["train"],
... eval_dataset=lm_dataset["test"], ... eval_dataset=lm_dataset["test"],
... data_collator=data_collator, ... data_collator=data_collator,
... tokenizer=tokenizer,
... ) ... )
>>> trainer.train() >>> trainer.train()