riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity

Add chat_template for tokenizer extracted from GGUF model (#32908)

Browse Source 
* add chat_template to gguf tokenizer

* add template through tokenizer config
This commit is contained in:
Isotr0py 2024-08-22 22:41:25 +08:00 committed by GitHub
parent 99d67f1a09
commit ee8c01f839
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/tokenization_utils_fast.py
View File

@ -121,8 +121,10 @@ class PreTrainedTokenizerFast(PreTrainedTokenizerBase):
gguf_param = load_gguf_checkpoint(kwargs.get("vocab_file"))
architecture = gguf_param["config"]["model_type"]
tokenizer_dict = gguf_param["tokenizer"]
tokenizer_config = gguf_param["tokenizer_config"]
fast_tokenizer, additional_kwargs = convert_gguf_tokenizer(architecture, tokenizer_dict)
kwargs.update(tokenizer_config)
if len(additional_kwargs) > 0:
kwargs.update(additional_kwargs)