riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-03 03:31:05 +06:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Fix for tokenizer.apply_chat_template with continue_final_message=True (#34214)

Browse Source 
* Strip final message

* Do full strip instead of rstrip

* Retrigger CI

---------

Co-authored-by: Matt <rocketknight1@gmail.com>
This commit is contained in:
Sebastian Schoennenbeck 2024-10-17 16:45:07 +02:00 committed by GitHub
parent b57c7bce21
commit f2846ad2b7
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/transformers/tokenization_utils_base.py
View File

@ -1874,7 +1874,7 @@ class PreTrainedTokenizerBase(SpecialTokensMixin, PushToHubMixin):
**template_kwargs,
)
if continue_final_message:
final_message = chat[-1]["content"]
final_message = chat[-1]["content"].strip()
rendered_chat = rendered_chat[: rendered_chat.rindex(final_message) + len(final_message)].rstrip()
rendered.append(rendered_chat)