riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 23 Packages Projects Releases Wiki Activity

Updating char_to_token documentation to note behaviour when trim_offsets is True (#33919)

Browse Source 
Updating char_to_token documentation.
This commit is contained in:
Adam Pocock 2024-10-05 08:13:26 -04:00 committed by GitHub
parent 612065efeb
commit a7e4e1a77c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
src/transformers/tokenization_utils_base.py
View File

@ -598,7 +598,8 @@ class BatchEncoding(UserDict):
Returns:
`int`: Index of the token.
`int`: Index of the token, or None if the char index refers to a whitespace only token and whitespace is
trimmed with `trim_offsets=True`.
"""
if not self._encodings: