riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity

Assertions to exceptions (#13692)

Browse Source 
* Raise exceptions instead of using assertions for control flow #12789

* # coding=utf-8

* Raise exceptions instead of using assertions for control flow

* Raise exceptions instead of using assertions for control flow

* Update src/transformers/tokenization_utils.py

Raise exceptions instead of using assertions for control flow

Co-authored-by: Suraj Patil <surajp815@gmail.com>

* Update src/transformers/tokenization_utils.py

Raise exceptions instead of using assertions for control flow

Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>

* Raise exceptions instead of using assertions for control flow

* test

* Raise exceptions instead of using assertions for control flow

Co-authored-by: MocktaiLEngineer <kavinarasu22@gmail.com>
Co-authored-by: Suraj Patil <surajp815@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
This commit is contained in:
MocktaiLEngineer 2021-09-22 18:44:29 +05:30 committed by GitHub
parent 27d4639779
commit a0c08aa36c
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 2 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
src/transformers/tokenization_utils.py
View File

@ -362,7 +362,8 @@ class PreTrainedTokenizer(PreTrainedTokenizerBase):
tokens_to_add = []
for token in new_tokens:
assert isinstance(token, str)
if not isinstance(token, str):
raise TypeError(f"Token {token} is not a string but a {type(token)}.")
if not special_tokens and hasattr(self, "do_lower_case") and self.do_lower_case:
token = token.lower()
if (