riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-02 19:21:31 +06:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Add RemBert to AutoTokenizer (#13224)

Browse Source
This commit is contained in:
Lysandre Debut 2021-08-23 19:16:48 +02:00 committed by GitHub
parent f1bb6f0839
commit 2772d3e79d
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 7 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
src/transformers/models/auto/tokenization_auto.py
View File

@ -199,6 +199,13 @@ else:
"MBart50TokenizerFast" if is_tokenizers_available() else None,
),
),
(
"rembert",
(
"RemBertTokenizer" if is_sentencepiece_available() else None,
"RemBertTokenizerFast" if is_tokenizers_available() else None,
),
),
]
)