Added Deberta model type support (#34308)

* Added Deberta model type for 'add_prefix_space' functionality

* housekeeping

---------

Co-authored-by: Filippos Ventirozos <filippos.ventirozos@autotrader.co.uk>
This commit is contained in:
Filippos Ventirozos 2024-10-23 10:15:36 +01:00 committed by GitHub
parent 5ba85de7a4
commit 343c8cb86f
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -366,7 +366,7 @@ def main():
)
tokenizer_name_or_path = model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path
if config.model_type in {"bloom", "gpt2", "roberta"}:
if config.model_type in {"bloom", "gpt2", "roberta", "deberta"}:
tokenizer = AutoTokenizer.from_pretrained(
tokenizer_name_or_path,
cache_dir=model_args.cache_dir,