riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-20 04:58:22 +06:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity

Raise an exception if the pipeline allocator can't determine the tokenizer from the model.

Browse Source
This commit is contained in:
Morgan Funtowicz 2019-12-13 14:12:54 +01:00
parent be5bf7b81b
commit 28e64ad5a4
1 changed files with 6 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
transformers/pipelines.py
View File

@ -370,11 +370,12 @@ def pipeline(task: str, model, config: Optional[PretrainedConfig] = None, tokeni
Utility factory method to build pipeline. Utility factory method to build pipeline.
""" """
# Try to infer tokenizer from model name (if provided as str) # Try to infer tokenizer from model name (if provided as str)
if tokenizer is None and isinstance(model, str): if not isinstance(tokenizer, PreTrainedTokenizer):
tokenizer = model if not isinstance(model, str):
else: # Impossible to guest what is the right tokenizer here
# Impossible to guest what is the right tokenizer here raise Exception('Tokenizer cannot be None if provided model is a PreTrainedModel instance')
raise Exception('Tokenizer cannot be None if provided model is a PreTrainedModel instance') else:
tokenizer = model
tokenizer = tokenizer if isinstance(tokenizer, PreTrainedTokenizer) else AutoTokenizer.from_pretrained(tokenizer) tokenizer = tokenizer if isinstance(tokenizer, PreTrainedTokenizer) else AutoTokenizer.from_pretrained(tokenizer)