transformers/docs/source/en/perf_infer_cpu.md

CPU

CPUs are a viable and cost-effective inference option. With a few optimization methods, it is possible to achieve good performance with large models on CPUs. These methods include fusing kernels to reduce overhead and compiling your code to a faster intermediate format that can be deployed in production environments.

This guide will show you a few ways to optimize inference on a CPU.

Optimum

Optimum is a Hugging Face library focused on optimizing model performance across various hardware. It supports ONNX Runtime (ORT), a model accelerator, for a wide range of hardware and frameworks including CPUs.

Optimum provides the [~optimum.onnxruntime.ORTModel] class for loading ONNX models. For example, load the optimum/roberta-base-squad2 checkpoint for question answering inference. This checkpoint contains a model.onnx file.

from transformers import AutoTokenizer, pipeline
from optimum.onnxruntime import ORTModelForQuestionAnswering

onnx_qa = pipeline("question-answering", model="optimum/roberta-base-squad2", tokenizer="deepset/roberta-base-squad2")

question = "What's my name?"
context = "My name is Philipp and I live in Nuremberg."
pred = onnx_qa(question, context)

Tip

Optimum includes an Intel extension that provides additional optimizations such as quantization, pruning, and knowledge distillation for Intel CPUs. This extension also includes tools to convert models to OpenVINO, a toolkit for optimizing and deploying models, for even faster inference.

BetterTransformer

BetterTransformer is a fastpath execution of specialized Transformers functions directly on the hardware level such as a CPU. There are two main components of the fastpath execution.

• fusing multiple operations into a single kernel for faster and more efficient execution
• skipping unnecessary computation of padding tokens with nested tensors

Warning

BetterTransformer isn't supported for all models. Check this list to see whether a model supports BetterTransformer.

BetterTransformer is available through Optimum with [~PreTrainedModel.to_bettertransformer].

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigscience/bloom")
model = model.to_bettertransformer()

TorchScript

TorchScript is an intermediate PyTorch model format that can be run in non-Python environments, like C++, where performance is critical. Train a PyTorch model and convert it to a TorchScript function or module with torch.jit.trace. This function optimizes the model with just-in-time (JIT) compilation, and compared to the default eager mode, JIT-compiled models offer better inference performance.

Tip

Refer to the Introduction to PyTorch TorchScript tutorial for a gentle introduction to TorchScript.

On a CPU, enable torch.jit.trace with the --jit_mode_eval flag in [Trainer].

python examples/pytorch/question-answering/run_qa.py \
--model_name_or_path csarron/bert-base-uncased-squad-v1 \
--dataset_name squad \
--do_eval \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir /tmp/ \
--no_cuda \
--jit_mode_eval