transformers/docs/source/en/model_doc/vit.md

Vision Transformer (ViT)

Vision Transformer (ViT) is a transformer adapted for computer vision tasks. An image is split into smaller fixed-sized patches which are treated as a sequence of tokens, similar to words for NLP tasks. ViT requires less resources to pretrain compared to convolutional architectures and its performance on large datasets can be transferred to smaller downstream tasks.

You can find all the original ViT checkpoints under the Google organization.

Tip

Click on the ViT models in the right sidebar for more examples of how to apply ViT to different computer vision tasks.

The example below demonstrates how to classify an image with [Pipeline] or the [AutoModel] class.

import torch
from transformers import pipeline

pipeline = pipeline(
    task="image-classification",
    model="google/vit-base-patch16-224",
    torch_dtype=torch.float16,
    device=0
)
pipeline(images="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")

import torch
import requests
from PIL import Image
from transformers import AutoModelForImageClassification, AutoImageProcessor

image_processor = AutoImageProcessor.from_pretrained(
    "google/vit-base-patch16-224",
    use_fast=True,
)
model = AutoModelForImageClassification.from_pretrained(
    "google/vit-base-patch16-224",
    torch_dtype=torch.float16,
    device_map="auto",
    attn_implementation="sdpa"
)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = image_processor(image, return_tensors="pt").to("cuda")

with torch.no_grad():
  logits = model(**inputs).logits
predicted_class_id = logits.argmax(dim=-1).item()

class_labels = model.config.id2label
predicted_class_label = class_labels[predicted_class_id]
print(f"The predicted class label is: {predicted_class_label}")

Notes

• The best results are obtained with supervised pretraining, and during fine-tuning, it may be better to use images with a resolution higher than 224x224.
• Use [ViTImageProcessorFast] to resize (or rescale) and normalize images to the expected size.
• The patch and image resolution are reflected in the checkpoint name. For example, google/vit-base-patch16-224, is the base-sized architecture with a patch resolution of 16x16 and fine-tuning resolution of 224x224.

ViTConfig

autodoc ViTConfig

ViTFeatureExtractor

autodoc ViTFeatureExtractor - call

ViTImageProcessor

autodoc ViTImageProcessor - preprocess

ViTImageProcessorFast

autodoc ViTImageProcessorFast - preprocess

ViTModel

autodoc ViTModel - forward

ViTForMaskedImageModeling

autodoc ViTForMaskedImageModeling - forward

ViTForImageClassification

autodoc ViTForImageClassification - forward

TFViTModel

autodoc TFViTModel - call

TFViTForImageClassification

autodoc TFViTForImageClassification - call

FlaxVitModel

autodoc FlaxViTModel - call

FlaxViTForImageClassification

autodoc FlaxViTForImageClassification - call