mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-06 14:20:04 +06:00
a2dec768a2
* beit-flax * updated FLAX_BEIT_MLM_DOCSTRING * removed bool_masked_pos from classification * updated Copyright * code refactoring: x -> embeddings * updated test: rm from_pt * Update docs/source/model_doc/beit.rst * model code dtype updates and other changes according to review * relative_position_bias revert back to pytorch design
120 lines
6.5 KiB
ReStructuredText
120 lines
6.5 KiB
ReStructuredText
..
|
|
Copyright 2021 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
BEiT
|
|
-----------------------------------------------------------------------------------------------------------------------
|
|
|
|
Overview
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The BEiT model was proposed in `BEiT: BERT Pre-Training of Image Transformers <https://arxiv.org/abs/2106.08254>`__ by
|
|
Hangbo Bao, Li Dong and Furu Wei. Inspired by BERT, BEiT is the first paper that makes self-supervised pre-training of
|
|
Vision Transformers (ViTs) outperform supervised pre-training. Rather than pre-training the model to predict the class
|
|
of an image (as done in the `original ViT paper <https://arxiv.org/abs/2010.11929>`__), BEiT models are pre-trained to
|
|
predict visual tokens from the codebook of OpenAI's `DALL-E model <https://arxiv.org/abs/2102.12092>`__ given masked
|
|
patches.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation
|
|
from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image
|
|
modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image
|
|
patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first "tokenize" the original image into
|
|
visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training
|
|
objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we
|
|
directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder.
|
|
Experimental results on image classification and semantic segmentation show that our model achieves competitive results
|
|
with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K,
|
|
significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains
|
|
86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%).*
|
|
|
|
Tips:
|
|
|
|
- BEiT models are regular Vision Transformers, but pre-trained in a self-supervised way rather than supervised. They
|
|
outperform both the original model (ViT) as well as Data-efficient Image Transformers (DeiT) when fine-tuned on
|
|
ImageNet-1K and CIFAR-100.
|
|
- As the BEiT models expect each image to be of the same size (resolution), one can use
|
|
:class:`~transformers.BeitFeatureExtractor` to resize (or rescale) and normalize images for the model.
|
|
- Both the patch resolution and image resolution used during pre-training or fine-tuning are reflected in the name of
|
|
each checkpoint. For example, :obj:`microsoft/beit-base-patch16-224` refers to a base-sized architecture with patch
|
|
resolution of 16x16 and fine-tuning resolution of 224x224. All checkpoints can be found on the `hub
|
|
<https://huggingface.co/models?search=microsoft/beit>`__.
|
|
- The available checkpoints are either (1) pre-trained on `ImageNet-22k <http://www.image-net.org/>`__ (a collection of
|
|
14 million images and 22k classes) only, (2) also fine-tuned on ImageNet-22k or (3) also fine-tuned on `ImageNet-1k
|
|
<http://www.image-net.org/challenges/LSVRC/2012/>`__ (also referred to as ILSVRC 2012, a collection of 1.3 million
|
|
images and 1,000 classes).
|
|
- BEiT uses relative position embeddings, inspired by the T5 model. During pre-training, the authors shared the
|
|
relative position bias among the several self-attention layers. During fine-tuning, each layer's relative position
|
|
bias is initialized with the shared relative position bias obtained after pre-training. Note that, if one wants to
|
|
pre-train a model from scratch, one needs to either set the :obj:`use_relative_position_bias` or the
|
|
:obj:`use_relative_position_bias` attribute of :class:`~transformers.BeitConfig` to :obj:`True` in order to add
|
|
position embeddings.
|
|
|
|
This model was contributed by `nielsr <https://huggingface.co/nielsr>`__. The JAX/FLAX version of this model was
|
|
contributed by `kamalkraj <https://huggingface.co/kamalkraj>`__. The original code can be found `here
|
|
<https://github.com/microsoft/unilm/tree/master/beit>`__.
|
|
|
|
BeitConfig
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.BeitConfig
|
|
:members:
|
|
|
|
|
|
BeitFeatureExtractor
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.BeitFeatureExtractor
|
|
:members: __call__
|
|
|
|
|
|
BeitModel
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.BeitModel
|
|
:members: forward
|
|
|
|
|
|
BeitForMaskedImageModeling
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.BeitForMaskedImageModeling
|
|
:members: forward
|
|
|
|
|
|
BeitForImageClassification
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.BeitForImageClassification
|
|
:members: forward
|
|
|
|
|
|
FlaxBeitModel
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.FlaxBeitModel
|
|
:members: __call__
|
|
|
|
|
|
FlaxBeitForMaskedImageModeling
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.FlaxBeitForMaskedImageModeling
|
|
:members: __call__
|
|
|
|
|
|
FlaxBeitForImageClassification
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.FlaxBeitForImageClassification
|
|
:members: __call__
|