mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-07 14:50:07 +06:00
17a7b49bda
* Update doc example feature extractor -> image processor * Apply suggestions from code review
82 lines
4.7 KiB
Plaintext
82 lines
4.7 KiB
Plaintext
<!--Copyright 2022 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
-->
|
|
|
|
# ViTMAE
|
|
|
|
## Overview
|
|
|
|
The ViTMAE model was proposed in [Masked Autoencoders Are Scalable Vision Learners](https://arxiv.org/abs/2111.06377v2) by Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li,
|
|
Piotr Dollár, Ross Girshick. The paper shows that, by pre-training a Vision Transformer (ViT) to reconstruct pixel values for masked patches, one can get results after
|
|
fine-tuning that outperform supervised pre-training.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*This paper shows that masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the
|
|
input image and reconstruct the missing pixels. It is based on two core designs. First, we develop an asymmetric encoder-decoder architecture, with an encoder that operates
|
|
only on the visible subset of patches (without mask tokens), along with a lightweight decoder that reconstructs the original image from the latent representation and mask
|
|
tokens. Second, we find that masking a high proportion of the input image, e.g., 75%, yields a nontrivial and meaningful self-supervisory task. Coupling these two designs
|
|
enables us to train large models efficiently and effectively: we accelerate training (by 3x or more) and improve accuracy. Our scalable approach allows for learning high-capacity
|
|
models that generalize well: e.g., a vanilla ViT-Huge model achieves the best accuracy (87.8%) among methods that use only ImageNet-1K data. Transfer performance in downstream
|
|
tasks outperforms supervised pre-training and shows promising scaling behavior.*
|
|
|
|
Tips:
|
|
|
|
- MAE (masked auto encoding) is a method for self-supervised pre-training of Vision Transformers (ViTs). The pre-training objective is relatively simple:
|
|
by masking a large portion (75%) of the image patches, the model must reconstruct raw pixel values. One can use [`ViTMAEForPreTraining`] for this purpose.
|
|
- An example Python script that illustrates how to pre-train [`ViTMAEForPreTraining`] from scratch can be found [here](https://github.com/huggingface/transformers/tree/main/examples/pytorch/image-pretraining).
|
|
One can easily tweak it for their own use case.
|
|
- A notebook that illustrates how to visualize reconstructed pixel values with [`ViTMAEForPreTraining`] can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/ViTMAE/ViT_MAE_visualization_demo.ipynb).
|
|
- After pre-training, one "throws away" the decoder used to reconstruct pixels, and one uses the encoder for fine-tuning/linear probing. This means that after
|
|
fine-tuning, one can directly plug in the weights into a [`ViTForImageClassification`].
|
|
- One can use [`ViTImageProcessor`] to prepare images for the model. See the code examples for more info.
|
|
- Note that the encoder of MAE is only used to encode the visual patches. The encoded patches are then concatenated with mask tokens, which the decoder (which also
|
|
consists of Transformer blocks) takes as input. Each mask token is a shared, learned vector that indicates the presence of a missing patch to be predicted. Fixed
|
|
sin/cos position embeddings are added both to the input of the encoder and the decoder.
|
|
- For a visual understanding of how MAEs work you can check out this [post](https://keras.io/examples/vision/masked_image_modeling/).
|
|
|
|
<img src="https://user-images.githubusercontent.com/11435359/146857310-f258c86c-fde6-48e8-9cee-badd2b21bd2c.png"
|
|
alt="drawing" width="600"/>
|
|
|
|
<small> MAE architecture. Taken from the <a href="https://arxiv.org/abs/2111.06377">original paper.</a> </small>
|
|
|
|
This model was contributed by [nielsr](https://huggingface.co/nielsr). TensorFlow version of the model was contributed by [sayakpaul](https://github.com/sayakpaul) and
|
|
[ariG23498](https://github.com/ariG23498) (equal contribution). The original code can be found [here](https://github.com/facebookresearch/mae).
|
|
|
|
|
|
## ViTMAEConfig
|
|
|
|
[[autodoc]] ViTMAEConfig
|
|
|
|
|
|
## ViTMAEModel
|
|
|
|
[[autodoc]] ViTMAEModel
|
|
- forward
|
|
|
|
|
|
## ViTMAEForPreTraining
|
|
|
|
[[autodoc]] transformers.ViTMAEForPreTraining
|
|
- forward
|
|
|
|
|
|
## TFViTMAEModel
|
|
|
|
[[autodoc]] TFViTMAEModel
|
|
- call
|
|
|
|
|
|
## TFViTMAEForPreTraining
|
|
|
|
[[autodoc]] transformers.TFViTMAEForPreTraining
|
|
- call
|