mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-04 21:30:07 +06:00
342961f669
* init vilt image processor fast * Refactor image processor tests to use loop for all processors * Add ViltImageProcessorFast with PyTorch-based optimized image processing * Change made automatically by make fixup command * Change made automatically by make fix-copies command * Fix type hints in ViltImageProcessorFast for Python compatibility * Define constants for image resizing based on COCO dataset aspect ratio * Add missing property initializations to ViltImageProcessorFast * Extract resize logic into dedicated method in ViltImageProcessorFast * Extract padding logic into dedicated method * Implement shape-based image grouping for optimized processing in Vilt * Update test suite to verify ViltImageProcessorFast attributes * Move variable declarations to _preprocess method parameters * Remove unused parameters * Rename _resize method to resize to override existing function * Remove whitespace * Remove unnecessary type check and conversion for stacked_images * Remove redundant loop and apply padding directly to stacked images * Refactor pad function to return images and mask as tuple instead of dict * Add tests comparing padding masks in slow and fast implementations * Update ViltImageProcessor tests to ensure compatibility between slow and fast implementations * Replace add_start_docstrings with auto_docstring in ViltImageProcessorFast * Move docstrings of custom args to ViltFastImageProcessorKwargs * Use reorder_images function for both masks and images --------- Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com>
114 lines
4.6 KiB
Markdown
114 lines
4.6 KiB
Markdown
<!--Copyright 2021 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|
rendered properly in your Markdown viewer.
|
|
|
|
-->
|
|
|
|
# ViLT
|
|
|
|
<div class="flex flex-wrap space-x-1">
|
|
<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
|
|
</div>
|
|
|
|
## Overview
|
|
|
|
The ViLT model was proposed in [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://arxiv.org/abs/2102.03334)
|
|
by Wonjae Kim, Bokyung Son, Ildoo Kim. ViLT incorporates text embeddings into a Vision Transformer (ViT), allowing it to have a minimal design
|
|
for Vision-and-Language Pre-training (VLP).
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*Vision-and-Language Pre-training (VLP) has improved performance on various joint vision-and-language downstream tasks.
|
|
Current approaches to VLP heavily rely on image feature extraction processes, most of which involve region supervision
|
|
(e.g., object detection) and the convolutional architecture (e.g., ResNet). Although disregarded in the literature, we
|
|
find it problematic in terms of both (1) efficiency/speed, that simply extracting input features requires much more
|
|
computation than the multimodal interaction steps; and (2) expressive power, as it is upper bounded to the expressive
|
|
power of the visual embedder and its predefined visual vocabulary. In this paper, we present a minimal VLP model,
|
|
Vision-and-Language Transformer (ViLT), monolithic in the sense that the processing of visual inputs is drastically
|
|
simplified to just the same convolution-free manner that we process textual inputs. We show that ViLT is up to tens of
|
|
times faster than previous VLP models, yet with competitive or better downstream task performance.*
|
|
|
|
<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/vilt_architecture.jpg"
|
|
alt="drawing" width="600"/>
|
|
|
|
<small> ViLT architecture. Taken from the <a href="https://arxiv.org/abs/2102.03334">original paper</a>. </small>
|
|
|
|
This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/dandelin/ViLT).
|
|
|
|
## Usage tips
|
|
|
|
- The quickest way to get started with ViLT is by checking the [example notebooks](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/ViLT)
|
|
(which showcase both inference and fine-tuning on custom data).
|
|
- ViLT is a model that takes both `pixel_values` and `input_ids` as input. One can use [`ViltProcessor`] to prepare data for the model.
|
|
This processor wraps a image processor (for the image modality) and a tokenizer (for the language modality) into one.
|
|
- ViLT is trained with images of various sizes: the authors resize the shorter edge of input images to 384 and limit the longer edge to
|
|
under 640 while preserving the aspect ratio. To make batching of images possible, the authors use a `pixel_mask` that indicates
|
|
which pixel values are real and which are padding. [`ViltProcessor`] automatically creates this for you.
|
|
- The design of ViLT is very similar to that of a standard Vision Transformer (ViT). The only difference is that the model includes
|
|
additional embedding layers for the language modality.
|
|
- The PyTorch version of this model is only available in torch 1.10 and higher.
|
|
|
|
## ViltConfig
|
|
|
|
[[autodoc]] ViltConfig
|
|
|
|
## ViltFeatureExtractor
|
|
|
|
[[autodoc]] ViltFeatureExtractor
|
|
- __call__
|
|
|
|
## ViltImageProcessor
|
|
|
|
[[autodoc]] ViltImageProcessor
|
|
- preprocess
|
|
|
|
## ViltImageProcessorFast
|
|
|
|
[[autodoc]] ViltImageProcessorFast
|
|
- preprocess
|
|
|
|
## ViltProcessor
|
|
|
|
[[autodoc]] ViltProcessor
|
|
- __call__
|
|
|
|
## ViltModel
|
|
|
|
[[autodoc]] ViltModel
|
|
- forward
|
|
|
|
## ViltForMaskedLM
|
|
|
|
[[autodoc]] ViltForMaskedLM
|
|
- forward
|
|
|
|
## ViltForQuestionAnswering
|
|
|
|
[[autodoc]] ViltForQuestionAnswering
|
|
- forward
|
|
|
|
## ViltForImagesAndTextClassification
|
|
|
|
[[autodoc]] ViltForImagesAndTextClassification
|
|
- forward
|
|
|
|
## ViltForImageAndTextRetrieval
|
|
|
|
[[autodoc]] ViltForImageAndTextRetrieval
|
|
- forward
|
|
|
|
## ViltForTokenClassification
|
|
|
|
[[autodoc]] ViltForTokenClassification
|
|
- forward
|