54a123f068
* Reverse dependency map shouldn't be created when test_all is set * [test_all] Remove dummies * Modular fixes * Update utils/check_repo.py Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> * [test_all] Better docs * [test_all] Update src/transformers/commands/chat.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * [test_all] Remove deprecated AdaptiveEmbeddings from the tests * [test_all] Doc builder * [test_all] is_dummy * [test_all] Import utils * [test_all] Doc building should not require all deps --------- Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com>
4.6 KiB
BLIP
Overview
The BLIP model was proposed in BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation by Junnan Li, Dongxu Li, Caiming Xiong, Steven Hoi.
BLIP is a model that is able to perform various multi-modal tasks including:
- Visual Question Answering
- Image-Text retrieval (Image-text matching)
- Image Captioning
The abstract from the paper is the following:
Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.
This model was contributed by ybelkada. The original code can be found here.
Resources
- Jupyter notebook on how to fine-tune BLIP for image captioning on a custom dataset
BlipConfig
autodoc BlipConfig - from_text_vision_configs
BlipTextConfig
autodoc BlipTextConfig
BlipVisionConfig
autodoc BlipVisionConfig
BlipProcessor
autodoc BlipProcessor
BlipImageProcessor
autodoc BlipImageProcessor - preprocess
BlipImageProcessorFast
autodoc BlipImageProcessorFast - preprocess
BlipModel
BlipModel is going to be deprecated in future versions, please use BlipForConditionalGeneration, BlipForImageTextRetrieval or BlipForQuestionAnswering depending on your usecase.
autodoc BlipModel - forward - get_text_features - get_image_features
BlipTextModel
autodoc BlipTextModel - forward
BlipTextLMHeadModel
autodoc BlipTextLMHeadModel
- forward
BlipVisionModel
autodoc BlipVisionModel - forward
BlipForConditionalGeneration
autodoc BlipForConditionalGeneration - forward
BlipForImageTextRetrieval
autodoc BlipForImageTextRetrieval - forward
BlipForQuestionAnswering
autodoc BlipForQuestionAnswering - forward
TFBlipModel
autodoc TFBlipModel - call - get_text_features - get_image_features
TFBlipTextModel
autodoc TFBlipTextModel - call
TFBlipTextLMHeadModel
autodoc TFBlipTextLMHeadModel
- forward
TFBlipVisionModel
autodoc TFBlipVisionModel - call
TFBlipForConditionalGeneration
autodoc TFBlipForConditionalGeneration - call
TFBlipForImageTextRetrieval
autodoc TFBlipForImageTextRetrieval - call
TFBlipForQuestionAnswering
autodoc TFBlipForQuestionAnswering - call