mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-04 21:30:07 +06:00
17742bd9c8
* i guessreverted all CdGen classes * style * llava onevision * fix copies * fix some tests * some more tests * dump * skip these * nevermind, i am dumb * revert fix not needed * fixup * fixup * another fixup * more fixup to make ci finally happy * fixup after rebasing * fix qwen tests * add internVL + typos here and there * image token index -> id * style * fix init weights * revert blip-2 not supported * address comments * fix copies * revert blip2 test file as well * as discussed internally, revert back CdGen models * fix some tests * fix more tests for compile * CI red * fix copies * enumerate explicitly allowed models * address comments * fix tests * fixup * style again * add tests for new model class * another fixup ( x _ x ) * [fixup] unused attributes can be removed post-deprecation
117 lines
4.3 KiB
Markdown
117 lines
4.3 KiB
Markdown
<!--Copyright 2024 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
|
|
rendered properly in your Markdown viewer.
|
|
|
|
-->
|
|
|
|
# Aria
|
|
|
|
<div class="flex flex-wrap space-x-1">
|
|
<img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
|
|
<img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
|
|
<img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
|
|
</div>
|
|
|
|
## Overview
|
|
|
|
The Aria model was proposed in [Aria: An Open Multimodal Native Mixture-of-Experts Model](https://huggingface.co/papers/2410.05993) by Li et al. from the Rhymes.AI team.
|
|
|
|
Aria is an open multimodal-native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. It has a Mixture-of-Experts architecture, with respectively 3.9B and 3.5B activated parameters per visual token and text token.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.*
|
|
|
|
This model was contributed by [m-ric](https://huggingface.co/m-ric).
|
|
The original code can be found [here](https://github.com/rhymes-ai/Aria).
|
|
|
|
## Usage tips
|
|
|
|
Here's how to use the model for vision tasks:
|
|
```python
|
|
import requests
|
|
import torch
|
|
from PIL import Image
|
|
|
|
from transformers import AriaProcessor, AriaForConditionalGeneration
|
|
|
|
model_id_or_path = "rhymes-ai/Aria"
|
|
|
|
model = AriaForConditionalGeneration.from_pretrained(
|
|
model_id_or_path, device_map="auto"
|
|
)
|
|
|
|
processor = AriaProcessor.from_pretrained(model_id_or_path)
|
|
|
|
image = Image.open(requests.get("http://images.cocodataset.org/val2017/000000039769.jpg", stream=True).raw)
|
|
|
|
messages = [
|
|
{
|
|
"role": "user",
|
|
"content": [
|
|
{"type": "image"},
|
|
{"text": "what is the image?", "type": "text"},
|
|
],
|
|
}
|
|
]
|
|
|
|
text = processor.apply_chat_template(messages, add_generation_prompt=True)
|
|
inputs = processor(text=text, images=image, return_tensors="pt")
|
|
inputs.to(model.device)
|
|
|
|
output = model.generate(
|
|
**inputs,
|
|
max_new_tokens=15,
|
|
stop_strings=["<|im_end|>"],
|
|
tokenizer=processor.tokenizer,
|
|
do_sample=True,
|
|
temperature=0.9,
|
|
)
|
|
output_ids = output[0][inputs["input_ids"].shape[1]:]
|
|
response = processor.decode(output_ids, skip_special_tokens=True)
|
|
```
|
|
|
|
|
|
## AriaImageProcessor
|
|
|
|
[[autodoc]] AriaImageProcessor
|
|
|
|
## AriaProcessor
|
|
|
|
[[autodoc]] AriaProcessor
|
|
|
|
## AriaTextConfig
|
|
|
|
[[autodoc]] AriaTextConfig
|
|
|
|
## AriaConfig
|
|
|
|
[[autodoc]] AriaConfig
|
|
|
|
## AriaTextModel
|
|
|
|
[[autodoc]] AriaTextModel
|
|
|
|
## AriaModel
|
|
|
|
[[autodoc]] AriaModel
|
|
|
|
## AriaTextForCausalLM
|
|
|
|
[[autodoc]] AriaTextForCausalLM
|
|
|
|
## AriaForConditionalGeneration
|
|
|
|
[[autodoc]] AriaForConditionalGeneration
|
|
- forward
|