transformers/docs/source/en/index.md
Arthur 19d58d31f1
Add MLLama (#33703)
* current changes

* nit

* Add cross_attenttion_mask to processor

* multi-image fixed

* Add cross_attenttion_mask to processor

* cross attn works in all cases

* WIP refactoring function for image processor

* WIP refactoring image processor functions

* Refactor preprocess to use global loops instead of list nested list comps

* Docstrings

* Add channels unification

* fix dtype issues

* Update docsrings and format

* Consistent max_image_tiles

* current script

* updates

* Add convert to rgb

* Add image processor tests

* updates!

* update

* god damn it I am dumb sometimes

* Precompute aspect ratios

* now this works, full match

* fix 😉

* nits

* style

* fix model and conversion

* nit

* nit

* kinda works

* hack for sdpa non-contiguous bias

* nits here and there

* latest c hanges

* merge?

* run forward

* Add aspect_ratio_mask

* vision attention mask

* update script and config variable names

* nit

* nits

* be able to load

* style

* nits

* there

* nits

* make forward run

* small update

* enable generation multi-turn

* nit

* nit

* Clean up a bit for errors and typos

* A bit more constant fixes

* 90B keys and shapes match

* Fix for 11B model

* Fixup, remove debug part

* Docs

* Make max_aspect_ratio_id to be minimal

* Update image processing code to match new implementation

* Adjust conversion for final checkpoint state

* Change dim in repeat_interleave (accordig to meta code)

* tmp fix for num_tiles

* Fix for conversion (gate<->up, q/k_proj rope permute)

* nits

* codestyle

* Vision encoder fixes

* pass cross attn mask further

* Refactor aspect ratio mask

* Disable text-only generation

* Fix cross attention layers order, remove q/k norm rotation for cross atention layers

* Refactor gated position embeddings

* fix bugs but needs test with new weights

* rope scaling should be llama3

* Fix rope scaling name

* Remove debug for linear layer

* fix copies

* Make mask prepare private func

* Remove linear patch embed

* Make precomputed embeddings as nn.Embedding module

* MllamaPrecomputedAspectRatioEmbedding with config init

* Remove unused self.output_dim

* nit, intermediate layers

* Rename ln and pos_embed

* vision_chunk_size -> image_size

* return_intermediate -> intermediate_layers_indices

* vision_input_dim -> hidden_size

* Fix copied from statements

* fix most tests

* Fix more copied from

* layer_id->layer_idx

* Comment

* Fix tests for processor

* Copied from for _prepare_4d_causal_attention_mask_with_cache_position

* Style fix

* Add MllamaForCausalLM

* WIP fixing tests

* Remove duplicated layers

* Remove dummy file

* Fix style

* Fix consistency

* Fix some TODOs

* fix language_model instantiation, add docstring

* Move docstring, remove todos for precomputed embeds (we cannot init them properly)

* Add initial docstrings

* Fix

* fix some tests

* lets skip these

* nits, remove print, style

* Add one more copied from

* Improve test message

* Make validate func private

* Fix dummy objects

* Refactor `data_format` a bit + add comment

* typos/nits

Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com>

* fix dummy objects and imports

* Add chat template config json

* remove num_kv_heads from vision attention

* fix

* move some commits and add more tests

* fix test

* Remove `update_key_name` from modeling utils

* remove num-kv-heads again

* some prelimiary docs

* Update chat template + tests

* nit, conversion script max_num_tiles from params

* Fix warning for text-only generation

* Update conversion script for instruct models

* Update chat template in converstion + test

* add tests for CausalLM model

* model_max_length, avoid null chat_template

* Refactor conversion script

* Fix forward

* Fix integration tests

* Refactor vision config + docs

* Fix default

* Refactor text config

* Doc fixes

* Remove unused args, fix docs example

* Squashed commit of the following:

commit b51ce5a2efffbecdefbf6fc92ee87372ec9d8830
Author: qubvel <qubvel@gmail.com>
Date:   Wed Sep 18 13:39:15 2024 +0000

    Move model + add output hidden states and output attentions

* Fix num_channels

* Add mllama text and mllama vision models

* Fixing repo consistency

* Style fix

* Fixing repo consistency

* Fixing unused config params

* Fix failed tests after refactoring

* hidden_activation -> hidden_act  for text mlp

* Remove from_pretrained from sub-configs

* Apply suggestions from code review

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Update src/transformers/models/mllama/convert_mllama_weights_to_hf.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Reuse lambda in conversion script

* Remove run.py

* Update docs/source/en/model_doc/mllama.md

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Update src/transformers/models/mllama/processing_mllama.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Remove unused LlamaTokenizerFast

* Fix logging

* Refactor gating

* Remove cycle for collecting intermediate states

* Refactor text-only check, add integration test for text-only

* Revert from pretrained to configs

* Fix example

* Add auto `bos_token` adding in processor

* Fix tips

* Update src/transformers/models/auto/tokenization_auto.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Enable supports_gradient_checkpointing model flag

* add eager/sdpa options

* don't skip attn tests and bring back GC skips (did i really remove those?)

* Fix signature, but get error with None gradient

* Fix output attention tests

* Disable GC back

* Change no split modules

* Fix dropout

* Style

* Add Mllama to sdpa list

* Add post init for vision model

* Refine config for MllamaForCausalLMModelTest and skipped tests for CausalLM model

* if skipped, say it, don't pass

* Clean vision tester config

* Doc for args

* Update tests/models/mllama/test_modeling_mllama.py

Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>

* Add cross_attention_mask to test

* typehint

* Remove todo

* Enable gradient checkpointing

* Docstring

* Style

* Fixing and skipping some tests for new cache

* Mark flaky test

* Skip `test_sdpa_can_compile_dynamic` test

* Fixing some offload tests

* Add direct GenerationMixin inheritance

* Remove unused code

* Add initializer_range to vision config

* update the test to make sure we show if split

* fix gc?

* Fix repo consistency

* Undo modeling utils debug changes

* Fix link

* mllama -> Mllama

* [mllama] -> [Mllama]

* Enable compile test for CausalLM model (text-only)

* Fix TextModel prefix

* Update doc

* Docs for forward, type hints, and vision model prefix

* make sure to reset

* fix init

* small script refactor and styling

* nit

* updates!

* some nits

* Interpolate embeddings for 560 size and update integration tests

* nit

* does not suppor static cache!

* update

* fix

* nit2

* this?

* Fix conversion

* Style

* 4x memory improvement with image cache AFAIK

* Token decorator for tests

* Skip failing tests

* update processor errors

* fix split issues

* style

* weird

* style

* fix failing tests

* update

* nit fixing the whisper tests

* fix path

* update

---------

Co-authored-by: raushan <raushan@huggingface.co>
Co-authored-by: pavel <ubuntu@ip-10-90-0-11.ec2.internal>
Co-authored-by: qubvel <qubvel@gmail.com>
Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com>
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
2024-09-25 19:56:25 +02:00

44 KiB

🤗 Transformers

State-of-the-art Machine Learning for PyTorch, TensorFlow, and JAX.

🤗 Transformers provides APIs and tools to easily download and train state-of-the-art pretrained models. Using pretrained models can reduce your compute costs, carbon footprint, and save you the time and resources required to train a model from scratch. These models support common tasks in different modalities, such as:

📝 Natural Language Processing: text classification, named entity recognition, question answering, language modeling, summarization, translation, multiple choice, and text generation.
🖼️ Computer Vision: image classification, object detection, and segmentation.
🗣️ Audio: automatic speech recognition and audio classification.
🐙 Multimodal: table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.

🤗 Transformers support framework interoperability between PyTorch, TensorFlow, and JAX. This provides the flexibility to use a different framework at each stage of a model's life; train a model in three lines of code in one framework, and load it for inference in another. Models can also be exported to a format like ONNX and TorchScript for deployment in production environments.

Join the growing community on the Hub, forum, or Discord today!

If you are looking for custom support from the Hugging Face team

HuggingFace Expert Acceleration Program

Contents

The documentation is organized into five sections:

  • GET STARTED provides a quick tour of the library and installation instructions to get up and running.

  • TUTORIALS are a great place to start if you're a beginner. This section will help you gain the basic skills you need to start using the library.

  • HOW-TO GUIDES show you how to achieve a specific goal, like finetuning a pretrained model for language modeling or how to write and share a custom model.

  • CONCEPTUAL GUIDES offers more discussion and explanation of the underlying concepts and ideas behind models, tasks, and the design philosophy of 🤗 Transformers.

  • API describes all classes and functions:

    • MAIN CLASSES details the most important classes like configuration, model, tokenizer, and pipeline.
    • MODELS details the classes and functions related to each model implemented in the library.
    • INTERNAL HELPERS details utility classes and functions used internally.

Supported models and frameworks

The table below represents the current support in the library for each of those models, whether they have a Python tokenizer (called "slow"). A "fast" tokenizer backed by the 🤗 Tokenizers library, whether they have support in Jax (via Flax), PyTorch, and/or TensorFlow.

Model PyTorch support TensorFlow support Flax Support
ALBERT
ALIGN
AltCLIP
Audio Spectrogram Transformer
Autoformer
Bark
BART
BARThez
BARTpho
BEiT
BERT
Bert Generation
BertJapanese
BERTweet
BigBird
BigBird-Pegasus
BioGpt
BiT
Blenderbot
BlenderbotSmall
BLIP
BLIP-2
BLOOM
BORT
BridgeTower
BROS
ByT5
CamemBERT
CANINE
Chameleon
Chinese-CLIP
CLAP
CLIP
CLIPSeg
CLVP
CodeGen
CodeLlama
Cohere
Conditional DETR
ConvBERT
ConvNeXT
ConvNeXTV2
CPM
CPM-Ant
CTRL
CvT
DAC
Data2VecAudio
Data2VecText
Data2VecVision
DBRX
DeBERTa
DeBERTa-v2
Decision Transformer
Deformable DETR
DeiT
DePlot
Depth Anything
DETA
DETR
DialoGPT
DiNAT
DINOv2
DistilBERT
DiT
DonutSwin
DPR
DPT
EfficientFormer
EfficientNet
ELECTRA
EnCodec
Encoder decoder
ERNIE
ErnieM
ESM
FairSeq Machine-Translation
Falcon
FalconMamba
FastSpeech2Conformer
FLAN-T5
FLAN-UL2
FlauBERT
FLAVA
FNet
FocalNet
Funnel Transformer
Fuyu
Gemma
Gemma2
GIT
GLPN
GPT Neo
GPT NeoX
GPT NeoX Japanese
GPT-J
GPT-Sw3
GPTBigCode
GPTSAN-japanese
Granite
GraniteMoeMoe
Graphormer
Grounding DINO
GroupViT
HerBERT
Hiera
Hubert
I-BERT
IDEFICS
Idefics2
ImageGPT
Informer
InstructBLIP
InstructBlipVideo
Jamba
JetMoe
Jukebox
KOSMOS-2
LayoutLM
LayoutLMv2
LayoutLMv3
LayoutXLM
LED
LeViT
LiLT
LLaMA
Llama2
Llama3
LLaVa
LLaVA-NeXT
LLaVa-NeXT-Video
LLaVA-Onevision
Longformer
LongT5
LUKE
LXMERT
M-CTC-T
M2M100
MADLAD-400
Mamba
mamba2
Marian
MarkupLM
Mask2Former
MaskFormer
MatCha
mBART
mBART-50
MEGA
Megatron-BERT
Megatron-GPT2
MGP-STR
Mimi
Mistral
Mixtral
Mllama
mLUKE
MMS
MobileBERT
MobileNetV1
MobileNetV2
MobileViT
MobileViTV2
MPNet
MPT
MRA
MT5
MusicGen
MusicGen Melody
MVP
NAT
Nemotron
Nezha
NLLB
NLLB-MOE
Nougat
Nyströmformer
OLMo
OLMoE
OmDet-Turbo
OneFormer
OpenAI GPT
OpenAI GPT-2
OpenLlama
OPT
OWL-ViT
OWLv2
PaliGemma
PatchTSMixer
PatchTST
Pegasus
PEGASUS-X
Perceiver
Persimmon
Phi
Phi3
PhoBERT
Pix2Struct
Pixtral
PLBart
PoolFormer
Pop2Piano
ProphetNet
PVT
PVTv2
QDQBert
Qwen2
Qwen2Audio
Qwen2MoE
Qwen2VL
RAG
REALM
RecurrentGemma
Reformer
RegNet
RemBERT
ResNet
RetriBERT
RoBERTa
RoBERTa-PreLayerNorm
RoCBert
RoFormer
RT-DETR
RT-DETR-ResNet
RWKV
SAM
SeamlessM4T
SeamlessM4Tv2
SegFormer
SegGPT
SEW
SEW-D
SigLIP
Speech Encoder decoder
Speech2Text
SpeechT5
Splinter
SqueezeBERT
StableLm
Starcoder2
SuperPoint
SwiftFormer
Swin Transformer
Swin Transformer V2
Swin2SR
SwitchTransformers
T5
T5v1.1
Table Transformer
TAPAS
TAPEX
Time Series Transformer
TimeSformer
Trajectory Transformer
Transformer-XL
TrOCR
TVLT
TVP
UDOP
UL2
UMT5
UniSpeech
UniSpeechSat
UnivNet
UPerNet
VAN
VideoLlava
VideoMAE
ViLT
VipLlava
Vision Encoder decoder
VisionTextDualEncoder
VisualBERT
ViT
ViT Hybrid
VitDet
ViTMAE
ViTMatte
ViTMSN
VITS
ViViT
Wav2Vec2
Wav2Vec2-BERT
Wav2Vec2-Conformer
Wav2Vec2Phoneme
WavLM
Whisper
X-CLIP
X-MOD
XGLM
XLM
XLM-ProphetNet
XLM-RoBERTa
XLM-RoBERTa-XL
XLM-V
XLNet
XLS-R
XLSR-Wav2Vec2
YOLOS
YOSO
ZoeDepth