mirror of https://github.com/huggingface/transformers.git synced 2025-07-19 20:48:22 +06:00

* first modeling code

* make repository

* still WIP

* update model

* add tests

* add latest change

* clean docstrings and copied from

* update docstrings md and readme

* correct chroma function

* correct copied from and remove unreleated test

* add doc to toctree

* correct imports

* add convert script to notdoctested

* Add suggestion from Sanchit

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* correct get_uncoditional_inputs docstrings

* modify README according to SANCHIT feedback

* add chroma to audio utils

* clean librosa and torchaudio hard dependencies

* fix FE

* refactor audio decoder -> audio encoder for consistency with previous musicgen

* refactor conditional -> encoder

* modify sampling rate logics

* modify license at the beginning

* refactor all_self_attns->all_attentions

* remove ignore copy from causallm generate

* add copied from for from_sub_models

* fix make copies

* add warning if audio is truncated

* add copied from where relevant

* remove artefact

* fix convert script

* fix torchaudio and FE

* modify chroma method according to feedback-> better naming

* refactor input_values->input_features

* refactor input_values->input_features and fix import fe

* add input_features to docstrigs

* correct inputs_embeds logics

* remove dtype conversion

* refactor _prepare_conditional_hidden_states_kwargs_for_generation ->_prepare_encoder_hidden_states_kwargs_for_generation

* change warning for chroma length

* Update src/transformers/models/musicgen_melody/convert_musicgen_melody_transformers.py

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

* change way to save wav, using soundfile

* correct docs and change to soundfile

* fix import

* fix init proj layers

* remove line breaks from md

* fix issue with docstrings

* add FE suggestions

* improve is in logics and remove useless imports

* remove custom from_pretrained

* simplify docstring code

* add suggestions for modeling tests

* make style

* update converting script with sanity check

* remove encoder attention mask from conditional generation

* replace musicgen melody checkpoints with official orga

* rename ylacombe->facebook in checkpoints

* fix copies

* remove unecessary warning

* add shape in code docstrings

* add files to slow doc tests

* fix md bug and add md to not_tested

* make fix-copies

* fix hidden states test and batching

---------

Co-authored-by: Sanchit Gandhi <93869735+sanchit-gandhi@users.noreply.github.com>

2024-03-18 13:06:12 +00:00

39 KiB

Raw Blame History

🤗 Transformers

State-of-the-art Machine Learning for PyTorch, TensorFlow, and JAX.

🤗 Transformers provides APIs and tools to easily download and train state-of-the-art pretrained models. Using pretrained models can reduce your compute costs, carbon footprint, and save you the time and resources required to train a model from scratch. These models support common tasks in different modalities, such as:

📝 Natural Language Processing: text classification, named entity recognition, question answering, language modeling, summarization, translation, multiple choice, and text generation.
🖼️ Computer Vision: image classification, object detection, and segmentation.
🗣️ Audio: automatic speech recognition and audio classification.
🐙 Multimodal: table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.

🤗 Transformers support framework interoperability between PyTorch, TensorFlow, and JAX. This provides the flexibility to use a different framework at each stage of a model's life; train a model in three lines of code in one framework, and load it for inference in another. Models can also be exported to a format like ONNX and TorchScript for deployment in production environments.

Join the growing community on the Hub, forum, or Discord today!

If you are looking for custom support from the Hugging Face team

The documentation is organized into five sections:

GET STARTED provides a quick tour of the library and installation instructions to get up and running.
TUTORIALS are a great place to start if you're a beginner. This section will help you gain the basic skills you need to start using the library.
HOW-TO GUIDES show you how to achieve a specific goal, like finetuning a pretrained model for language modeling or how to write and share a custom model.
CONCEPTUAL GUIDES offers more discussion and explanation of the underlying concepts and ideas behind models, tasks, and the design philosophy of 🤗 Transformers.
API describes all classes and functions:
- MAIN CLASSES details the most important classes like configuration, model, tokenizer, and pipeline.
- MODELS details the classes and functions related to each model implemented in the library.
- INTERNAL HELPERS details utility classes and functions used internally.

Supported models and frameworks

The table below represents the current support in the library for each of those models, whether they have a Python tokenizer (called "slow"). A "fast" tokenizer backed by the 🤗 Tokenizers library, whether they have support in Jax (via Flax), PyTorch, and/or TensorFlow.

Model	PyTorch support	TensorFlow support	Flax Support
ALBERT	✅	✅	✅
ALIGN	✅	❌	❌
AltCLIP	✅	❌	❌
Audio Spectrogram Transformer	✅	❌	❌
Autoformer	✅	❌	❌
Bark	✅	❌	❌
BART	✅	✅	✅
BARThez	✅	✅	✅
BARTpho	✅	✅	✅
BEiT	✅	❌	✅
BERT	✅	✅	✅
Bert Generation	✅	❌	❌
BertJapanese	✅	✅	✅
BERTweet	✅	✅	✅
BigBird	✅	❌	✅
BigBird-Pegasus	✅	❌	❌
BioGpt	✅	❌	❌
BiT	✅	❌	❌
Blenderbot	✅	✅	✅
BlenderbotSmall	✅	✅	✅
BLIP	✅	✅	❌
BLIP-2	✅	❌	❌
BLOOM	✅	❌	✅
BORT	✅	✅	✅
BridgeTower	✅	❌	❌
BROS	✅	❌	❌
ByT5	✅	✅	✅
CamemBERT	✅	✅	❌
CANINE	✅	❌	❌
Chinese-CLIP	✅	❌	❌
CLAP	✅	❌	❌
CLIP	✅	✅	✅
CLIPSeg	✅	❌	❌
CLVP	✅	❌	❌
CodeGen	✅	❌	❌
CodeLlama	✅	❌	✅
Cohere	✅	❌	❌
Conditional DETR	✅	❌	❌
ConvBERT	✅	✅	❌
ConvNeXT	✅	✅	❌
ConvNeXTV2	✅	✅	❌
CPM	✅	✅	✅
CPM-Ant	✅	❌	❌
CTRL	✅	✅	❌
CvT	✅	✅	❌
Data2VecAudio	✅	❌	❌
Data2VecText	✅	❌	❌
Data2VecVision	✅	✅	❌
DeBERTa	✅	✅	❌
DeBERTa-v2	✅	✅	❌
Decision Transformer	✅	❌	❌
Deformable DETR	✅	❌	❌
DeiT	✅	✅	❌
DePlot	✅	❌	❌
Depth Anything	✅	❌	❌
DETA	✅	❌	❌
DETR	✅	❌	❌
DialoGPT	✅	✅	✅
DiNAT	✅	❌	❌
DINOv2	✅	❌	❌
DistilBERT	✅	✅	✅
DiT	✅	❌	✅
DonutSwin	✅	❌	❌
DPR	✅	✅	❌
DPT	✅	❌	❌
EfficientFormer	✅	✅	❌
EfficientNet	✅	❌	❌
ELECTRA	✅	✅	✅
EnCodec	✅	❌	❌
Encoder decoder	✅	✅	✅
ERNIE	✅	❌	❌
ErnieM	✅	❌	❌
ESM	✅	✅	❌
FairSeq Machine-Translation	✅	❌	❌
Falcon	✅	❌	❌
FastSpeech2Conformer	✅	❌	❌
FLAN-T5	✅	✅	✅
FLAN-UL2	✅	✅	✅
FlauBERT	✅	✅	❌
FLAVA	✅	❌	❌
FNet	✅	❌	❌
FocalNet	✅	❌	❌
Funnel Transformer	✅	✅	❌
Fuyu	✅	❌	❌
Gemma	✅	❌	✅
GIT	✅	❌	❌
GLPN	✅	❌	❌
GPT Neo	✅	❌	✅
GPT NeoX	✅	❌	❌
GPT NeoX Japanese	✅	❌	❌
GPT-J	✅	✅	✅
GPT-Sw3	✅	✅	✅
GPTBigCode	✅	❌	❌
GPTSAN-japanese	✅	❌	❌
Graphormer	✅	❌	❌
GroupViT	✅	✅	❌
HerBERT	✅	✅	✅
Hubert	✅	✅	❌
I-BERT	✅	❌	❌
IDEFICS	✅	❌	❌
ImageGPT	✅	❌	❌
Informer	✅	❌	❌
InstructBLIP	✅	❌	❌
Jukebox	✅	❌	❌
KOSMOS-2	✅	❌	❌
LayoutLM	✅	✅	❌
LayoutLMv2	✅	❌	❌
LayoutLMv3	✅	✅	❌
LayoutXLM	✅	❌	❌
LED	✅	✅	❌
LeViT	✅	❌	❌
LiLT	✅	❌	❌
LLaMA	✅	❌	✅
Llama2	✅	❌	✅
LLaVa	✅	❌	❌
Longformer	✅	✅	❌
LongT5	✅	❌	✅
LUKE	✅	❌	❌
LXMERT	✅	✅	❌
M-CTC-T	✅	❌	❌
M2M100	✅	❌	❌
MADLAD-400	✅	✅	✅
Mamba	✅	❌	❌
Marian	✅	✅	✅
MarkupLM	✅	❌	❌
Mask2Former	✅	❌	❌
MaskFormer	✅	❌	❌
MatCha	✅	❌	❌
mBART	✅	✅	✅
mBART-50	✅	✅	✅
MEGA	✅	❌	❌
Megatron-BERT	✅	❌	❌
Megatron-GPT2	✅	✅	✅
MGP-STR	✅	❌	❌
Mistral	✅	❌	✅
Mixtral	✅	❌	❌
mLUKE	✅	❌	❌
MMS	✅	✅	✅
MobileBERT	✅	✅	❌
MobileNetV1	✅	❌	❌
MobileNetV2	✅	❌	❌
MobileViT	✅	✅	❌
MobileViTV2	✅	❌	❌
MPNet	✅	✅	❌
MPT	✅	❌	❌
MRA	✅	❌	❌
MT5	✅	✅	✅
MusicGen	✅	❌	❌
MusicGen Melody	✅	❌	❌
MVP	✅	❌	❌
NAT	✅	❌	❌
Nezha	✅	❌	❌
NLLB	✅	❌	❌
NLLB-MOE	✅	❌	❌
Nougat	✅	✅	✅
Nyströmformer	✅	❌	❌
OneFormer	✅	❌	❌
OpenAI GPT	✅	✅	❌
OpenAI GPT-2	✅	✅	✅
OpenLlama	✅	❌	❌
OPT	✅	✅	✅
OWL-ViT	✅	❌	❌
OWLv2	✅	❌	❌
PatchTSMixer	✅	❌	❌
PatchTST	✅	❌	❌
Pegasus	✅	✅	✅
PEGASUS-X	✅	❌	❌
Perceiver	✅	❌	❌
Persimmon	✅	❌	❌
Phi	✅	❌	❌
PhoBERT	✅	✅	✅
Pix2Struct	✅	❌	❌
PLBart	✅	❌	❌
PoolFormer	✅	❌	❌
Pop2Piano	✅	❌	❌
ProphetNet	✅	❌	❌
PVT	✅	❌	❌
PVTv2	✅	❌	❌
QDQBert	✅	❌	❌
Qwen2	✅	❌	❌
RAG	✅	✅	❌
REALM	✅	❌	❌
Reformer	✅	❌	❌
RegNet	✅	✅	✅
RemBERT	✅	✅	❌
ResNet	✅	✅	✅
RetriBERT	✅	❌	❌
RoBERTa	✅	✅	✅
RoBERTa-PreLayerNorm	✅	✅	✅
RoCBert	✅	❌	❌
RoFormer	✅	✅	✅
RWKV	✅	❌	❌
SAM	✅	✅	❌
SeamlessM4T	✅	❌	❌
SeamlessM4Tv2	✅	❌	❌
SegFormer	✅	✅	❌
SegGPT	✅	❌	❌
SEW	✅	❌	❌
SEW-D	✅	❌	❌
SigLIP	✅	❌	❌
Speech Encoder decoder	✅	❌	✅
Speech2Text	✅	✅	❌
SpeechT5	✅	❌	❌
Splinter	✅	❌	❌
SqueezeBERT	✅	❌	❌
StableLm	✅	❌	❌
Starcoder2	✅	❌	❌
SwiftFormer	✅	❌	❌
Swin Transformer	✅	✅	❌
Swin Transformer V2	✅	❌	❌
Swin2SR	✅	❌	❌
SwitchTransformers	✅	❌	❌
T5	✅	✅	✅
T5v1.1	✅	✅	✅
Table Transformer	✅	❌	❌
TAPAS	✅	✅	❌
TAPEX	✅	✅	✅
Time Series Transformer	✅	❌	❌
TimeSformer	✅	❌	❌
Trajectory Transformer	✅	❌	❌
Transformer-XL	✅	✅	❌
TrOCR	✅	❌	❌
TVLT	✅	❌	❌
TVP	✅	❌	❌
UDOP	✅	❌	❌
UL2	✅	✅	✅
UMT5	✅	❌	❌
UniSpeech	✅	❌	❌
UniSpeechSat	✅	❌	❌
UnivNet	✅	❌	❌
UPerNet	✅	❌	❌
VAN	✅	❌	❌
VideoMAE	✅	❌	❌
ViLT	✅	❌	❌
VipLlava	✅	❌	❌
Vision Encoder decoder	✅	✅	✅
VisionTextDualEncoder	✅	✅	✅
VisualBERT	✅	❌	❌
ViT	✅	✅	✅
ViT Hybrid	✅	❌	❌
VitDet	✅	❌	❌
ViTMAE	✅	✅	❌
ViTMatte	✅	❌	❌
ViTMSN	✅	❌	❌
VITS	✅	❌	❌
ViViT	✅	❌	❌
Wav2Vec2	✅	✅	✅
Wav2Vec2-BERT	✅	❌	❌
Wav2Vec2-Conformer	✅	❌	❌
Wav2Vec2Phoneme	✅	✅	✅
WavLM	✅	❌	❌
Whisper	✅	✅	✅
X-CLIP	✅	❌	❌
X-MOD	✅	❌	❌
XGLM	✅	✅	✅
XLM	✅	✅	❌
XLM-ProphetNet	✅	❌	❌
XLM-RoBERTa	✅	✅	✅
XLM-RoBERTa-XL	✅	❌	❌
XLM-V	✅	✅	✅
XLNet	✅	✅	❌
XLS-R	✅	✅	✅
XLSR-Wav2Vec2	✅	✅	✅
YOLOS	✅	❌	❌
YOSO	✅	❌	❌

39 KiB Raw Blame History

🤗 Transformers

If you are looking for custom support from the Hugging Face team

Contents

Supported models and frameworks

39 KiB

Raw Blame History