riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-04 05:10:06 +06:00
Code Issues Actions 26 Packages Projects Releases Wiki Activity
a396f4324b
transformers/docs/source/en/model_doc/granite_speech.md
Quentin Gallouédec de24fb63ed
Use HF papers (#38184) 
* Use hf papers

* Hugging Face papers

* doi to hf papers

* style
2025-06-13 11:07:09 +00:00

3.1 KiB
Raw Blame History

Granite Speech

PyTorch

Overview

The Granite Speech model is a multimodal language model, consisting of a speech encoder, speech projector, large language model, and LoRA adapter(s). More details regarding each component for the current (Granite 3.2 Speech) model architecture may be found below.

  1. Speech Encoder: A Conformer encoder trained with Connectionist Temporal Classification (CTC) on character-level targets on ASR corpora. The encoder uses block-attention and self-conditioned CTC from the middle layer.

  2. Speech Projector: A query transformer (q-former) operating on the outputs of the last encoder block. The encoder and projector temporally downsample the audio features to be merged into the multimodal embeddings to be processed by the llm.

  3. Large Language Model: The Granite Speech model leverages Granite LLMs, which were originally proposed in this paper.

  4. LoRA adapter(s): The Granite Speech model contains a modality specific LoRA, which will be enabled when audio features are provided, and disabled otherwise.

Note that most of the aforementioned components are implemented generically to enable compatability and potential integration with other model architectures in transformers.

This model was contributed by Alexander Brooks, Avihu Dekel, and George Saon.

Usage tips

  • This model bundles its own LoRA adapter, which will be automatically loaded and enabled/disabled as needed during inference calls. Be sure to install PEFT to ensure the LoRA is correctly applied!

GraniteSpeechConfig

autodoc GraniteSpeechConfig

GraniteSpeechEncoderConfig

autodoc GraniteSpeechEncoderConfig

GraniteSpeechProcessor

autodoc GraniteSpeechProcessor

GraniteSpeechFeatureExtractor

autodoc GraniteSpeechFeatureExtractor

GraniteSpeechForConditionalGeneration

autodoc GraniteSpeechForConditionalGeneration - forward