# Granite Speech

## Overview The Granite Speech model is a multimodal language model, consisting of a speech encoder, speech projector, large language model, and LoRA adapter(s). More details regarding each component for the current (Granite 3.2 Speech) model architecture may be found below. 1. Speech Encoder: A [Conformer](https://huggingface.co/papers/2005.08100) encoder trained with Connectionist Temporal Classification (CTC) on character-level targets on ASR corpora. The encoder uses block-attention and self-conditioned CTC from the middle layer. 2. Speech Projector: A query transformer (q-former) operating on the outputs of the last encoder block. The encoder and projector temporally downsample the audio features to be merged into the multimodal embeddings to be processed by the llm. 3. Large Language Model: The Granite Speech model leverages Granite LLMs, which were originally proposed in [this paper](https://huggingface.co/papers/2408.13359). 4. LoRA adapter(s): The Granite Speech model contains a modality specific LoRA, which will be enabled when audio features are provided, and disabled otherwise. Note that most of the aforementioned components are implemented generically to enable compatability and potential integration with other model architectures in transformers. This model was contributed by [Alexander Brooks](https://huggingface.co/abrooks9944), [Avihu Dekel](https://huggingface.co/Avihu), and [George Saon](https://huggingface.co/gsaon). ## Usage tips - This model bundles its own LoRA adapter, which will be automatically loaded and enabled/disabled as needed during inference calls. Be sure to install [PEFT](https://github.com/huggingface/peft) to ensure the LoRA is correctly applied! ## GraniteSpeechConfig [[autodoc]] GraniteSpeechConfig ## GraniteSpeechEncoderConfig [[autodoc]] GraniteSpeechEncoderConfig ## GraniteSpeechProcessor [[autodoc]] GraniteSpeechProcessor ## GraniteSpeechFeatureExtractor [[autodoc]] GraniteSpeechFeatureExtractor ## GraniteSpeechForConditionalGeneration [[autodoc]] GraniteSpeechForConditionalGeneration - forward