transformers/docs/source/en/gguf.md

GGUF

GGUF is a file format used to store models for inference with GGML, a fast and lightweight inference framework written in C and C++. GGUF is a single-file format containing the model metadata and tensors.

The GGUF format also supports many quantized data types (refer to quantization type table for a complete list of supported quantization types) which saves a significant amount of memory, making inference with large models like Whisper and Llama feasible on local and edge devices.

Transformers supports loading models stored in the GGUF format for further training or finetuning. The GGUF format is dequantized to fp32 where the full model weights are available and compatible with PyTorch.

Tip

Models that support GGUF include Llama, Mistral, Qwen2, Qwen2Moe, Phi3, Bloom, Falcon, StableLM, GPT2, and Starcoder2.

Add the gguf_file parameter to [~PreTrainedModel.from_pretrained] to specify the GGUF file to load.

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF"
filename = "tinyllama-1.1b-chat-v1.0.Q6_K.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

Once you're done tinkering with the model, save and convert it back to the GGUF format with the convert-hf-to-gguf.py script.

tokenizer.save_pretrained("directory")
model.save_pretrained("directory")

!python ${path_to_llama_cpp}/convert-hf-to-gguf.py ${directory}