# GGML

GGML is an open-source tensor library for machine learning, designed for efficient inference on consumer hardware. It is co-developed with llama.cpp, a widely used tool for running large language models locally.

GGML is a general-purpose tensor library written in C, designed for machine learning applications with a focus on efficient execution on consumer hardware. It was created by Georgi Gerganov and is co-developed with llama.cpp, an open-source software library that performs inference on various large language models such as Meta's Llama. GGML provides the underlying tensor operations and hardware backends that enable llama.cpp to run models on CPUs, GPUs, and other accelerators, making local AI inference accessible to a broad audience.

The library emphasizes strict memory management and multi-threading, which allows it to achieve high performance even on systems without dedicated graphics cards. GGML has become a foundational component in the local AI ecosystem, with llama.cpp considered the de facto standard for local inference tools, including popular applications like Ollama and LM Studio.

## Background

Georgi Gerganov began work on the GGML library in late September 2022, drawing inspiration from Fabrice Bellard's work on LibNC. Before GGML, Gerganov developed whisper.cpp, a similar library that implemented OpenAI's Whisper speech-to-text model. This earlier project laid the groundwork for the design principles and technical approach later applied to GGML and llama.cpp.

## Development

llama.cpp started development in March 2023 as a pure C/C++ implementation of Llama inference code with no external dependencies. The project aimed to improve performance on computers without GPUs or other dedicated hardware, and it quickly gained traction among users with limited hardware capabilities. Initially designed for CPUs, llama.cpp later added support for GPUs and NPUs. As of May 2026, the project has over 109,000 stars on GitHub.

Key milestones include the introduction of FlashAttention on April 30, 2024, which significantly improved attention computation efficiency. On April 10, 2025, libmtmd was introduced, reinvigorating support for multimodal models. On December 17, 2025, full acceleration on Android and ChromeOS devices was enabled via a new GUI binding, allowing native app development beyond the previous cross-compiling and CLI approach.

## Architecture

llama.cpp supports multiple hardware targets, including x86, ARM, Metal, BLAS, BLIS, zDNN, ZenDNN, SYCL, MUSA, CUDA, HIP, CANN, OpenCL, RPC, and Vulkan (version 1.2 or greater). These backends form the GGML tensor library, which is used by the model-specific llama.cpp frontend. The library leverages CPU extensions for optimization: AVX, AVX2, AVX-512, AVX-VNNI, and AMX for x86-64; Neon, i8MM, SVE, SVE2, SME, and SME2 for AArch64; and VXE2 for S390x. Apple silicon is an important target for the project.

GGML supports features aimed at edge inference, including ahead-of-time model quantization, on-the-fly KV-cache quantization, speculative decoding, and partial offloading of model layers to system RAM, enabling devices to load models that exceed GPU VRAM capacity. For frontend communication, llama.cpp offers OpenAI-compatible endpoints like v1/chat/completions and grammar-based output formatting as JSON.

## GGUF File Format

The GGUF (GGML Universal File) format is a binary format that stores both tensors and metadata in a single file, designed for fast saving and loading of model data. Introduced in August 2023 by the llama.cpp project, GGUF was created to maintain backward compatibility as support for additional model architectures was added. It superseded earlier formats such as GGML and is typically produced by converting models from other machine learning libraries like PyTorch.

### Design

GGUF focuses on quantization, reducing the precision of model weights to lower memory usage and increase speed, at the cost of some accuracy. It supports 2-bit to 8-bit quantized integer types, common floating-point formats such as float32, float16, and bfloat16, and 1.58-bit quantization. The format includes essential information for running GPT-like language models, such as tokenizer vocabulary, context length, tensor info, and other attributes.

## Models

llama.cpp supports a wide range of large language models, including Llama, Mistral, Gemma, DeepSeek, gpt-oss, Phi, and Qwen. This broad compatibility has made GGML and llama.cpp essential tools for developers and researchers working with [large language models](https://www.wikiprompt.org/wiki/large-language-model) on local hardware.

## See Also

- [Ollama](https://www.wikiprompt.org/wiki/ollama) - A tool for running LLMs locally
- [LM Studio](https://www.wikiprompt.org/wiki/lm-studio) - Desktop application for local LLM interaction
- [vLLM](https://www.wikiprompt.org/wiki/vllm) - High-performance inference engine

---
Source: https://www.wikiprompt.org/wiki/ggml
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-09T01:53:32.656979+00:00