# llama.cpp llama.cpp is an open-source C++ library for efficient inference of large language models, primarily on CPU hardware, co-developed with the GGML tensor library. It has become the de facto standard for local LLM inference tools. llama.cpp is an open-source software library that performs inference on various [large language models](https://www.wikiprompt.org/wiki/large-language-model) such as Meta's Llama model. It is co-developed alongside the GGML project, a general-purpose tensor library written in C. The library focuses on efficient execution on computers without dedicated GPUs, making machine-learning models accessible on standard hardware. The project includes command-line tools and a server with a simple web interface. It has become the de facto standard as the core of almost all local inference tools, including Ollama and LM Studio, enabling users to run [generative AI](https://www.wikiprompt.org/wiki/generative-ai) models on their own devices. ## Background In late September 2022, Georgian developer Georgi Gerganov began work on the GGML library, a C library implementing tensor algebra. Gerganov designed GGML with strict memory management and multi-threading as primary goals. The creation was inspired by Fabrice Bellard's work on LibNC, a C library for tensor computations. Before starting llama.cpp, Gerganov had developed a similar library called whisper.cpp, which implemented Whisper, a speech-to-text model created by [OpenAI](https://www.wikiprompt.org/wiki/openai). This earlier project established his approach to efficient model inference in low-level languages. ## Development llama.cpp began development in March 2023 as an implementation of Llama inference code in pure C/C++ with no external dependencies. This approach improved performance on computers without GPUs or other specialized hardware, which was a primary project goal. It quickly gained prominence among users lacking dedicated hardware, as it could run on a central processing unit (CPU) alone. While initially designed for CPUs, support for GPU and neural processing unit (NPU) backends was added later. As of May 2026, the project has more than 109,000 stars on GitHub, reflecting its broad adoption. Key development milestones include: FlashAttention was introduced on April 30, 2024, improving the [attention mechanism](https://www.wikiprompt.org/wiki/multi-head-attention) efficiency. On April 10, 2025, libmtmd was introduced to reinvigorate support for multimodal models that had previously been stagnant. On December 17, 2025, full acceleration on Android and ChromeOS devices was introduced via a new GUI binding, allowing native app development beyond previous cross-compiling and running a command-line interface in an adb shell. ## Architecture llama.cpp supports multiple hardware targets, including x86, ARM, Metal, BLAS, BLIS, zDNN, ZenDNN, SYCL, MUSA, CUDA, HIP, CANN, OpenCL, RPC, and Vulkan version 1.2 or greater. These backends are part of the GGML tensor library, which is used by the front-end model-specific llama.cpp code. The library makes use of several CPU extensions for optimization. For x86-64, it uses AVX, AVX2, AVX-512, AVX-VNNI, and AMX instruction sets. For AArch64 (ARM64), it uses NEON, i8MM, SVE, SVE2, SME, and SME2. For s390x, it uses the Vector Enhancement Facility 2 (VXE2). [Apple silicon](https://www.wikiprompt.org/wiki/arm-holdings) is an important target for the project. Several features target inference on edge devices. These include ahead-of-time model quantization, on-the-fly key-value (kv) cache quantization, [speculative decoding](https://www.wikiprompt.org/wiki/speculative-decoding), and partial offloading of model layers to system RAM. The latter allows devices to load models that would otherwise be too large to fit entirely in GPU VRAM. For frontend communication, llama.cpp provides OpenAI-compatible endpoints such as v1/chat/completions and supports grammar-based output formatting as JSON. This enables [cloud services](https://www.wikiprompt.org/wiki/amazon-web-services) to integrate with the library. ## GGUF File Format The GGUF (GGML Universal File) format is a binary format that stores both tensors and metadata in a single file, designed for fast saving and loading of model data. It was introduced in August 2023 by the project team to maintain backwards compatibility as support for other model architectures was added. GGUF superseded the previous GGML format and is typically produced by converting models developed with other machine learning libraries such as PyTorch. The format focuses on quantization, which is the act of reducing precision in model weights. Quantization reduces memory usage and increases speed, albeit at the cost of reduced model accuracy. GGUF supports 2-bit to 8-bit quantized integer types, common floating-point data formats such as float32, float16, and bfloat16, and 1.58-bit quantization. It contains the tokenizer vocabulary, context length, tensor info, and other attributes needed for a GPT-like language model. The byte-level structure uses little-endian ordering, divided into a metadata block and a tensors information block. This structure enables efficient serialization to and from storage for dynamic.
## Models llama.cpp supports many large language models, including Llama, Mistral, Gemma, DeepSeek, gpt-oss, Phi, and Qwen. This variety makes it a flexible solution for diverse [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) use cases, from chat applications to more specialized tasks. Community tools such as Ollama and LM Studio are built on the project, and it often uses proprietary ecosystems like [anthropic](https://www.wikiprompt.org/wiki/anthropic) on local infrastructure. --- Source: https://www.wikiprompt.org/wiki/llama-cpp License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-09T01:53:53.746628+00:00