# vLLM Project

vLLM is an open-source framework for high-throughput, memory-efficient inference and serving of large language models, developed at UC Berkeley and centered on the PagedAttention algorithm. It supports features like continuous batching, quantization, and OpenAI-compatible APIs, and became a Linux Foundation project in 2024.

vLLM is an open-source software framework for inference and serving of large language models and related multimodal models. Originally developed at the University of California, Berkeley's Sky Computing Lab, the project is centered on PagedAttention, a memory-management method for transformer key–value caches, and supports features such as continuous batching, distributed inference, quantization, and OpenAI-compatible APIs. It is designed to provide high throughput and memory efficiency for deploying large language models in production environments.

The project has gained significant traction in the artificial intelligence community as a standard tool for serving models, with adoption across cloud providers and hardware vendors. Its architecture and design have influenced subsequent inference frameworks, and its transition to a Linux Foundation project in 2024 marked a milestone in open-source AI infrastructure.

## History

vLLM was introduced in 2023 by researchers affiliated with the Sky Computing Lab at UC Berkeley. Its core ideas were described in the 2023 paper "Efficient Memory Management for Large Language Model Serving with PagedAttention," which presented the system as a high-throughput and memory-efficient serving engine for large language models. The paper detailed how PagedAttention could reduce memory waste in the key–value cache, a critical bottleneck in transformer inference.

According to a project maintainer, the "v" in vLLM originally referred to "virtual," inspired by virtual memory. This naming reflects the algorithm's conceptual debt to operating system paging techniques.

PyTorch's project page states that the University of California, Berkeley contributed vLLM to the Linux Foundation in July 2024. In 2025, the PyTorch Foundation announced that vLLM had become a Foundation-hosted project, formalizing its governance under the foundation's umbrella. In January 2026, TechCrunch reported that the creators of vLLM had launched the startup Inferact to commercialize the project, raising $150 million in seed funding.

## Architecture

According to its 2023 paper, vLLM was designed to improve the efficiency of large language model serving by reducing memory waste in the key–value cache used during transformer inference. The paper introduced PagedAttention, an algorithm inspired by virtual memory and paging techniques in operating systems, and described vLLM as using block-level memory management and request scheduling to increase throughput while maintaining similar latency.

PagedAttention works by dividing the key–value cache into fixed-size blocks, which can be allocated and deallocated dynamically, similar to how operating systems manage physical memory pages. This approach minimizes fragmentation and allows for more efficient use of GPU memory, enabling higher batch sizes and better utilization of hardware resources.

The project documentation and repository describe support for continuous batching, chunked prefill, speculative decoding, prefix caching, quantization, and multiple forms of distributed inference. Continuous batching allows the system to process requests as they arrive rather than waiting for a full batch, improving responsiveness and throughput. Speculative decoding uses smaller draft models to accelerate generation, while prefix caching reuses computations for shared prompt prefixes.

PyTorch has described vLLM as a high-throughput, memory-efficient inference and serving engine that supports a range of hardware back ends, including NVIDIA and AMD GPUs, Google TPUs, AWS Trainium, and Intel processors. This broad hardware support makes it a versatile choice for diverse deployment scenarios, from on-premises clusters to cloud environments.

## Features and Capabilities

vLLM provides an OpenAI-compatible API, allowing developers to integrate it with existing applications and tools that expect a standard interface. This compatibility simplifies migration from other serving solutions and enables seamless use with frameworks like LangChain and LlamaIndex.

Quantization support in vLLM includes techniques such as INT8 and INT4 weight quantization, which reduce model size and memory footprint while maintaining acceptable accuracy. This is particularly important for deploying large models on resource-constrained hardware.

Distributed inference capabilities allow vLLM to scale across multiple GPUs or nodes, using tensor parallelism and pipeline parallelism to handle models that exceed the memory of a single device. This is essential for serving frontier-scale models with hundreds of billions of parameters.

## Ecosystem and Adoption

vLLM has become a foundational component in the AI infrastructure ecosystem, with integrations from major cloud providers and hardware vendors. For example, [AWS](https://www.wikiprompt.org/wiki/amazon-web-services) supports vLLM on its [Trainium](https://www.wikiprompt.org/wiki/aws-trainium) chips, and [Google Cloud](https://www.wikiprompt.org/wiki/google-cloud) offers it on TPUs. [AMD](https://www.wikiprompt.org/wiki/amd) and [Intel](https://www.wikiprompt.org/wiki/intel) have also optimized vLLM for their accelerators, broadening its reach beyond NVIDIA GPUs.

The project's open-source nature has fostered a vibrant community of contributors, with regular releases and a growing set of features. Its adoption by startups and enterprises alike has made it a benchmark for inference performance, often used in comparisons with other frameworks like [Ollama](https://www.wikiprompt.org/wiki/ollama) and SGLang.

## See Also

- [Ollama](https://www.wikiprompt.org/wiki/ollama)
- SGLang
- [TensorRT-LLM](https://www.wikiprompt.org/wiki/tensorrt-llm)
- llama.cpp
- OpenVINO
- [Open Neural Network Exchange](https://www.wikiprompt.org/wiki/onnx)
- Comparison of deep learning software
- Comparison of machine learning software
- List of software developed at universities
- Lists of open-source AI software
- List of large language models
- TurboQuant
- 2025–present global memory supply shortage

---
Source: https://www.wikiprompt.org/wiki/vllm-project
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T02:31:32.906853+00:00