vLLM

vLLM is an open-source inference and serving engine for large language models, originally developed in the Sky Computing Lab at UC Berkeley. It is the default choice for teams who want to self-host open-weight models at production scale — delivering more requests per second from the same GPU hardware than any comparable serving system, with near-zero memory waste on the KV cache (the memory structure that stores computed context during generation). As of mid-2026, vLLM supports over 200 model architectures, has over 2,000 contributors, and is actively maintained by a broad open-source community.

How vLLM works

Serving an LLM at scale has two hard problems: memory and throughput. A standard inference server allocates a fixed block of GPU memory per request to store the key-value cache that the model needs to generate each token. Most of that memory sits idle for most of the request’s lifetime, which means the GPU is only partially utilised even when it appears busy. vLLM solves both problems with two core mechanisms.

PagedAttention is vLLM’s memory management system, inspired by virtual memory paging in operating systems. Instead of allocating a large contiguous block of GPU memory per request, PagedAttention divides the KV cache into small fixed-size pages and allocates them on demand. Pages for different requests can interleave on the GPU without wasting space between them. The result is near-zero memory fragmentation and the ability to run many more concurrent requests on the same hardware — typically 2–4x more than naive approaches.

Continuous batching is the scheduling mechanism that pairs with PagedAttention. In a traditional static-batching server, you wait for a full batch of requests before starting inference, then wait for all of them to finish before accepting new ones. This creates latency spikes and GPU idle time. Continuous batching processes requests as they arrive and inserts new ones into the running batch the moment a slot frees up. GPU utilisation stays consistently high regardless of request arrival rate.

On top of these two foundations, vLLM adds:

• Multi-GPU parallelism: Tensor parallelism splits model weight matrices across multiple GPUs so that individual layers run in parallel. Pipeline parallelism distributes model layers across GPUs for very large models that do not fit on a single device. Sequence parallelism handles long-context requests efficiently across GPU memory boundaries.
• Quantization: vLLM supports FP8, INT8, INT4, GPTQ, AWQ, and GGUF quantization formats. Running a 70B model in FP8 rather than BF16 roughly halves GPU memory requirements at minimal quality cost, making larger models accessible on smaller hardware.
• OpenAI-compatible API: The default serving mode exposes an HTTP endpoint that mirrors the OpenAI chat completions API. Any client built against OpenAI’s SDK connects to vLLM without code changes — just swap the base URL.

What you can build with vLLM

Self-hosted inference API for internal applications. Deploy vLLM on your own cloud infrastructure with Llama 3, Mistral, or Qwen, point your internal tools at the vLLM endpoint, and remove your dependency on OpenAI or Anthropic for applications that do not require frontier model capability. Data stays in your infrastructure, costs drop at scale, and you control the model version.

Production RAG backend with controlled latency. Run vLLM as the generation layer in a retrieval-augmented generation pipeline, tuned for the specific latency budget your product needs. Because vLLM’s continuous batching keeps GPU utilisation high, you can size your infrastructure to your 95th-percentile latency target rather than your peak throughput target, which reduces infrastructure cost significantly.

Model evaluation harness for rapid experimentation. Spin up a vLLM endpoint for each model version you want to evaluate — Llama 3.1 70B vs Qwen 2.5 72B vs your fine-tune — run your full eval suite against each, and compare quality metrics side by side. Because vLLM is free and fast, you can run evaluations at a scale that would be prohibitively expensive via hosted API.

Fine-tuned model serving after training. Once you have fine-tuned an open-weight model on your proprietary data using a tool like Axolotl or Unsloth, deploy the checkpoint with vLLM for production-grade serving. vLLM loads HuggingFace checkpoints natively and handles the infrastructure layer so you focus on the model, not the server.

Multi-tenant inference platform for an internal team or a SaaS product. vLLM’s continuous batching and memory efficiency make it practical to serve multiple tenants from a shared GPU pool. Pair it with a reverse proxy that handles authentication and rate limiting per tenant, and you have the core of a model inference platform without managing a Kubernetes cluster of single-user containers.

Is vLLM free?

vLLM is fully free and open-source under the Apache 2.0 license. There is no hosted tier, no enterprise edition with a paywall, and no usage fee. Your cost is entirely the GPU compute you provision to run it — a GPU cloud VM (AWS, GCP, Azure, Lambda Labs, RunPod) or your own hardware.

How to get started with vLLM

Install with pip install vllm. Start a server with vllm serve meta-llama/Llama-3.1-8B-Instruct (or any HuggingFace model ID). The server starts on port 8000 with an OpenAI-compatible endpoint. Point your OpenAI client at http://localhost:8000/v1 and your existing code works without modification. The vLLM documentation includes quickstarts for single-GPU, multi-GPU, and Docker deployments.

For a guided path through building production-grade AI systems — including model serving, agent architecture, and cloud deployment — the BuildrLabs Agentic AI Bootcamp covers the full stack in person over 16 weeks.