vLLM Deep Intuition

1. One-Sentence Essence

vLLM is an operating system for the KV cache: it treats GPU memory like virtual memory — paging the attention cache into fixed-size blocks — so that a single GPU can keep dozens of generations in flight at once without wasting memory or stalling.

Everything else vLLM does — the throughput numbers, the OpenAI-compatible server, the scheduler, the quantization support — is downstream of one decision: stop treating each request’s memory as a contiguous slab, and start treating it as pages in a virtual address space. If you internalize that one idea, the rest of vLLM stops being a grab-bag of features and becomes a single coherent system.

---

2. The Problem It Solved

To understand why vLLM exists, you have to understand what was painful about serving a large language model in 2022, before it appeared.

When a transformer generates text autoregressively, it produces one token at a time. To avoid recomputing attention over the entire sequence at every step, it caches the key and value vectors for every token it has already seen — the KV cache. This cache is the dominant consumer of GPU memory during serving. For a 13B model, a single token’s worth of KV cache is on the order of hundreds of kilobytes; a long conversation of thousands of tokens, multiplied across many concurrent users, runs into tens of gigabytes fast.

The naive way to manage this — the way every serving system did it before vLLM — was to pre-allocate one contiguous chunk of memory per request, sized to the maximum possible sequence length. If your model supports 2048 tokens, every request reserves room for 2048 tokens’ worth of KV cache up front, even if it only ends up generating 30. The result was catastrophic waste. Measured utilization of that reserved memory sat around 20–40%. The rest was internal fragmentation (reserved-but-unused space inside a request’s slab) and external fragmentation (free gaps between slabs too small to hold a new request). You were paying for an A100 and using a third of it.

There was a second, compounding problem: batching. GPUs are throughput monsters that are starved by serial work. To use one efficiently you must process many requests at once. But requests don’t finish at the same time — one user asks for a 10-token answer, another for 800. Traditional static batching waited for the entire batch to finish before starting the next one, so the whole batch ran at the speed of its slowest member, and the GPU sat idle as finished requests held their slots hostage.

In 2023, Woosuk Kwon and colleagues at UC Berkeley’s Sky Computing Lab published Efficient Memory Management for Large Language Model Serving with PagedAttention and released vLLM alongside it. Their insight was almost embarrassingly simple in hindsight: operating systems solved fragmentation forty years ago, with virtual memory and paging. Apply the same idea to the KV cache. Don’t give a request a contiguous slab. Give it pages, allocated on demand, mapped through a page table. Combine that with continuous batching — swapping finished requests out and new ones in at every single decoding step, not every batch — and you push memory utilization above 90% and throughput up 2–4× over the previous state of the art. That combination is why vLLM became, within two years, the default open-source engine for production LLM deployment.

---

3. The Concepts You Need

These are the terms the rest of the document leans on. Read them once; they’re the vocabulary of the whole system.

The model and its memory:

• KV cache — The stored key and value vectors for every token already processed in a sequence. It exists so the model doesn’t recompute attention over the whole history at each step. It grows linearly with sequence length and is the main thing competing for GPU memory. This is the resource vLLM is built to manage.
• Prefill — The first forward pass over a request’s prompt. All prompt tokens are processed at once (they’re already known), producing the initial KV cache and the first output token. Prefill is compute-bound: lots of matrix math, little memory traffic relative to compute.
• Decode — Every subsequent step, generating one token at a time. Each decode step reads the entire KV cache to compute attention for one new token. Decode is memory-bound: little compute, but it must stream the whole cache through the GPU. Prefill and decode having opposite bottlenecks is a theme that explains much of vLLM’s scheduler design.

PagedAttention’s vocabulary:

• Block (page) — A fixed-size chunk of KV cache holding a fixed number of tokens, typically 16. The atomic unit of allocation. (Note: this is vLLM’s paging block, unrelated to a CUDA thread block.)
• Logical blocks — A sequence’s view of its own cache: block 0, block 1, block 2… contiguous and simple, the way the model code thinks about it.
• Physical blocks — The actual locations in GPU memory where those blocks live. They can be anywhere, in any order.
• Block table (page table) — The per-sequence mapping from logical block → physical block. This indirection is the entire trick: logical contiguity, physical scattering.
• Copy-on-write (COW) — When two sequences share a physical block and one needs to diverge, vLLM copies just that one block before modifying it. Borrowed directly from OS fork().

Scheduling and serving:

• Continuous batching (a.k.a. iteration-level scheduling) — Re-deciding the batch composition at every decoding step rather than every batch. Finished requests are evicted and waiting ones admitted mid-flight.
• Waiting / running queues — New requests land in the waiting queue; once admitted (prefilled) they move to the running queue and decode until done.
• Chunked prefill — Splitting a long prompt’s prefill across multiple steps so it can share batches with decode work, instead of monopolizing a step.
• Token budget (max_num_batched_tokens) — The cap on how many tokens a single forward pass may process. The scheduler packs work up to this budget each step.
• Preemption — When the system runs out of KV cache, vLLM kicks a running request out (either swapping its cache to CPU, or discarding and recomputing it later). The robustness valve — and a performance red flag.

Optimizations you’ll tune:

• Automatic prefix caching (APC) — Reusing cached KV blocks across requests that share a prefix (e.g., the same system prompt). Enabled by default in modern vLLM.
• Speculative decoding — Using a cheap “draft” mechanism to propose several tokens at once, then verifying them in a single forward pass of the big model, to cut latency.
• Tensor parallelism (TP) — Splitting each layer’s weights across GPUs so they cooperate on every token. The standard way to fit a model too big for one card.
• Pipeline parallelism (PP) — Splitting the model by layers across GPUs (or nodes), each holding a contiguous stack of layers.
• Quantization — Storing weights (and optionally the KV cache) in lower precision — FP8, FP4, INT4 — to fit more in memory and move it faster.

---

4. The Distilled Introduction

This is the section that replaces the tutorials. By the end you should be able to install vLLM, serve a model, hit it with requests, and understand the workflow well enough to operate it.

Installation and the two ways to use it

vLLM is a Python library with heavy CUDA kernels underneath. On a Linux box with an NVIDIA GPU and recent CUDA drivers:

pip install vllm

That’s the common case. In production you almost always use the official Docker image instead, because it pins the exact CUDA/PyTorch/kernel combination that’s known to work — getting that stack right by hand is a real source of pain:

docker run --gpus all --ipc=host -p 8000:8000 \
  -e HUGGING_FACE_HUB_TOKEN=your_token \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct

There are two fundamentally different ways to drive vLLM, and conflating them is a common beginner confusion:

1. The offline LLM class — for batch jobs, evals, and offline generation. You hold the engine in your own Python process and feed it a list of prompts:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
params = SamplingParams(temperature=0.8, max_tokens=256)
outputs = llm.generate(["The future of AI is", "Explain entropy:"], params)
for o in outputs:
    print(o.outputs[0].text)

Note you pass a list of prompts. vLLM batches them internally — this is where the throughput lives. Feeding prompts one at a time in a loop defeats the entire point.

2. The OpenAI-compatible server — for online serving. This is how vLLM is used in production. You launch a server that speaks the OpenAI API:

vllm serve meta-llama/Llama-3.1-8B-Instruct

Now any OpenAI client works against it — you just point the base URL at your server:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
)
print(resp.choices[0].message.content)

This OpenAI compatibility is a huge part of vLLM’s adoption: you can swap a managed API for self-hosted vLLM by changing one URL.

The workflow, conceptually

Whichever entry point you use, the engine does the same thing. You submit requests with sampling parameters (temperature, top-p, top-k, max tokens, stop sequences, etc.). The engine puts them in a waiting queue. Its scheduler admits as many as the KV cache can hold, runs prefill on their prompts, then loops through decode steps generating tokens. As requests finish, their memory is freed and waiting requests are admitted — continuously, not in fixed batches. Output tokens stream back as they’re produced.

You don’t manage batches. You don’t manage memory. You submit requests and tune a handful of knobs. The engine does the rest. That’s the product.

The handful of knobs you’ll actually set

Out of dozens of engine arguments, these are the ones you touch on day one:

• --model — the HuggingFace repo ID or local path.
• --tensor-parallel-size N — shard the model across N GPUs on one node. Use this when the model doesn’t fit on one card. Set it to the number of GPUs you’re giving this model.
• --gpu-memory-utilization 0.90 — what fraction of each GPU’s VRAM vLLM is allowed to claim (default 0.9). vLLM pre-allocates this much at startup: model weights first, then everything left over becomes KV cache. Higher = more cache = more concurrency, but leave headroom or you OOM.
• --max-model-len N — the maximum context length (prompt + output) you’ll support. Lowering it below the model’s native max frees memory for more concurrent requests.
• --max-num-seqs N — max concurrent sequences in a batch (default 256). This caps concurrency.
• --max-num-batched-tokens N — the per-step token budget. Bigger favors throughput; smaller favors latency under bursty load.
• --dtype / --quantization / --kv-cache-dtype — precision controls. fp8 weights and fp8 KV cache roughly halve memory at a small accuracy cost.
• --enable-prefix-caching — reuse KV blocks across shared prefixes (on by default in current versions).

A realistic production launch for a 70B model on two H100s looks like:

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 2 \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 512 \
  --max-num-batched-tokens 65536 \
  --enable-chunked-prefill

You don’t need to understand every flag yet — the Judgment Calls section (§8) explains when and why to change each. The point for now: this is the whole surface area. vLLM is a system you tune, not one you program.

Monitoring

vLLM exposes Prometheus metrics at /metrics. The two you watch obsessively are vllm:gpu_cache_usage_perc (how full the KV cache is — if it’s pegged at 100%, you’re memory-starved and probably preempting) and vllm:num_requests_waiting (queue depth — if it climbs, you’re under-provisioned). These two numbers tell you almost everything about whether your deployment is healthy. We’ll return to them in How It Breaks (§10).

---

5. The Mental Model

Three ideas. Hold these and you can predict vLLM’s behavior without reading the docs.

Core Idea 1: The KV cache is virtual memory, and vLLM is its OS.

A sequence sees a clean, contiguous run of logical blocks. Physically those blocks are scattered across GPU memory, located on demand, one 16-token page at a time, via a block table. This is exactly how an operating system gives a process the illusion of a contiguous address space while physical RAM is fragmented and shared.

What it predicts:

• Memory waste drops to near zero. A request only ever holds the pages it’s actually filled, plus at most one partially-filled page. Internal fragmentation is bounded by one block (≤16 tokens); external fragmentation is gone entirely because any free page fits any request. This is why utilization jumped from ~30% to >90%.
• Sharing is free and automatic. If two requests share a prompt prefix, their block tables can point at the same physical blocks. That’s prefix caching (§7). When one must diverge, copy-on-write clones just the one affected block. Parallel sampling, beam search, and shared system prompts all fall out of this for free.
• Concurrency is a memory-arithmetic problem, not a configuration mystery. How many requests can you serve at once? However many fit in (total VRAM × utilization − model weights) ÷ KV-cache-per-token. Every concurrency question reduces to this division. If you understand it, the tuning knobs stop being magic.

Core Idea 2: The batch is reassembled every single step.

Traditional serving picks a batch, runs it to completion, repeats. vLLM re-decides the batch composition at every decoding iteration. A request that finishes at step 200 frees its pages immediately, and a waiting request is admitted at step 201 — not after the whole batch drains.

What it predicts:

• The GPU stays saturated regardless of how mismatched request lengths are. A 10-token reply and an 800-token reply coexist; the short one leaves early, its slot instantly reused. No request waits on the slowest member of its cohort.
• Throughput is dominated by how full you keep the batch, which is dominated by how much KV cache you have. More cache → more simultaneous sequences → higher GPU utilization → higher throughput. This is why gpu-memory-utilization is the single most consequential knob.
• Latency and throughput trade off through batch size. A fuller batch means each request shares the GPU with more neighbors, so its per-token latency rises even as aggregate tokens/sec rises. There is no setting that maximizes both. (See §8.)

Core Idea 3: Prefill and decode are opposite workloads, and the scheduler’s job is to mix them well.

Prefill is compute-bound (crunching a whole prompt at once). Decode is memory-bound (streaming the whole cache to emit one token). Run them separately and you alternately waste compute and waste memory bandwidth. The modern scheduler, especially with chunked prefill, deliberately mixes a little prefill and a lot of decode into the same forward pass so the GPU’s compute units and its memory bus are both busy.

What it predicts:

• A flood of new long prompts will spike your existing users’ inter-token latency, because prefill steals compute from decode — unless chunked prefill is breaking those prompts into bites that interleave politely.
• There’s an inherent tension between time-to-first-token (TTFT) and inter-token-latency (ITL). Prioritize prefill and new requests answer fast but ongoing ones stutter; prioritize decode and ongoing generation is smooth but new requests wait. The max_num_batched_tokens budget is where you set that dial.
• Tuning vLLM is mostly tuning this mix, which is why so many of the knobs are really about how the scheduler packs each step.

---

6. The Architecture in Plain English

Let’s follow a request from HTTP to tokens and watch where state lives.

A request arrives at the API server (the vllm serve front end) as an OpenAI-format JSON call. The server tokenizes the prompt and hands a request object to the engine core. In the modern V1 architecture (rearchitected in early 2025), the API server and the engine core run in separate processes, connected by a fast IPC channel. This matters more than it sounds: as GPUs got faster, the CPU work — tokenizing, scheduling, detokenizing, streaming HTTP — became the bottleneck, so V1 isolates that CPU work in its own process so it can run in parallel with GPU execution instead of blocking it.

Inside the engine core sits the scheduler. It owns two queues (waiting, running) and the KV cache manager — the block allocator that hands out and reclaims physical pages. At each step the scheduler decides: which requests run this iteration, and how many tokens of each. It packs work up to the token budget (max_num_batched_tokens), preferring to keep all running decodes going and filling leftover budget with new prefills (chunking a prefill if it won’t fit). It asks the KV cache manager to allocate blocks for any new tokens. If the manager has no free blocks, the scheduler preempts a running request to make room.

The scheduler emits a compact description of the step — which sequences, which block tables, which token positions — and sends it to the worker(s). Here’s a key V1 design point: workers are now stateful. Instead of the scheduler shipping the full request state every step (expensive, and it grows with batch size), it ships only the diff since last step, and the worker maintains a persistent batch of input tensors that it incrementally patches. This is the “near-zero CPU overhead” goal in practice.

Each worker owns one GPU. It runs the model forward pass. The attention layers use vLLM’s custom PagedAttention kernel, which — unlike a normal attention kernel that assumes the KV cache is one contiguous tensor — walks the block table and gathers the scattered physical pages. This kernel is the beating heart of the system; it’s what lets paged memory work without killing GPU performance. Modern vLLM pairs it with FlashAttention 3-class kernels to handle the highly dynamic batches (mixed prefill+decode) that V1 produces.

If you set --tensor-parallel-size > 1, there are multiple workers, one per GPU, and each holds a shard of every layer’s weights. They run the forward pass in lockstep, exchanging partial results via NCCL all-reduce after the sharded matrix multiplies. The KV cache is also sharded across them. They share a single global view of the paging system, so the scheduler reasons about cache as one pool.

The forward pass produces logits for the next token of every sequence. The sampler (now running on-GPU in V1) applies temperature, top-p, top-k, penalties, and any structured-output constraints, and picks the next token for each. Those tokens go back to the engine core, which appends them to each sequence (allocating a new block when a sequence crosses a 16-token boundary), checks stop conditions, and streams finished tokens back through the API server to the client. Then it loops — re-scheduling, re-deciding the batch — for the next step.

Where state lives, the question that unlocks everything: the authoritative request state (queues, which blocks belong to whom) lives in the scheduler/KV-cache-manager in the engine-core process on CPU. The actual KV cache data lives in GPU memory as physical blocks. The model weights live in GPU memory, sharded if TP>1. The input tensors for the current step live persistently on the worker, patched by diffs. No single component holds it all — and that separation is exactly what lets the CPU-bound and GPU-bound parts run concurrently.

---

7. The Things That Bite You

Each of these surprises someone in their first months. Each connects back to the mental model.

1. Concurrency-of-one looks slow, and people conclude vLLM is slow. What you’d expect: send a request, measure latency, it represents vLLM’s speed. What actually happens: with one request in flight, the batch (Core Idea 2) has one member, the GPU is massively underutilized, and per-request latency is unremarkable. vLLM’s win is throughput under concurrency, not single-stream latency. Benchmarking with a serial client — one request, wait, next request — measures the one thing vLLM isn’t optimized for and makes it look mediocre. Handle it: always benchmark with concurrent load (vllm bench serve or a real concurrent client). If GPU utilization is under 70% while requests queue, you’re not sending enough concurrent load, not hitting a vLLM limit.

2. gpu-memory-utilization is pre-allocated, so “it fit yesterday” doesn’t mean it fits today. What you’d expect: vLLM uses memory as needed. What actually happens: it grabs that fraction of VRAM at startup and carves KV cache out of the leftover (Core Idea 1). If another process (a logging sidecar, a second model, a notebook) is already holding VRAM, vLLM’s “90%” is 90% of a smaller pie, the KV cache shrinks, and you silently get far less concurrency — or an OOM at load. Handle it: give vLLM a clean GPU, account for co-tenants, and watch gpu_cache_usage_perc to confirm you got the cache you expected.

3. Preemption silently wrecks your tail latency. What you’d expect: if the system is overloaded, requests just queue. What actually happens: when KV cache fills mid-generation, vLLM preempts in-flight requests — either swapping their cache to CPU or discarding and recomputing it later. You’ll see a log line about preemption by RECOMPUTE mode. Throughput holds, but the preempted requests’ latency spikes hideously, poisoning your p99. Handle it: treat any preemption in steady state as a capacity bug. Raise gpu-memory-utilization, lower max-num-seqs, shorten max-model-len, or add GPUs via tensor parallelism — the warning itself lists these.

4. max-model-len set to the model’s max can be a self-inflicted wound. What you’d expect: set the longest context the model supports, to be safe. What actually happens: KV cache is reserved per the maximum sequence the system must support; a huge max-model-len means each potential sequence reserves a large worst-case footprint, slashing how many you can run concurrently. Handle it: set max-model-len to what your workload actually uses, not the model’s theoretical ceiling. Dropping from 128K to 16K can multiply your concurrency.

5. Quantizing weights does not shrink the KV cache. What you’d expect: “I quantized to FP8, memory problem solved.” What actually happens: --quantization fp8 shrinks the weights. The KV cache is a separate pool and stays in its original precision unless you also set --kv-cache-dtype fp8. For long-context, high-concurrency workloads the cache, not the weights, is the binding constraint. Handle it: if you’re memory-bound on concurrency rather than on fitting the model, the KV-cache dtype is the lever that matters.

6. Combining bleeding-edge features can quietly corrupt outputs, not just crash. What you’d expect: features are independent; enable what you like. What actually happens: feature interactions are where vLLM’s sharp edges live. Prefix caching + certain speculative-decoding methods have, in real releases, produced ~20% accuracy drops or kernel crashes on specific model families — a silent quality regression, the worst kind. Handle it: when you stack APC + speculative decoding + an exotic architecture (e.g., Mamba/hybrid models), run an accuracy eval, not just a smoke test, and pin your vLLM version.

7. Tensor parallelism only works within a node, and the network is the catch. What you’d expect: set tensor-parallel-size 8 and it spans whatever GPUs you have. What actually happens: TP all-reduces after every layer, which is extremely bandwidth-hungry. It’s designed for GPUs on one node connected by NVLink. Stretch TP across nodes over ordinary Ethernet and the all-reduce latency dominates everything. Handle it: use TP within a node; use pipeline parallelism (or data-parallel replicas) across nodes.

8. The “first request after startup” is slow, and it’s not a bug. What you’d expect: once it says ready, it’s at full speed. What actually happens: vLLM compiles CUDA graphs and warms kernels on first use; the very first requests pay that one-time cost. Handle it: send warmup requests before putting an instance into a load balancer’s rotation, and don’t include cold-start requests in benchmarks.

---

8. The Judgment Calls

These are the decisions that separate someone who runs vLLM from someone who operates it.

1. Throughput vs. latency: where to set the batch. Situation: you can’t maximize aggregate tokens/sec and minimize per-request latency at once (Core Idea 2). Throughput-optimized: high max-num-seqs (512+), large max-num-batched-tokens (64K+), high memory utilization — for batch/offline jobs, evals, async pipelines. Latency-optimized: lower concurrency caps, smaller token budget — for interactive chat where TTFT and smooth streaming matter. What experienced people do: decide which SLO you’re serving first, then tune toward it; don’t try to split the difference and get a deployment that’s mediocre at both. The tell: if it’s a chatbot, optimize latency; if it’s a nightly summarization job, optimize throughput.

2. How aggressive to set gpu-memory-utilization. Situation: higher = more KV cache = more concurrency, but less headroom. Conservative (0.85): safe when you have co-tenant processes, spiky inputs, or activation spikes from long prompts. Aggressive (0.92–0.95): squeezes maximum concurrency from a dedicated GPU. What experienced people do: push it up on a clean, dedicated GPU until gpu_cache_usage_perc is healthy and you see zero preemptions under peak load, then back off a notch for safety. Treat any startup OOM or steady-state preemption as the signal you went too far.

3. Tensor parallelism vs. quantization to fit a big model. Situation: a 70B model won’t fit on one GPU. TP: split across GPUs, keep full precision, pay NVLink-bandwidth overhead and tie up N cards. Quantization (FP8/FP4): fit on fewer GPUs at a small accuracy cost. What experienced people do: quantize first — FP8 on modern hardware is nearly free in quality and saves whole GPUs — and reach for TP only when even the quantized model won’t fit or when you need the aggregate memory bandwidth of multiple cards for throughput. Combine them for the largest models.

4. Tensor parallelism vs. data-parallel replicas for scaling throughput. Situation: one GPU’s worth of model fits, but you need more total throughput. More TP: one model instance spread wider — doesn’t add throughput once the model already fits; TP is for fitting, not scaling. Data-parallel replicas: run N independent vLLM instances behind a load balancer. What experienced people do: if the model already fits on the GPUs you’d use for TP, run replicas instead — N copies serve N× the traffic and degrade independently. Reserve TP for the fitting problem.

5. Chunked prefill: on or off. Situation: long prompts arriving alongside ongoing generation. Off: prefill monopolizes steps, spiking existing users’ ITL when a big prompt lands. On: prefill is broken into budget-sized chunks that interleave with decode, smoothing latency and improving GPU utilization (Core Idea 3). What experienced people do: enable it for any interactive workload with variable prompt lengths; it’s increasingly the default. The one caution is tuning max-num-batched-tokens so chunks are large enough to be efficient but small enough to keep decode responsive.

6. Prefix caching: when it actually pays. Situation: APC reuses KV blocks across shared prefixes. Big win: large shared system prompts, few-shot templates, multi-turn chat replaying history, RAG with a fixed instruction block — anything where many requests share a long head. Negligible: high-entropy, all-unique prompts, where you pay bookkeeping for no reuse. What experienced people do: leave it on (it’s default and rarely hurts), but know whether your traffic actually has shared prefixes before crediting it for performance — and re-verify accuracy if you combine it with speculative decoding (§7.6).

7. Speculative decoding: worth the complexity? Situation: you want lower latency per request. Worth it: latency-critical, lower-concurrency serving where the GPU has spare compute to verify draft tokens; predictable domains (code, structured output) where draft acceptance is high. Not worth it: already-saturated high-throughput batch serving — speculation spends the spare compute you don’t have, and can reduce throughput. What experienced people do: treat it as a latency tool for under-utilized GPUs, measure the acceptance rate (low acceptance means you’re wasting verification compute), and watch for the feature-interaction bugs in §7.6.

8. KV cache precision: FP16 vs FP8. Situation: you’re concurrency-bound, not weight-bound. FP16 KV: full fidelity, half the concurrency. FP8 KV: roughly doubles how many sequences fit, at a small and usually acceptable accuracy cost. What experienced people do: for long-context, high-concurrency serving, FP8 KV cache is often the single highest-leverage memory win — bigger than weight quantization, because at long context the cache dwarfs the weights. Validate on your eval set before trusting it in a quality-sensitive setting.

9. Pin the version, or ride latest. Situation: vLLM ships fast and changes a lot. Latest: newest models, kernels, speed. Pinned: stability, reproducibility, known-good feature interactions. What experienced people do: pin a specific version (and the matching Docker image) in production, upgrade deliberately with an eval gate, and never let “we’ll just use :latest” decide your accuracy for you. The feature-interaction landmines (§7.6) make this non-negotiable for quality-sensitive workloads.

10. vLLM vs. a managed API in the first place. Situation: should you self-host at all? Self-host vLLM: high sustained volume where per-token economics dominate, data-residency/privacy needs, custom or fine-tuned models, full control of latency. Managed API: spiky or low volume, no GPU ops capability, want zero operational burden. What experienced people do: self-host when your GPUs would stay busy — vLLM’s economics come from high utilization (Core Idea 2). A vLLM instance serving three requests an hour is more expensive and more painful than an API call. The break-even is about utilization, not ideology.

---

9. The Commands and APIs That Actually Matter

Grouped by task, with the why.

Serving:

vllm serve <model> [flags]      # launch the OpenAI-compatible server

The flags that carry the weight, all explained above: --tensor-parallel-size, --gpu-memory-utilization, --max-model-len, --max-num-seqs, --max-num-batched-tokens, --quantization, --kv-cache-dtype, --enable-chunked-prefill, --enable-prefix-caching, --dtype. Memorize this set; it’s 90% of operating vLLM.

Offline generation:

from vllm import LLM, SamplingParams
llm = LLM(model="...", tensor_parallel_size=2, gpu_memory_utilization=0.9)
out = llm.generate(prompts_list, SamplingParams(temperature=0.7, max_tokens=512))

Same engine, same knobs, batch interface. The critical habit: pass all your prompts as one list so the engine can batch them.

Sampling parameters worth knowing: temperature, top_p, top_k, max_tokens, stop, presence_penalty/frequency_penalty, n (multiple samples — cheap thanks to prefix sharing), logprobs, and guided_json/guided_regex/guided_grammar for structured outputs (constraining generation to valid JSON/schema, vastly more reliable than asking nicely in the prompt).

Multi-GPU and scaling, conceptually:

• --tensor-parallel-size N — fit a model across N GPUs in one node.
• --pipeline-parallel-size N — split layers across nodes when TP can’t cross the network (§7.7).
• Run multiple independent vllm serve processes behind a load balancer (set CUDA_VISIBLE_DEVICES per instance) for data-parallel throughput scaling (§8.4).

Observability:

GET /metrics      # Prometheus metrics
GET /health       # liveness

The metrics that matter: vllm:gpu_cache_usage_perc, vllm:num_requests_waiting, vllm:num_requests_running, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds, and any preemption counter. Wire these to dashboards before you go live.

Benchmarking:

vllm bench serve   # drive concurrent load against a running server

Use this — with concurrency — to find your real throughput and the latency/throughput curve, rather than guessing or measuring serially (§7.1).

---

10. How It Breaks

When something’s wrong, this is where to look.

Failure: CUDA out of memory at startup. Symptoms: engine dies during model load or KV-cache profiling. Root cause: model weights + requested KV cache exceed available VRAM (Core Idea 1), often because another process holds memory or gpu-memory-utilization/max-model-len is too high. Diagnose: nvidia-smi for co-tenants; check the startup logs for how much it tried to allocate. Fix: lower gpu-memory-utilization, lower max-model-len, quantize weights, or add TP to spread weights across GPUs.

Failure: throughput far below expectations, GPU under 70% busy. Symptoms: num_requests_running low, GPU idle, yet requests queue. Root cause: you’re not feeding enough concurrent load, or max-num-seqs/max-num-batched-tokens is throttling the batch (Core Idea 2). Diagnose: check whether your client is actually concurrent (§7.1); check gpu_cache_usage_perc — if it’s low, you have spare cache and should raise concurrency caps. Fix: raise max-num-seqs and the token budget; fix the client to send concurrent requests.

Failure: tail latency spikes, p99 erratic. Symptoms: most requests fine, some catastrophically slow; preemption warnings in logs. Root cause: KV cache saturating and preempting in-flight requests (§7.3). Diagnose: watch gpu_cache_usage_perc pegged near 100% and the preemption counter rising. Fix: more cache (raise utilization, FP8 KV, more GPUs) or less load per instance (lower max-num-seqs, add replicas).

Failure: output quality regressed after a change. Symptoms: eval scores drop, no crash. Root cause: a feature interaction (prefix caching × speculative decoding × exotic architecture, §7.6) or an aggressive quantization. Diagnose: bisect — disable speculative decoding, then prefix caching, then revert KV-cache dtype, re-running the eval each time. Fix: drop the offending combination and/or pin to a known-good version.

Failure: model won’t load / “trust remote code” / tokenizer errors. Symptoms: errors during model init. Root cause: model architecture not yet supported by your vLLM version, missing --trust-remote-code, or a version/kernel mismatch. Fix: check the supported-models list for your version; upgrade (or pin) to one that supports the architecture; use the official Docker image to avoid stack mismatches.

The general debugging workflow: (1) nvidia-smi — who holds the GPU, how full is it. (2) Startup logs — how much KV cache vLLM actually got. (3) /metrics — cache usage, queue depth, running count, preemptions. (4) Reproduce with a concurrent benchmark, not a serial one. (5) If quality is the issue, bisect features and check the version. In that order, those five steps resolve the large majority of real incidents.

---

11. The Downsides / Disadvantages

Honest accounting. vLLM is excellent; here’s what adopting it actually costs you.

1. It is a memory-tuning system, and that tuning never fully goes away. Where it comes from: the entire design (Core Idea 1) makes concurrency a function of a memory budget you must hand-balance across weights, cache, context length, and headroom. What it costs: every model, every GPU type, every workload shape requires re-tuning a handful of interacting knobs, and the configs don’t transfer — change the hardware and you re-tune. Dealbreaker when: you have many small, heterogeneous deployments and no one to own the tuning. Livable when: you have a few high-volume models worth tuning carefully once.

2. Single-stream latency is not its strength. Where it comes from: the throughput win comes from batching many requests (Core Idea 2); a lone request gets none of that benefit and some scheduler overhead. What it costs: if your use case is genuinely one-request-at-a-time and latency-critical, vLLM’s headline advantage evaporates and a leaner setup might beat it. Dealbreaker when: low-concurrency, ultra-low-latency, single-stream. Livable when: you have real concurrency, which is most production serving.

3. The release pace is a double-edged sword. Where it comes from: vLLM moves extraordinarily fast — new models and kernels land constantly. What it costs: APIs shift, defaults change between versions, and feature interactions break in ways that sometimes degrade quality silently rather than failing loudly (§7.6). You must pin versions and gate upgrades with evals — operational discipline that “free open source” quietly demands. Dealbreaker when: you have no capacity for an eval gate and need set-and-forget stability for years. Livable when: you can invest in a real upgrade process. What people think mitigates it but doesn’t: “we’ll just use :latest” — that’s how you ship a 20% accuracy regression you didn’t measure.

4. It is a GPU-serving engine, not a platform. Where it comes from: vLLM deliberately does one thing — efficient inference on one model instance. What it costs: autoscaling, multi-model routing, request prioritization across tenants, gateway features, A/B routing, billing — all of that is your problem, built around vLLM with Kubernetes, a gateway, a load balancer, and glue. The “free” engine sits inside a non-trivial platform you build and operate. Dealbreaker when: you wanted a turnkey serving platform. Livable when: you have platform engineers, or you adopt the surrounding ecosystem (e.g., the production-stack / router projects) deliberately.

5. Failure modes degrade gracefully into expensive confusion. Where it comes from: preemption (§7.3) keeps the system up under overload by trading latency, and feature bugs regress quality without crashing. What it costs: the system rarely fails loudly. It gets slow at the tail, or subtly wrong, and you only notice if you’re watching the right metrics and running evals. The cost is the monitoring discipline required to even see the problems. Dealbreaker when: you can’t invest in observability. Livable when: you wire up /metrics and evals from day one.

6. Hardware and stack sensitivity. Where it comes from: peak performance depends on specific kernels (FlashAttention-class), specific GPU capabilities (FP8 on Hopper+, NVLink for TP), and an exactly-matched CUDA/PyTorch stack. What it costs: on older GPUs, without NVLink, or on non-NVIDIA hardware, you get a meaningfully degraded subset of the performance the benchmarks promise, and getting the software stack right by hand is a recurring source of pain (hence the reliance on the official Docker image). Dealbreaker when: you’re locked to older or non-NVIDIA hardware and expect the headline numbers. Livable when: you’re on modern NVIDIA GPUs and use the provided images.

7. The configuration surface is genuinely large. Where it comes from: the breadth of optimizations — parallelism modes, quantization variants, scheduling policies, caching, speculation — each with its own flags and interactions. What it costs: there is real cognitive load in knowing which 8 of the dozens of knobs matter for your case, and the interactions (§7.6, §8) mean you can’t treat them as independent. The learning curve to operate it well (not just launch it) is steeper than “pip install” suggests. Dealbreaker when: never, really — but budget the ramp-up time honestly.

---

12. The Taste Test

What separates a config written by someone who understands vLLM from one copied off the first blog post.

Memory tuning — naive vs. understanding:

• Naive: --max-model-len 131072 “to be safe,” then puzzlement about low concurrency.
• Understands: --max-model-len 8192 because that’s what the workload uses, freeing cache for 4× the concurrent requests. Knows context length is a memory-budget decision (Core Idea 1), not a safety margin.

Quantization — naive vs. understanding:

• Naive: --quantization fp8 and declares the memory problem solved.
• Understands: adds --kv-cache-dtype fp8 too, because at high concurrency the cache is the binding constraint, not the weights (§7.5).

Scaling — naive vs. understanding:

• Naive: model fits on one GPU but throughput is low, so they bump --tensor-parallel-size to 4 and wonder why throughput barely moved.
• Understands: runs 4 replicas behind a load balancer, because TP fits models, replicas scale throughput (§8.4).

Benchmarking — naive vs. understanding:

• Naive: a for loop sending one request at a time, concludes “vLLM is only as fast as raw transformers.”
• Understands: vllm bench serve with realistic concurrency, reads the latency/throughput curve, knows a single stream measures the one thing vLLM doesn’t optimize (§7.1).

Operations — naive vs. understanding:

• Naive: :latest image, no metrics, finds out about preemption from user complaints.
• Understands: pinned version, /metrics on a dashboard, alerts on gpu_cache_usage_perc and preemption counters, warmup requests before load-balancer rotation, an eval gate on upgrades.

The red flags in a config review: a giant max-model-len with no justification; weight quantization without KV-cache dtype on a concurrency-bound deployment; TP set higher than needed to fit the model; :latest in production; no /metrics scraping; speculative decoding stacked with prefix caching on an exotic model and no accuracy eval. Any one of these says “configured by tutorial, not by understanding.”

---

13. Where to Go Deeper

• The PagedAttention paper — Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023). The source. Read it once for the OS-analogy framing and the fragmentation measurements; everything in vLLM descends from it. Read after you’ve internalized §5.
• The vLLM V1 blog post (A Major Upgrade to vLLM’s Core Architecture, Jan 2025). The clearest explanation of the modern process architecture, persistent batch, and why CPU overhead drove the rewrite. Read it to understand the system you’re actually running today.
• The official docs — the Optimization and Tuning, Engine Arguments, and V1 guide pages. Skip the marketing; these three are the operational core. Keep Engine Arguments open while you tune.
• The PagedAttention kernel design doc (in the vLLM docs). When you want to understand how paged memory is made fast on a GPU — the block-table-walking attention kernel — this is the page. For the curious; not required to operate vLLM.
• The Sarathi-Serve work on chunked prefill / stall-free scheduling. The research behind §8.5; read it to understand the prefill/decode mixing tradeoff (Core Idea 3) at depth.
• A hands-on project: stand up vllm serve on a single GPU, wire /metrics to a local Prometheus + Grafana, and drive it with vllm bench serve at increasing concurrency until you observe preemption. Watching gpu_cache_usage_perc climb to 100% and tail latency explode in real time teaches Core Ideas 1 and 2 better than any paragraph.

---

14. The Final Verdict

After all of it, here’s the honest take: vLLM is the closest thing the open-source world has to a “just works” answer for serving LLMs at scale — but “just works” is doing some load-bearing work in that sentence, because it works brilliantly exactly when you feed it the workload it was built for (real concurrency, GPUs you intend to keep busy) and works unremarkably when you don’t. It is not magic; it is good systems engineering, and like all good systems engineering its benefits are conditional on understanding the model underneath.

What it gets profoundly right: the central analogy. Recognizing that the KV-cache fragmentation problem was the same problem operating systems solved with paging — and then actually building the page table, the block allocator, the copy-on-write, the custom attention kernel to make it fast — is one of those ideas that feels obvious only after someone has had it. The 2–4× throughput gain wasn’t a clever hack; it was the natural consequence of finally modeling the problem correctly. The continuous-batching insight (reassemble the batch every step) is the same kind of move: stop accepting an artificial constraint everyone had stopped questioning. That intellectual clarity is why vLLM won.

What it costs you: perpetual tuning, version vigilance, and the quiet realization that the “free engine” lives inside a platform you have to build and operate. The failures are graceful, which sounds good until you understand that graceful failure means quiet failure — slow tails and silent quality regressions instead of honest crashes. The shape of the regret, if you feel it, is this: you adopted vLLM expecting a product and got an extremely good component, and the work of turning a component into a service was yours all along.

Who should reach for it: teams with sustained, concurrent inference volume on modern NVIDIA GPUs, who have (or will build) the observability and ops discipline to run it well, and who benefit from controlling their own models, latency, and per-token economics. Who shouldn’t: anyone with spiky, low-volume traffic, no GPU-ops capacity, or a genuine need for single-stream minimal latency — for them a managed API is cheaper in both dollars and pain, and reaching for vLLM would be a quiet mistake.

What you should now believe: believe that vLLM’s performance is real and conditional on keeping the GPU busy. Don’t believe a single-stream benchmark that says it’s slow — that benchmark is measuring the wrong thing. When someone says “we use vLLM,” understand they mean “we run an inference engine and built a platform around it,” not “we bought a serving solution.” And hold onto the one line worth carrying out of all of this: in vLLM, every performance question is a memory question in disguise — concurrency, throughput, latency, cost, they all reduce to how many tokens of KV cache you can hold and how full you keep the batch. Learn to do that division in your head, and you understand vLLM better than most people running it in production.

---

The ideas are mine. The writing is AI assisted