vLLM Tutorial: How to Deploy and Serve LLMs on GPU Cloud (2026)

vLLM is the de facto open-source inference engine for production LLM serving in 2026, running on GPU cloud, it delivers 3–5x more throughput than a naive PyTorch inference loop on the same H100, at a fraction of hyperscaler cost.

Most teams start with a model, a GPU instance, and a Python script. That works for one user. Under 10 concurrent requests, GPU utilisation drops to 30–40% and queue times spike. The fix is not bigger hardware. It is the right inference engine.

vLLM, originally built at UC Berkeley's Sky Computing Lab and now maintained by 2,000+ contributors including NVIDIA, Hugging Face, and Red Hat, solves this with two core innovations: PagedAttention and vLLM continuous batching. This vLLM tutorial walks through what those are, how to deploy vLLM on GPU cloud step by step, and how to pick the right GPU for your model size and budget.

This post is part of the packet.ai LLM inference cluster. For a full breakdown of inference cost across GPU models, see our guide to GPU cluster options on packet.ai.

How vLLM Works: PagedAttention and Continuous Batching Explained

Standard LLM inference pre-allocates a contiguous VRAM block for every request, sized to the maximum sequence length. On a workload with variable prompt lengths, 60–80% of that allocation sits empty. GPU utilisation hovers around 30–40% at moderate request rates, even on an H100.

vLLM solves this with two techniques that compound on each other.

PagedAttention applies OS virtual memory paging to KV cache management. Rather than one contiguous block per request, it divides VRAM into fixed-size 16-token blocks and allocates them on demand. Logical block tables map each request's attention positions to non-contiguous physical blocks. The result: memory waste drops by 55–80%, and the same GPU can serve significantly more concurrent requests. On an 80 GB H100 running a 7B FP16 model, this is the difference between 30 concurrent requests and 100+, depending on sequence length distribution (Kwon et al., SOSP 2023).

vLLM continuous batching operates at the decode-iteration level rather than the request level. At each step, the scheduler checks the queue. When one request finishes, a new one joins the active batch immediately, with no idle slots. Static batching leaves finished slots locked until the whole batch completes; continuous batching eliminates that waste entirely.

vLLM also ships with prefix caching (automatic prefix caching, APC), which reuses KV blocks across requests that share a system prompt or few-shot prefix. For RAG pipelines and agent workflows where every request starts with the same 2,000-token context, this cuts prefill compute to near zero for repeat prefixes.

The current stable release is v0.25.1 (July 2026). It requires Python 3.10–3.14 and an NVIDIA GPU with compute capability 7.0 or above (Volta and later: V100, T4, A100, H100, H200, B200, RTX 6000 Pro).

vLLM vs TGI vs SGLang: Which Inference Engine for 2026 Workloads

TGI is no longer a real option. Hugging Face moved TGI to maintenance mode in December 2025 and now contributes directly to vLLM and SGLang. TGI accepts minor patches and documentation fixes, but no new features are coming. The 2026 decision is vLLM versus SGLang.

The practical rule: use vLLM for 70B+ models, diverse model types, or when you want the largest community and broadest hardware support. Use SGLang if your workload is heavily prefix-shared: agent pipelines, RAG with fixed system prompts, or structured output at scale. The throughput gap on large models is 3–5%, which is within noise for most production workloads (Spheron H100 benchmarks, March 2026; Rawlinson 2026 dual-H100 benchmark).

On dual H100 serving Llama 3 70B, vLLM achieves 210ms TTFT at p50 and 20ms TPOT at p50; SGLang is roughly 10% ahead on TTFT, a difference that matters for interactive chat but not for batch workloads.

GPU Selection for vLLM: H100, H200, or B200: VRAM Math and Cost per Token

The right GPU for vLLM depends on two numbers: model size in VRAM and your target cost per million output tokens.

Quick VRAM sizing rule: a model in BF16 needs roughly 2 bytes per parameter. Llama 3.1 8B needs ~16 GB; Llama 3.1 70B needs ~140 GB. Add 20% headroom for KV cache at target concurrency. An H200 at 141 GB fits Llama 3.1 70B in BF16 with room for KV cache, while an H100 at 80 GB requires FP8 quantization or tensor parallelism across two GPUs for the same model.

packet.ai H200 SXM starts at $2.49/hr, which is 44% below the $4.50/hr on-demand median on AWS and Azure for the same GPU, with no egress markup and no multi-year lock-in.

On cost per million output tokens: for Llama 3.3 70B at FP8, a single H200 delivers approximately $0.50/M tokens versus $0.17/M tokens on a B200 (SemiAnalysis InferenceX benchmarks, 2026). If your workload runs above 50M tokens/month, the B200's higher hourly rate pays for itself quickly.

How to Deploy vLLM on GPU Cloud: Step-by-Step Setup

This section covers a single-GPU vLLM deployment on GPU cloud from instance launch to a working OpenAI-compatible API endpoint. The commands work on any H100 or H200 instance running Ubuntu 22.04 with CUDA 12.1+ and Docker.

docker pull vllm/vllm-openai:latest

docker run --gpus all \
  --ipc=host \
  -p 8000:8000 \
  --shm-size 8g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HUGGING_FACE_HUB_TOKEN=your_token_here \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 64

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "What is vLLM?"}],
    "max_tokens": 200
  }'

watch -n 1 nvidia-smi dmon -s u

Two flags matter most for production tuning: --gpu-memory-utilization controls how much VRAM vLLM reserves for KV cache (0.90 is safer than the default 0.95 on variable-length workloads); --max-num-seqs caps concurrent sequences in the batch. Start conservative and raise both after profiling real traffic.

vLLM Tensor Parallelism: Multi-GPU Config for 70B+ Models

A 70B model in BF16 needs roughly 140 GB of VRAM. That exceeds any single H100 (80 GB) and sits at the edge of a single H200 (141 GB) when KV cache is included. For these models, vLLM tensor parallelism splits every transformer layer across multiple GPUs. Each GPU holds a fraction of the attention heads and MLP weights and processes its shard in parallel.

One flag enables it: --tensor-parallel-size N, where N is the number of GPUs. It must equal your provisioned GPU count exactly.

# 70B model on 2x H200 (or 4x H100) via tensor parallelism
docker run --gpus all \
  --ipc=host \
  -p 8000:8000 \
  --shm-size 16g \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -e HUGGING_FACE_HUB_TOKEN=your_token_here \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --dtype float16 \
  --tensor-parallel-size 2 \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 64 \
  --enable-prefix-caching

NVLink-connected GPUs (H100 SXM, H200 SXM, B200 SXM) scale at roughly 1.84x per GPU added via tensor parallelism, near-linear. PCIe-connected consumer cards (RTX 4090, L40S) scale at roughly 1.4x per card, because PCIe bandwidth (~64 GB/s bidirectional) becomes the bottleneck for all-reduce operations between GPUs. For large-model tensor parallelism, SXM interconnect is not optional.

For models above a single node (Llama 3.1 405B on 8x H200 clusters or DeepSeek V3 on multi-node B200), vLLM uses Ray for distributed orchestration and NCCL for inter-GPU communication. Plan for at least 100 Gbps InfiniBand between nodes; standard 10 Gbps Ethernet causes NCCL to become the bottleneck for models above 70B parameters.

vLLM Quantization: FP8, AWQ, and When to Use Each

vLLM quantization reduces VRAM requirements and increases throughput by lowering the bit-width of model weights and activations. Three formats matter in practice for 2026 GPU hardware.

FP8 is the right default for H100 and H200 on production workloads where you control the checkpoint. AWQ is better when you need to run an off-the-shelf 70B model on a single 80 GB H100 without tensor parallelism. It cuts VRAM from ~140 GB to ~70 GB with acceptable quality. GPTQ is worth considering when an AWQ checkpoint is not publicly available for your target model.

B200 adds native FP4 support (NVFP4), which doubles effective memory bandwidth utilisation versus FP8 and is the cost-per-token frontier for large-model inference in 2026. FP4 is enabled via --dtype nvfp4 on Blackwell hardware, so verify your vLLM version supports it before deployment.

On the same B200 hardware, TensorRT-LLM updates alone dropped inference cost from $0.11 to $0.02 per million tokens over two months in early 2026 by switching to optimised FP4 kernels (NVIDIA Developer, April 2026). Software stack choice moves CPM as much as hardware choice does.

Frequently asked questions

Last reviewed: July 23, 2026. Deploy vLLM on H100, H200, or B200 clusters on packet.ai. Browse available GPU clusters or check current pricing at packet.ai H200 and packet.ai B200.