8 Large Language Models on a Single NVIDIA Blackwell Server

The NVIDIA RTX PRO 6000 Blackwell Server Edition has 96 GB of GDDR7 VRAM per GPU — enough to run 70B-parameter models on a single card using 4-bit quantisation. We put eight of them on one server and deployed a different open-source LLM on each GPU.

The RTX PRO 6000 Blackwell sits in an interesting position: it’s a professional GPU, not a datacenter GPU, which means it’s available on-demand without waitlists, at a fraction of H100 prices. For inference serving at 30B–70B parameter range, the 96 GB VRAM per card changes what’s possible on a single GPU.

The hardware: RTX PRO 6000 Blackwell Server Edition

The RTX PRO 6000 Blackwell Server Edition is built on the GB202 chip — Blackwell architecture, Compute Capability 12.0 — with 96 GB of GDDR7 per GPU. An 8-GPU server provides 768 GB total VRAM.

The 96 GB VRAM is the key spec. Previous-generation professional GPUs topped out at 48 GB (RTX 6000 Ada). Doubling that unlocks unquantised 70B models on a single GPU — previously a multi-GPU requirement — and makes Mistral Large 123B possible with 4-bit quantisation.

Independent benchmarks from databasemart.com show the PRO 6000 reaching 8,990 tokens/s on Llama-3-8B and 5,160 tokens/s on Qwen-14B under high-concurrency vLLM serving — outperforming H100 at those model sizes while costing significantly less per hour.

The 8 models: throughput benchmarks

One model per GPU, each running vLLM v0.15.1 with --enforce-eager. Results are steady-state throughput under production serving load:

Mistral Large 123B requires 96 GB VRAM even at 4-bit quantisation. It simply cannot run on an H100 80GB, an A100 80GB, or any previous single-GPU option. The RTX PRO 6000’s memory headroom is the only reason it appears in this list.

vLLM on Blackwell: the --enforce-eager flag

All 8 models run on vLLM v0.15.1 with --enforce-eager. CUDA graph compilation currently causes a performance regression on Blackwell GB202 architecture. This is a vLLM/CUDA driver issue being tracked upstream and is expected to be resolved in a future release. Once CUDA graph compilation stabilises on Blackwell, throughput will improve further.

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --enforce-eager \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

Token Factory: all 8 models at $0.10/M tokens

All 8 models are available through the Token Factory API at $0.10/M tokens for both input and output — the same OpenAI-compatible endpoint, same streaming support, same SDK.

To use any of the 8 models via Token Factory, change one line in your existing OpenAI SDK code:

from openai import OpenAI

client = OpenAI(
    base_url="https://dash.packet.ai/api/v1",
    api_key="your-packet-api-key"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Hello"}]
)

Frequently asked questions

Last reviewed: 10 June 2026. Try Token Factory → or browse RTX PRO 6000 clusters on packet.ai →