Can Gemma 4 12B run on a single GPU?

Yes. It's specifically sized for single-GPU deployment: vLLM's official recipe lists a single GPU with 40GB or more as the minimum for BF16, which fits on cards like the L40S (48GB). With the official QAT quantized checkpoint, it fits on considerably smaller single GPUs as well.

What does encoder-free mean for Gemma 4 12B?

Rather than using separate frozen vision and audio encoders that feed into the language model, Gemma 4 12B projects raw pixel patches and raw audio waveforms directly into the model's hidden dimension through lightweight embedders. This reduces multimodal latency and avoids the fragmented memory footprint of maintaining multiple separate encoder models.

How do I deploy Gemma 4 12B with vLLM?

A single command works for a basic deployment: vllm serve google/gemma-4-12B-it with max-model-len and gpu-memory-utilization set explicitly. For text-only workloads, skip multimodal memory allocation. To serve the quantized version, point the same command at the google/gemma-4-12B-it-qat-w4a16-ct checkpoint; no separate quantization flag is needed since vLLM auto-detects it.

Is Gemma 4 12B better than Gemma 4 26B?

Not outright better, but close on standard benchmarks despite roughly half the parameter count, according to Google's own published comparisons. The 12B variant's main advantages are its encoder-free multimodal architecture and its smaller deployment footprint.

Guide

Gemma 4 12B Deployment: Which Single GPU Actually Fits?

Google says 16GB. vLLM says 40GB. Both are right, for different questions. Here is what it actually takes to run Gemma 4 12B on a GPU in the cloud.

packet.ai Team

August 10, 2026

Gemma 4 12B is Google's newest mid-sized open model, released June 3, 2026, and the first medium Gemma with a fully encoder-free multimodal architecture: text, image, audio, and video all feed directly into the same decoder-only transformer, no separate vision or audio tower. Google's own guidance targets laptops (16GB unified memory), but vLLM's official recipe lists the real cloud-serving minimum as a single 40GB+ GPU in BF16, and an official Google QAT checkpoint drops that to roughly 8GB. This post covers what it actually takes to deploy Gemma 4 12B on a GPU in the cloud, not on a MacBook.

Key takeaways

Google's "16GB VRAM" figure is for on-device, single-user, unified-memory inference. vLLM's own recipe lists 40GB+ for a real single-GPU BF16 server deployment
Gemma 4 12B is encoder-free: raw pixels and raw audio waveforms are projected directly into the language model, no separate vision or audio encoder to load or manage
Google ships an official QAT W4A16 checkpoint that cuts weight memory from 22.8GB to 8.3GB, a 64% reduction, with minimal quality loss
An L40S (48GB) comfortably fits Gemma 4 12B in BF16 with real room for KV cache and concurrency. An RTX 5090 (32GB) fits it too, though it's not yet bookable on packet.ai
vLLM supports Gemma 4 12B's full feature set out of the box: thinking mode, tool calling, structured outputs, and speculative decoding via an official assistant checkpoint

This post is part of the packet.ai Model Deployment Guides cluster. For the underlying VRAM math this post builds on, see the VRAM requirements guide, which covers formulas and GPU sizing for models generally; this post applies that math specifically to a real Gemma 4 deployment on a single GPU.

What Gemma 4 12B Actually Is

Gemma 4 12B is a dense, 12-billion-parameter decoder-only transformer, released as a mid-generation addition to the Gemma 4 family (which launched in April 2026 with E2B, E4B, 26B MoE, and 31B Dense variants). It sits between the small edge models and the 26B/31B high-end models, and Google's own benchmark comparisons put it close to the 26B variant on standard evaluations despite being roughly half the parameter count.

The architecturally notable part is how it handles multimodal input. Most multimodal models bolt separate vision and audio encoders onto a text backbone, which adds latency and a fragmented memory footprint. Gemma 4 12B instead uses a 35M-parameter vision embedder that projects raw 48x48 pixel patches directly into the model's hidden dimension with a single matrix multiplication, and a similarly direct audio wave projection that slices raw 16kHz audio into 40ms frames and feeds them straight into the language model. There's no frozen vision transformer or conformer audio encoder sitting in front of the LLM. Everything, text, image, audio, and video, shares the same weights.

Note

Google's own launch materials frame Gemma 4 12B primarily as a laptop model: LM Studio, Ollama, and native macOS desktop apps via LiteRT-LM. That framing is accurate for on-device, single-user inference, but it's not the number to use when sizing a GPU for actual server deployment with real concurrency.

Gemma 4 12B VRAM Reality Check: 16GB vs. 40GB vs. 8GB

Three different, all-correct numbers float around for Gemma 4 GPU sizing, and they answer three different questions:

Figure	Source	What it actually answers
16GB	Google's developer guide	Single-user, interactive, on-device, unified memory (laptop, Apple Silicon)
40GB+	vLLM's official recipe	Real single-GPU server deployment, BF16, with headroom for batching and KV cache
8.3GB	Google's official QAT W4A16 checkpoint	Weight memory only, 4-bit quantized, before KV cache and framework overhead

The 40GB+ figure is the one to plan a cloud deployment around. It comes directly from vLLM's own supported-models table for Gemma 4, which lists Gemma 4 12B IT at a 1x GPU minimum with 40GB or more in BF16. That number already accounts for the fact that a production server needs headroom beyond raw weight memory: KV cache for concurrent requests, activation memory, and framework overhead, not just enough VRAM to load the checkpoint once.

The 8.3GB figure comes from a different, legitimate source: Google's official Quantization-Aware Training (QAT) checkpoint, the sanctioned way to run Gemma 4 quantized deployment, using 4-bit integer weights with 16-bit activations to cut the 22.8GB BF16 weight footprint by roughly 64%. This is a real, sanctioned way to run Gemma 4 12B on much smaller hardware, but it's a weight-memory number, not a full deployment budget; you still need to add KV cache and overhead on top to get an accurate VRAM budget for a real server.

Gemma 4 vLLM Deployment

vLLM has an official Gemma 4 recipe with a dedicated entry for the 12B variant. Gemma 4 GPU deployment starts with a single vllm serve command:

vllm serve google/gemma-4-12B-it --max-model-len 32768 --gpu-memory-utilization 0.90

max-model-len is worth setting deliberately rather than leaving at the model's full 256K context default; a shorter context window frees VRAM for KV cache and concurrent requests. gpu-memory-utilization between 0.90 and 0.95 gives vLLM's scheduler the most room to work with once the weights are loaded.

For text-only serving where you don't need the multimodal capabilities, skip multimodal memory allocation entirely:

vllm serve google/gemma-4-12B-it --max-model-len 32768 --gpu-memory-utilization 0.90 --limit-mm-per-prompt image=0,audio=0

To serve the official QAT W4A16 checkpoint instead, no quantization flag is required; vLLM auto-detects the quantization config directly from the checkpoint:

vllm serve google/gemma-4-12B-it-qat-w4a16-ct --max-model-len 32768 --gpu-memory-utilization 0.90

Gemma 4 12B also supports thinking mode, tool calling, and structured JSON output through vLLM's standard flags, and an official speculative decoding assistant checkpoint is available if you want to try that on top, with vLLM's documentation recommending 4-8 speculative tokens for this model size.

The takeaway to carry into the hardware decision below: fitting the weights is the minimum requirement, not the deployment requirement. Once KV cache, concurrency, and Gemma 4 12B's multimodal workload enter the picture, the GPU you actually want can be considerably larger than the checkpoint size alone would suggest.

Which Single GPU Actually Fits It

Against the real 40GB+ BF16 figure, here's what fits on packet.ai today:

L40S, 48GB: the straightforward fit. At $0.92/GPU-hr on packet.ai, L40S clears the 40GB+ BF16 requirement with genuine room left over for KV cache and moderate concurrency, without needing quantization at all. This is the least complicated way to run Gemma 4 12B in BF16 on a single GPU today.

RTX 5090, 32GB: fits the quantized checkpoint, not currently bookable. RTX 5090 is listed as coming soon on packet.ai, not yet live. Its 32GB is below the 40GB+ BF16 minimum for the full-precision model, but comfortably covers the 8.3GB QAT checkpoint with room to spare, once it launches.

Smaller cards, quantized only. Below 32GB, the QAT W4A16 checkpoint's 8.3GB weight footprint is what makes single-GPU deployment realistic at all; the full BF16 checkpoint won't fit without sharding across multiple GPUs.

The practical decision: if 48GB is available and BF16 quality matters, L40S is the simple answer. If you specifically need to run on a smaller or cheaper card, the official QAT checkpoint is the sanctioned path, not a community quantization with unclear quality tradeoffs. For teams weighing the best single GPU for a genuinely multimodal deployment without a second encoder model to manage, Gemma 4 12B is a strong current answer.

Running This on packet.ai

Deciding between BF16 on L40S and the QAT checkpoint on smaller hardware is exactly the kind of comparison that's cheap to actually test rather than guess at. packet.ai's Dynamic tier bills hourly, so spinning up an L40S instance to benchmark Gemma 4 12B's real throughput and quality against your own workload costs a fraction of committing to a Dedicated instance first. If your workload specifically needs the smaller QAT footprint on a consumer-class card, the RTX 5090 product page has a waitlist for when that GPU goes live on packet.ai.

Frequently asked questions

It depends on the deployment target. Google's own developer guide lists 16GB for on-device, single-user, unified-memory inference on a laptop. For a real cloud server deployment in BF16 with room for concurrent requests, vLLM's official recipe lists a minimum of 40GB on a single GPU. Google's official QAT W4A16 checkpoint reduces weight memory alone to about 8.3GB, before adding KV cache and overhead.

Last reviewed: August 10, 2026. For the general VRAM formulas this post's numbers build on, see the VRAM requirements guide.