Most people hit a wall running Ollama locally. Here's when renting a cloud GPU solves it, and why renting often beats buying hardware you'll rarely max out.
Most people start running Ollama on their laptop. Eventually they hit one of four walls: the model no longer fits, responses get painfully slow, teammates need access too, or the machine is tied up doing something else. Renting a GPU solves all four at once, and costs less per hour than most people expect.
Key takeaways
This guide covers the four reasons teams move Ollama to the cloud, how to pick the right GPU, and how to get a model running in under ten minutes on packet.ai.
Ollama works fine locally for small models on machines with a modern GPU. The reasons to move it to a cloud instance are specific and predictable.
Model doesn’t fit locally
A 70B model at FP16 needs 140 GB of VRAM. No consumer GPU has that. A cloud instance does.
Local inference is too slow
A MacBook with Apple Silicon can run 7B models, but token throughput on a dedicated GPU is 10 to 50 times faster depending on batch size.
Teammates need access
A cloud instance has a public IP. You can expose Ollama on port 11434 and share access without anyone needing to install anything locally.
Local machine is busy
Running Ollama on your laptop while trying to use it for anything else degrades both. A cloud instance runs in the background without competing for resources.
The GPU decision for Ollama is almost entirely a VRAM question. Ollama loads model weights into GPU memory and keeps them there. If the model doesn’t fit, Ollama falls back to CPU inference, which is orders of magnitude slower.
The RTX 6000 Pro is worth calling out specifically for Ollama: at 96 GB and $0.66/hr, it is currently the cheapest single GPU on packet.ai that runs Llama 3.3 70B at FP8 without spilling to CPU. The A100 (80 GB, $0.69/hr) also fits 70B at FP8, with slightly less headroom for KV cache. For teams that want to run 70B in production at the lowest possible hourly rate, either of those two GPUs is the practical starting point.
The setup process for Ollama on a cloud GPU is short. packet.ai instances provision with CUDA pre-installed; Ollama itself is a single binary.
Log in to packet.ai, select a GPU (RTX 4090, L40S, A100, or RTX Pro 6000 depending on model size), and launch an on-demand instance. SSH access is available within 5 minutes.
ssh ubuntu@<your-instance-ip>Ollama provides a one-line install script that detects CUDA automatically and sets up the service.
curl -fsSL https://ollama.com/install.sh | shAfter installation, Ollama runs as a system service on port 11434 and auto-detects the GPU.
ollama pull llama3.3:70b
ollama run llama3.3:70bFor the 70B model, use llama3.3:70b-instruct-q8_0 (Q8, ~70 GB) on a 96 GB instance or llama3.3:70b-instruct-q4_K_M (Q4, ~40 GB) on a 48 GB instance.
To make Ollama accessible from outside the instance, set the host environment variable before starting the service:
OLLAMA_HOST=0.0.0.0 ollama serveThe API is then available at http://<your-instance-ip>:11434. You can point any OpenAI-compatible client at it by changing the base URL.
Running Ollama locally has no hourly cost, but it ties up the machine and caps you at whatever VRAM your local GPU has. A cloud GPU is available in any size, billed by the hour, and leaves your local machine free.
For a team running Ollama a few hours a day, the economics of a cloud GPU are straightforward: less than the cost of a coffee per session, no machine tied up, and teammates can connect from anywhere.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
