GPU Snapshots Demystified: What Actually Survives Pod Termination

GPU pod snapshots on packet.ai save a configuration bookmark - not a disk image. Everything outside the persistent workspace - installed packages, config files, modified system paths - does not survive a snapshot unless it was written to the workspace volume.

What a Snapshot Actually Saves

When you take a snapshot of a packet.ai GPU pod, the system saves:

• Pod configuration: GPU type, disk size, region, environment template selected at launch
• Persistent workspace contents: Everything in the workspace volume at snapshot time - files, conda environments, downloaded models, training checkpoints
• Workspace metadata: Directory structure, file permissions, symlinks within the workspace

What a snapshot does NOT save:

• Packages installed with apt-get to system paths
• Packages installed with pip install to system Python (outside a conda env in the workspace)
• Changes to /etc, /usr, /opt, or other system directories
• Running processes or GPU memory state
• Cron jobs or systemd services added to the system
• SSH keys added to system-level authorized_keys (outside the workspace)

The practical implication: the snapshot preserves your data and workspace-level configuration. It does not preserve system configuration. If your workflow depends on system packages, you need a different persistence strategy.

The Persistent Workspace in Detail

Every packet.ai GPU pod has a persistent workspace volume - a separate storage volume that mounts at a fixed path in the pod filesystem. This volume persists across pod restarts, template updates, and snapshots.

Default workspace path: /workspace

What lives in the workspace by default:

• Your home directory files if home is symlinked to workspace (configuration-dependent)
• Files you explicitly write to /workspace
• Conda environments created in /workspace/miniconda3 or similar paths
• HuggingFace model cache if you set HF_HOME=/workspace/.cache/huggingface
• Dataset files downloaded to workspace paths
• Training checkpoints saved to workspace paths

What does NOT live in the workspace by default:

• System Python packages (/usr/lib/python3, /usr/local/lib/python3)
• Packages installed with sudo pip install
• apt packages
• Changes to /root or /home if those are not in the workspace volume

Making System Changes Persistent

If your workflow requires system packages or system-level configuration, there are two approaches:

Option 1: Setup scripts in the workspace

Write a setup script to /workspace/setup.sh that installs your required packages and configuration:

#!/bin/bash
# /workspace/setup.sh
apt-get update -qq
apt-get install -y libsndfile1 ffmpeg
pip install librosa soundfile --quiet

Run this script on pod start - either manually on each new pod or via the pod startup command configuration in the dashboard. The script is in your workspace (persists across snapshots), and the packages it installs are available for the current pod session.

Option 2: Conda environments in the workspace

Use conda to manage all packages, with conda installed in the workspace:

# Install miniconda to workspace
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p /workspace/miniconda3
export PATH=/workspace/miniconda3/bin:$PATH

# Create environment
conda create -n myenv python=3.11 -y
conda activate myenv
conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia

The conda installation and all environments are in /workspace/miniconda3 - they survive pod restarts and snapshots. Add export PATH=/workspace/miniconda3/bin:$PATH to your .bashrc (also in the workspace) to activate on session start.

Storage Architecture and Performance

packet.ai pod storage has two components:

Container layer: The base OS and pre-installed software (CUDA, PyTorch, JupyterLab). Read-only. Resets to the template state on pod restart. Fast to load because it is a container image.

Workspace volume: NVMe-backed persistent storage. Read-write. Survives pod restarts and snapshots. Size is configurable at pod creation.

NVMe storage specs on packet.ai nodes: sequential read 3-5 GB/s, sequential write 2-4 GB/s, random read IOPS 500k-1M. Loading a 70B FP16 model (140 GB) from NVMe takes 30-60 seconds. Writing training checkpoints is fast enough that checkpoint frequency is a recovery strategy decision, not a performance constraint.

Snapshot Use Cases That Work

• Saving your dataset: Download a large dataset to the workspace, snapshot. Restore the snapshot to get the dataset on a new pod without re-downloading.
• Saving model weights: Download a 70B model to the workspace (sets up the HuggingFace cache), snapshot. New pods from this snapshot start with the model already cached.
• Saving conda environments: Build a complex conda environment in /workspace/miniconda3, snapshot. New pods get the full environment without re-installing.
• Saving training checkpoints: Snapshot mid-training. If the training pod terminates, restore to continue from the checkpoint rather than from scratch.
• Sharing environments: Build an environment, snapshot, share the snapshot ID with a teammate. They launch a new pod from your snapshot.

Snapshot Use Cases That Do Not Work As Expected

• Saving apt packages: apt packages are not in the workspace volume. A snapshot does not preserve them. Use setup scripts in the workspace to reinstall on pod start.
• Saving system Python packages: pip packages installed to system Python are not in the workspace. Use conda in the workspace instead.
• Saving system configuration: Changes to /etc, SSH daemon config, nginx config, and similar system-level changes are not preserved. Use workspace setup scripts.
• Incremental snapshots of large datasets: Snapshots checkpoint the full workspace volume. Snapshotting a 1 TB workspace takes proportionally longer than a 100 GB workspace. For large datasets that do not change, a single snapshot after the initial download is more efficient than repeated snapshots.