GPU Hardware 17 terms

Bandwidth (Memory Bandwidth)

The rate at which data moves between a GPU's VRAM and its compute cores, measured in TB/s. Higher bandwidth means fewer memory bottlenecks during LLM inference. The NVIDIA B200 delivers 8 TB/s - critical for serving large models fast.

Bare Metal

A server you access directly, with no hypervisor or virtualization layer sitting between your workload and the hardware. You get the full, raw GPU - no shared overhead, no noisy neighbors. packet.ai's Bare Metal tier gives you exclusive access to a physical node.

CUDA (Compute Unified Device Architecture)

NVIDIA's parallel computing platform and programming model. Almost every major AI/ML framework (PyTorch, TensorFlow, JAX) runs on CUDA. When people say a library is "GPU-accelerated," they usually mean CUDA-accelerated.

CUDA Cores

The basic processing units inside an NVIDIA GPU that handle parallel compute tasks. More CUDA cores generally means more parallelism for matrix operations - the backbone of neural network training and inference.

FLOPS (Floating Point Operations Per Second)

A measure of raw GPU compute power. Modern AI hardware is measured in TFLOPS (trillions) or PFLOPS (quadrillions). FP16 and BF16 FLOPS are the relevant numbers for LLM workloads. The B200 delivers 4.5 PFLOPS of FP8 compute.

FP8 / FP16 / BF16 / FP32

Numeric precision formats used in GPU compute. Lower precision (FP8) fits more data in VRAM and runs faster - useful for inference. Higher precision (FP32) is more accurate - used in some training scenarios. BF16 is the current sweet spot for training large models.

GPU (Graphics Processing Unit)

A parallel processor originally designed for rendering graphics, repurposed for AI because its thousands of smaller cores excel at the matrix math that powers neural networks. Every product on packet.ai runs on NVIDIA data center GPUs.

HBM (High Bandwidth Memory)

The memory technology used in enterprise-class GPUs like the H100 and B200. Stacked directly on the chip package for extreme bandwidth. HBM3e on the B200 delivers 8 TB/s - compared to ~1 TB/s on consumer GDDR6 cards.

HGX

NVIDIA's server board standard for multi-GPU systems. An HGX B200 node packs 8 B200 GPUs connected via NVLink. When packet.ai says "HGX B200," it means the full 8-GPU board, not a single chip.

MIG (Multi-Instance GPU)

NVIDIA's feature on A100 and H100 that partitions a single GPU into up to 7 isolated slices, each with its own VRAM, compute, and memory bandwidth. Useful for running multiple smaller inference workloads on one physical GPU.

NVLink

NVIDIA's high-speed GPU-to-GPU interconnect. Where PCIe peaks at ~64 GB/s bidirectional, NVLink 4.0 on H100s delivers 900 GB/s. Essential for multi-GPU training where tensors need to move rapidly across GPUs.

NVSwitch

A dedicated NVIDIA chip that enables all-to-all NVLink communication between every GPU in an HGX node. Without NVSwitch, GPUs can only communicate with a few neighbors. With it, any GPU can talk to any other at full bandwidth.

PCIe (Peripheral Component Interconnect Express)

The interface connecting GPUs to the CPU and the rest of the server. PCIe GPUs share bandwidth with the CPU over this bus. SXM variants use NVLink instead, which is significantly faster for multi-GPU workloads.

SXM

NVIDIA's high-performance GPU form factor used in HGX nodes. SXM GPUs mount directly to a baseboard and connect via NVLink/NVSwitch, bypassing PCIe entirely. SXM variants have higher TDP and higher performance than their PCIe counterparts.

Tensor Cores

Specialized compute units inside NVIDIA GPUs designed for matrix multiply-accumulate operations - the dominant operation in neural network training and inference. Tensor Cores accelerate FP16, BF16, INT8, and FP8 workloads significantly faster than CUDA Cores alone.

TDP (Thermal Design Power)

The maximum sustained power a GPU draws under full load, measured in watts. The B200 SXM has a 1,000W TDP; the H100 SXM is 700W. Relevant for bare metal deployments where power affects total cost of ownership.

VRAM (Video RAM)

The on-chip memory of a GPU, separate from system RAM. The primary constraint when running large models - your model weights plus KV cache must fit in VRAM or you hit OOM (out-of-memory) errors. The B200 has 192 GB HBM3e; the H100 has 80 GB.

LLM & Inference 18 terms

Attention Mechanism

The component of transformer models that lets the model weigh the relevance of different input tokens when generating each output token. Computationally expensive and quadratic in sequence length. Flash Attention is a popular optimization that reduces VRAM usage.

Batching (Dynamic Batching)

Grouping multiple inference requests together and processing them in a single GPU forward pass. Dramatically improves GPU utilization and throughput. vLLM, TGI, and TensorRT-LLM all implement dynamic batching.

Context Window

The maximum number of tokens a model can consider at once - both input and output combined. Llama 3.1 supports up to 128K tokens. Longer context means more VRAM required for the KV cache.

Fine-tuning

The process of continuing to train a pre-trained model on a smaller, task-specific dataset to adapt its behavior. Techniques include full fine-tuning, LoRA, QLoRA, and DPO. Requires more GPU memory and longer runtimes than inference.

Flash Attention

An optimized attention algorithm that reorders matrix operations to avoid writing intermediate results to HBM, dramatically reducing memory bandwidth consumption. Nearly all modern inference stacks ship with Flash Attention support.

GQA (Grouped Query Attention)

An attention variant used in Llama 3, Mistral, and other modern models that reduces KV cache size by sharing key-value heads across multiple query heads. Significantly improves inference throughput and reduces VRAM pressure.

Inference

Running a trained model on new input to generate output. When you send a prompt to an API and get a response, that's inference. GPU-intensive but shorter than training. Token Factory is packet.ai's inference layer.

KV Cache

The stored key-value attention states from previously processed tokens, reused to avoid recomputation on each new token. Grows with sequence length and batch size. On long-context workloads, KV cache can consume more VRAM than the model weights themselves.

LoRA (Low-Rank Adaptation)

A parameter-efficient fine-tuning technique that inserts small trainable matrices into each layer of a model instead of updating all weights. Trains 10-100x faster, uses far less VRAM, and produces a small adapter file. QLoRA adds quantization on top.

Prefill

The first phase of LLM inference where the full input prompt is processed to build the initial KV cache. Computationally intensive and scales with prompt length. After prefill, the model enters the decode (token generation) phase.

Prefix Caching

A technique where the KV cache for a shared prefix (e.g., a long system prompt) is computed once and reused across multiple requests. Reduces time-to-first-token and compute cost. Supported by vLLM and TGI.

Quantization

Reducing the numerical precision of model weights to compress model size and speed up inference. Common schemes: INT8, INT4, GPTQ, AWQ. A 70B model in FP16 needs ~140 GB VRAM; in INT4 it fits in ~35 GB. Trade-off: slight quality degradation.

RLHF (Reinforcement Learning from Human Feedback)

A training technique used to align LLMs with human preferences by training a reward model on human comparisons, then using RL to optimize the base model against it. Used to produce instruction-following models like ChatGPT and Claude.

Speculative Decoding

An inference acceleration technique where a small draft model generates several candidate tokens, and the larger target model verifies them in parallel. Achieves 2-3x speedups on latency-sensitive workloads with no quality loss.

TGI (Text Generation Inference)

Hugging Face's production-grade inference server. Supports continuous batching, tensor parallelism, Flash Attention, and quantization. A common choice for self-hosted LLM serving on GPU cloud.

Tokens

The basic units models operate on - roughly 1 token = 0.75 words in English. Models are priced per million input/output tokens. Token Factory charges $0.10/million tokens across supported open-source models.

TTFT (Time to First Token)

The latency between sending a request and receiving the first output token. Drives perceived responsiveness in chat and interactive applications. Dominated by prefill time for long prompts.

vLLM

An open-source inference engine with PagedAttention, continuous batching, and tensor parallelism. The most widely deployed open-source LLM serving framework. First-class support for Llama, Mistral, Qwen, DeepSeek, and most open models.

Billing & Pricing 6 terms

Bare Metal Tier

Direct access to a physical GPU server - no virtualization overhead, exclusive resources, maximum performance. Suited for large training runs, multi-GPU workloads, or compliance-sensitive deployments. See packet.ai Bare Metal.

Dedicated GPU

A GPU instance reserved exclusively for one tenant. No resource sharing, predictable performance. Critical for production inference workloads where latency consistency matters. See packet.ai Dedicated tier.

Dynamic GPU

An on-demand GPU instance without commitment - spin up, pay by the hour, terminate when done. Best for experimentation, batch jobs, and variable workloads. See packet.ai Dynamic tier.

Egress

Data transferred out of a cloud environment to the public internet or another provider. Many GPU cloud providers charge $0.08-0.12/GB for egress. packet.ai does not charge egress fees.

On-Demand

A billing model where you pay per hour of GPU use with no upfront commitment. Flexible but typically priced higher than reserved capacity. packet.ai's Dynamic instances are on-demand.

Reserved / Committed

A billing model where you commit to a GPU for a fixed period (monthly, annually) in exchange for a lower rate. Suited for stable, predictable workloads. Reduces cost vs. on-demand at the expense of flexibility.

Networking 4 terms

InfiniBand

A high-performance network interconnect used to link GPUs and nodes in large training clusters. HDR InfiniBand delivers 200 Gb/s per port; NDR delivers 400 Gb/s. Essential for multi-node distributed training - NCCL collective operations run over InfiniBand.

NCCL (NVIDIA Collective Communications Library)

NVIDIA's library for GPU-to-GPU communication during distributed training and inference. Implements collective operations like AllReduce and AllGather. Performance depends heavily on the underlying network (NVLink > InfiniBand > Ethernet).

RDMA (Remote Direct Memory Access)

A networking technique that allows one machine to directly access another's memory without involving the OS or CPU. Used with InfiniBand and RoCE to achieve low-latency, high-throughput GPU-to-GPU data transfers in multi-node clusters.

RoCE (RDMA over Converged Ethernet)

An alternative to InfiniBand that delivers RDMA capabilities over standard Ethernet infrastructure. Lower cost than InfiniBand but slightly higher latency. Used in some large-scale GPU clusters where Ethernet infrastructure is preferred.

Storage 3 terms

NVMe (Non-Volatile Memory Express)

The storage protocol and interface used by modern SSDs. NVMe drives deliver dramatically lower latency and higher throughput than SATA SSDs. Critical for data pipelines where training datasets are read repeatedly from local disk.

Object Storage

A storage model for unstructured data (files, model checkpoints, datasets) accessed via an S3-compatible API. Scales to petabytes, cheap per GB, but higher latency than local NVMe. Common for storing model weights and training datasets.

Persistent Storage

Storage that survives after a compute instance is terminated. Essential for saving model checkpoints, datasets, and outputs between training runs. On packet.ai, persistent volumes can be attached to new instances.

packet.ai Products 4 terms

Dedicated Tier

packet.ai's reserved GPU tier - your instance is isolated on dedicated hardware, guaranteed performance, not shared with other tenants. Available for A100, H100, H200, L40S, and other enterprise GPUs. See Dedicated GPU Cloud.

Dynamic Tier

packet.ai's on-demand GPU tier - spin up in minutes, pay per hour, no commitments. Available for B200, H100, H200, RTX 4090, RTX 5090, RTX 6000 Pro, and others. See Dynamic GPU Cloud.

Token Factory

packet.ai's OpenAI-compatible LLM inference API. Per-token billing, scale-to-zero, no infrastructure management. Supports Llama 3, Qwen, DeepSeek, Mistral, Kimi K3, MiniMax M3, and other open models. $0.10/million tokens. See Token Factory.

GPU Cluster

packet.ai's multi-node GPU offering - multiple servers linked with high-speed networking for distributed training and large-scale inference. Available on request. See GPU Clusters.