GPU & AI Glossary | packet.ai

Resources GPU & AI Glossary Plain-English definitions for GPU hardware, LLM inference, billing, and AI infrastructure terms. No filler.

Last updated: August 2026 115 terms across 8 categories

No terms match your search. Try a different keyword.

GPU Hardware 45 terms

Bandwidth (Memory Bandwidth): The rate at which data moves between a GPU's VRAM and its compute cores, measured in TB/s. Higher bandwidth means fewer memory bottlenecks during LLM inference. The NVIDIA B200 delivers 8 TB/s - critical for serving large models fast.
Bare Metal: A server you access directly, with no hypervisor or virtualization layer sitting between your workload and the hardware. You get the full, raw GPU - no shared overhead, no noisy neighbors. packet.ai's Bare Metal tier gives you exclusive access to a physical node.
CUDA (Compute Unified Device Architecture): NVIDIA's parallel computing platform and programming model. Almost every major AI/ML framework (PyTorch, TensorFlow, JAX) runs on CUDA. When people say a library is "GPU-accelerated," they usually mean CUDA-accelerated.
CUDA Cores: The basic processing units inside an NVIDIA GPU that handle parallel compute tasks. More CUDA cores generally means more parallelism for matrix operations - the backbone of neural network training and inference.
Distributed Training: Training a neural network across multiple GPUs or nodes simultaneously, combining data parallelism, tensor parallelism, and pipeline parallelism to handle models and datasets too large for a single device. Communication overhead between GPUs is the primary bottleneck - gradient synchronization requires high-bandwidth interconnects (NVLink within a node, InfiniBand across nodes). A well-connected 8xH100 node delivers 900 GB/s all-to-all NVLink bandwidth, making it the baseline unit for serious distributed training runs.
FLOPS (Floating Point Operations Per Second): A measure of raw GPU compute power. Modern AI hardware is measured in TFLOPS (trillions) or PFLOPS (quadrillions). FP16 and BF16 FLOPS are the relevant numbers for LLM workloads. The B200 delivers 4.5 PFLOPS of FP8 compute.
FP8 / FP16 / BF16 / FP32: Numeric precision formats used in GPU compute. Lower precision (FP8) fits more data in VRAM and runs faster - useful for inference. Higher precision (FP32) is more accurate - used in some training scenarios. BF16 is the current sweet spot for training large models.
GPU (Graphics Processing Unit): A parallel processor originally designed for rendering graphics, repurposed for AI because its thousands of smaller cores excel at the matrix math that powers neural networks. Every product on packet.ai runs on NVIDIA data center GPUs.
GPU Orchestration: The automated management of GPU resources across infrastructure - provisioning, scheduling, monitoring, scaling, and lifecycle management without manual intervention. Without orchestration, GPUs sit idle, users queue for manual access, and fragmentation compounds. Modern GPU orchestration platforms go beyond basic scheduling to handle multi-tenancy, billing metering, auto-scaling, and software-defined GPU pooling across large fleets.
GPU Scheduler: The software component that decides which workload runs on which GPU, when, and with what priority. Simple schedulers treat GPUs as discrete units. Advanced schedulers operate at the resource level - understanding VRAM requirements, NVLink topology, and tenant priority to place workloads optimally. packet.ai runs haishare, a custom scheduler built by hosted.ai that uses time-quantum-based sharing, credit systems, and preemption to maximize utilization across a multi-tenant fleet.
GPU Utilization: The percentage of a GPU's compute capacity actively doing useful work at any given moment. Low utilization means you're paying for hardware that isn't working - the core inefficiency GPU orchestration platforms exist to solve. Memory utilization (VRAM usage) is a separate metric: a GPU can be compute-idle but memory-full, blocking new workloads from scheduling. Both numbers matter for fleet efficiency.
GPU Virtualization: Abstracting a physical GPU into one or more logical resources that can be assigned independently to different workloads or tenants. Rather than treating a GPU as all-or-nothing, virtualization introduces a layer that manages access to compute, memory, and bandwidth - allowing multiple workloads to share one physical device. The primary motivation is utilization: expensive GPUs should not sit idle because a single underutilizing workload owns the whole card.
HBM (High Bandwidth Memory): The memory technology used in enterprise-class GPUs like the H100 and B200. Stacked directly on the chip package for extreme bandwidth. HBM3e on the B200 delivers 8 TB/s - compared to ~1 TB/s on consumer GDDR6 cards.
HGX: NVIDIA's server board standard for multi-GPU systems. An HGX B200 node packs 8 B200 GPUs connected via NVLink. When packet.ai says "HGX B200," it means the full 8-GPU board, not a single chip.
MIG (Multi-Instance GPU): NVIDIA's feature on A100 and H100 that partitions a single GPU into up to 7 isolated slices, each with its own VRAM, compute, and memory bandwidth. Useful for running multiple smaller inference workloads on one physical GPU.
NPU (Neural Processing Unit): A low-power processor core designed for on-device AI inference - found in Apple M-series chips, Qualcomm Snapdragon, and Intel Core Ultra. NPUs run quantized smaller models locally without sending data to a cloud API, consuming milliwatts vs. the hundreds of watts a data center GPU draws. They are not replacements for cloud GPUs: NPUs cannot train models, cannot serve large frontier models, and have a fraction of the memory capacity.
NVLink: NVIDIA's high-speed GPU-to-GPU interconnect. Where PCIe peaks at ~64 GB/s bidirectional, NVLink 4.0 on H100s delivers 900 GB/s. Essential for multi-GPU training where tensors need to move rapidly across GPUs.
NVSwitch: A dedicated NVIDIA chip that enables all-to-all NVLink communication between every GPU in an HGX node. Without NVSwitch, GPUs can only communicate with a few neighbors. With it, any GPU can talk to any other at full bandwidth.
PCIe (Peripheral Component Interconnect Express): The interface connecting GPUs to the CPU and the rest of the server. PCIe GPUs share bandwidth with the CPU over this bus. SXM variants use NVLink instead, which is significantly faster for multi-GPU workloads.
SXM: NVIDIA's high-performance GPU form factor used in HGX nodes. SXM GPUs mount directly to a baseboard and connect via NVLink/NVSwitch, bypassing PCIe entirely. SXM variants have higher TDP and higher performance than their PCIe counterparts.
Tensor Cores: Specialized compute units inside NVIDIA GPUs designed for matrix multiply-accumulate operations - the dominant operation in neural network training and inference. Tensor Cores accelerate FP16, BF16, INT8, and FP8 workloads significantly faster than CUDA Cores alone.
TDP (Thermal Design Power): The maximum sustained power a GPU draws under full load, measured in watts. The B200 SXM has a 1,000W TDP; the H100 SXM is 700W. Relevant for bare metal deployments where power affects total cost of ownership.
TPU (Tensor Processing Unit): Google's custom AI accelerator, built specifically for the matrix operations that dominate neural network workloads. TPUs are fast and efficient within the Google Cloud ecosystem but require XLA compilation and have limited compatibility with the CUDA-native tools - vLLM, TGI, Flash Attention - that power most open-source AI workloads. In practice, the vast majority of GPU cloud inference and training runs on NVIDIA GPUs, not TPUs.
VRAM (Video RAM): The on-chip memory of a GPU, separate from system RAM. The primary constraint when running large models - your model weights plus KV cache must fit in VRAM or you hit OOM (out-of-memory) errors. The B200 has 192 GB HBM3e; the H100 has 80 GB.
Elastic GPU Provisioning: Dynamically scaling GPU allocation up or down in response to workload demand without manual intervention. A workload can start with 1 GPU, scale to 8 during training, and release to 0 when idle - paying only for what it uses.
GPU Generation (Ampere / Hopper / Blackwell / Ada): NVIDIA's GPU microarchitectures in successive generations. Ampere (2020): A100, RTX 3090 - introduced MIG and BF16. Hopper (2022): H100, H200 - added FP8, NVLink 4.0. Blackwell (2024): B200, GB200 - 192 GB HBM3e, 4.5 PFLOPS FP8. Ada Lovelace (2022): RTX 4090, RTX 6000 Pro - consumer/workstation line.
GPU Overcommit / Oversubscription: Allocating more virtual GPU capacity than physical hardware supports, betting that not all tenants peak simultaneously. Common when workload patterns are bursty - works at 60-70% average utilization.
GPU Partitioning: Dividing a single physical GPU into isolated slices assigned to separate workloads. Unlike time-slicing, partitioning gives each slice dedicated VRAM and compute running simultaneously. NVIDIA MIG is the hardware implementation on A100 and H100.
GPU Passthrough: Giving a VM direct, exclusive access to a physical GPU by bypassing the hypervisor entirely. The VM sees the GPU as bare metal with no virtualization overhead. Maximum performance; the GPU cannot be shared.
GPU Pooling: Aggregating multiple physical GPUs across servers into a shared resource pool allocated dynamically to workloads. Rather than static per-server assignment, pooling lets the scheduler place GPU capacity where needed in real time - improving utilization and reducing idle hardware.
Hypervisor: Software that creates and manages VMs by abstracting physical hardware. In GPU contexts it enables GPU passthrough (direct VM access) or vGPU (shared access). Bare metal skips it entirely.
Software-Defined GPU Virtualization: An abstraction layer that decouples GPU resources from physical hardware through software, enabling dynamic pooling and multi-tenant sharing without hardware partitioning features like MIG. Pools multiple physical GPUs and provisions slices on demand, optimizing utilization across a fleet.
Spatial Multiplexing: Running multiple workloads on a GPU simultaneously by dividing SM (streaming multiprocessor) capacity spatially. More efficient than time-slicing for workloads that do not saturate the GPU alone - no switching overhead.
Tenant Isolation: Ensuring workloads from different users on shared GPU infrastructure cannot access each other's data or interfere with performance. Operates at multiple levels: memory (separate VRAM), compute (dedicated quotas), and network (separate VLANs).
Time-Slicing (Temporal Multiplexing): A GPU sharing technique where multiple workloads take turns accessing the GPU in rapid rotation, each getting a time quantum. The GPU runs one workload at a time. Simple and broadly compatible, but workloads can interfere during context switches and memory is not isolated between tenants.
vGPU (Virtual GPU): A software-defined slice of a physical GPU presented to a VM or container as a logical GPU device. Allows multiple tenants to share one card with isolation - each gets a portion of VRAM and compute.
ECC Memory: Error-Correcting Code memory - GPU VRAM that can detect and correct single-bit memory errors automatically, preventing silent data corruption during long training runs or large inference jobs. Enterprise data center GPUs (A100, H100, B200) support ECC; consumer cards (RTX 4090, RTX 5090) typically do not. For training runs lasting days or weeks, ECC matters - a single bit flip in a gradient update can corrupt an entire model without ECC catching it.
GPU Autoscaling: Automatically adding or removing GPU instances in response to real-time workload demand, without manual intervention. An autoscaler monitors queue depth, latency, or GPU utilization and provisions new instances when load increases, then terminates them when it drops. Critical for inference APIs and batch pipelines with variable traffic patterns.
GPU Chargeback: Allocating the cost of shared GPU infrastructure back to the teams, projects, or business units that consumed it, based on metered usage data. Common in enterprise and research environments where multiple teams share a GPU cluster. Requires accurate GPU metering as a prerequisite and a cost model ($/GPU-hour or $/token) to translate usage into charges.
GPU Idle Time: The portion of time a provisioned GPU spends with no active workload - compute utilization at or near zero while still consuming power and incurring cost. Idle time is the primary waste metric in GPU infrastructure. Causes include inefficient scheduling, workload queuing gaps, and jobs that hold GPUs between runs. Minimizing idle time is the core economic motivation behind dynamic GPU tiers and intelligent schedulers.
GPU Metering: Tracking GPU resource consumption at a granular level - per user, per job, per team, or per tenant - for billing, chargeback, or capacity planning. Metering captures wall-clock GPU time, VRAM usage, and compute utilization separately. Without metering, GPU costs are opaque and impossible to allocate accurately across teams or customers.
MPS (Multi-Process Service): NVIDIA's CUDA feature that allows multiple processes to share a single GPU simultaneously by funneling their CUDA contexts through a single MPS server process. Unlike time-slicing (which switches between processes), MPS runs them concurrently on the GPU's compute resources. Better utilization for underutilizing workloads, but a crash in one client can affect all others sharing the same GPU.
Resource Fragmentation: The condition where GPU capacity exists across a fleet but cannot be efficiently allocated because available resources don't match workload requirements. Example: a fleet with 4x single free GPUs cannot serve a job requiring 4 GPUs on the same node with NVLink. Fragmentation is a core problem that intelligent schedulers like hosted.ai's haishare are built to solve.
Spot Instance: A GPU instance available at a steep discount (typically 50-80% below on-demand pricing) in exchange for the risk of interruption - the provider can reclaim the instance with little or no notice when demand spikes. Suited for fault-tolerant, checkpointed training jobs that can resume from the last saved state. Not suitable for latency-sensitive inference. packet.ai's Dynamic tier offers on-demand GPU access without interruption risk.
Cloud Bursting: Temporarily expanding capacity from on-premises or reserved GPU infrastructure into on-demand cloud instances when local resources are saturated. A training job that normally runs on owned hardware can burst to packet.ai Dynamic instances during peak demand, then scale back when done. Avoids the capital cost of sizing owned infrastructure for peak load.

Models & AI 8 terms

Closed Model: A model whose weights are proprietary - you access it only through the provider's API, pay per token, and have no ability to self-host or inspect the architecture. GPT-4o, Claude, and Gemini are closed models. The trade-off: managed reliability and frontier capability, against higher per-token cost, no customization of base weights, and full dependence on the provider's pricing and availability.
Foundation Model: A large model trained on broad general-purpose data, designed to be adapted across many downstream tasks rather than built for one specific use. GPT-4, Llama 3, and Claude 3 are all foundation models. The paradigm shift they represent: train once at massive scale, then fine-tune or prompt for specific applications - rather than training a new model per task.
LLM (Large Language Model): A neural network trained on massive text datasets to predict and generate text token by token. Scale is what makes them capable - modern LLMs range from 7B parameters (single GPU, fast) to 405B (multi-node, expensive). The primary infrastructure constraint is VRAM: a 70B model in FP16 needs ~140 GB just to load, before handling a single request.
Model: A trained neural network - a fixed set of numerical weights that transform inputs into outputs. When someone says "run Llama 3.1 70B," they mean: load that specific weight file into GPU VRAM and serve requests against it. Model size (parameter count) directly determines your VRAM requirement, GPU count, and inference cost.
Model Weights: The actual files containing a model's learned parameters, stored on disk and loaded into GPU VRAM at serving time. A Llama 3.1 70B in FP16 is roughly 140 GB across multiple .safetensors files. Every GPU serving process must load the full weights before handling a single request - weight size is the hard floor on VRAM requirements, before accounting for KV cache or batch overhead.
Open Model: A model whose weights are publicly released - anyone can download, self-host, fine-tune, and modify them. The ecosystem is led by Meta Llama, Mistral, Qwen, and DeepSeek. Self-hosting open models on packet.ai GPU instances costs a fraction of closed API pricing at meaningful volume, at the cost of owning your own serving stack and reliability.
Parameters (1B, 7B, 70B, 405B): The numerical weights inside a neural network, learned during training, that encode the model's knowledge and behavior. Parameter count is shorthand for model size - a 70B model has 70 billion individual floating-point numbers. More parameters means higher capability but more VRAM: 2 bytes per parameter in FP16, so a 70B model needs ~140 GB, a 405B model needs a full 8-GPU node.
Tensor Parallelism: A parallelism strategy that splits a model's weight matrices across multiple GPUs, each computing a shard of each layer simultaneously with results aggregated via all-reduce. It is the standard approach for serving models too large for a single GPU - a 70B model at 140 GB requires at least 2x80GB cards. Tensor parallelism demands tight GPU-to-GPU communication at every layer, making NVLink-connected nodes within a single server the optimal environment for it.

LLM & Inference 39 terms

Attention Mechanism: The component of transformer models that lets the model weigh the relevance of different input tokens when generating each output token. Computationally expensive and quadratic in sequence length. Flash Attention is a popular optimization that reduces VRAM usage.
Batching (Dynamic Batching): Grouping multiple inference requests together and processing them in a single GPU forward pass. Dramatically improves GPU utilization and throughput. vLLM, TGI, and TensorRT-LLM all implement dynamic batching.
Cached Input Pricing: A discounted rate applied to input tokens served from an existing KV cache rather than recomputed fresh. When your prompt contains a repeated prefix - a fixed system prompt, a long document - providers can skip reprocessing it and charge 50-90% less. The savings compound at scale: a 2,000-token system prompt sent across 10M requests is 20B tokens, most of which should be hitting cache.
Context Window: The maximum number of tokens a model can consider at once - both input and output combined. Llama 3.1 supports up to 128K tokens. Longer context means more VRAM required for the KV cache.
Decode Phase: The phase of LLM inference where the model generates output tokens one at a time, following the prefill phase. Each step takes the current token plus the KV cache and produces the next token - sequentially, not in parallel. Decode is memory-bandwidth-bound rather than compute-bound: each step reads all model weights from VRAM but does relatively little compute. Larger batch sizes during decode amortize those weight reads across multiple requests, improving throughput without proportionally increasing latency.
Fine-tuning: The process of continuing to train a pre-trained model on a smaller, task-specific dataset to adapt its behavior. Techniques include full fine-tuning, LoRA, QLoRA, and DPO. Requires more GPU memory and longer runtimes than inference.
Flash Attention: An optimized attention algorithm that reorders matrix operations to avoid writing intermediate results to HBM, dramatically reducing memory bandwidth consumption. Nearly all modern inference stacks ship with Flash Attention support.
GQA (Grouped Query Attention): An attention variant used in Llama 3, Mistral, and other modern models that reduces KV cache size by sharing key-value heads across multiple query heads. Significantly improves inference throughput and reduces VRAM pressure.
Inference: Running a trained model on new input to generate output. When you send a prompt to an API and get a response, that's inference. GPU-intensive but shorter than training. Token Factory is packet.ai's inference layer.
Input Token: A token from your request that the model processes before generating any output - your system prompt, user message, conversation history, and any documents you pass. Input tokens are cheaper than output tokens because they're processed in parallel during prefill. At scale, long system prompts sent across millions of requests add up fast - prefix caching exists specifically to reduce this cost.
KV Cache: The stored key-value attention states from previously processed tokens, reused to avoid recomputation on each new token. Grows with sequence length and batch size. On long-context workloads, KV cache can consume more VRAM than the model weights themselves.
LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning technique that inserts small trainable matrices into each layer of a model instead of updating all weights. Trains 10-100x faster, uses far less VRAM, and produces a small adapter file. QLoRA adds quantization on top.
OpenAI-Compatible API: An inference API that follows the same request and response format as OpenAI's /v1/chat/completions endpoint, allowing any app built for OpenAI to switch providers by changing a base URL and API key - no code changes required. The OpenAI format has become the de facto industry standard. packet.ai's Token Factory is fully OpenAI-compatible: change base_url to https://api.packet.ai/v1 and your existing SDK calls run against Llama, Qwen, DeepSeek, or Mistral at a fraction of the cost.
Output Token: A token the model generates in response to your input. Output tokens are produced sequentially, one at a time, making them more compute-intensive and typically 3-5x more expensive per million than input tokens. Controlling max_tokens and profiling your average output length is the fastest way to reduce inference costs.
Prefill: The first phase of LLM inference where the full input prompt is processed to build the initial KV cache. Computationally intensive and scales with prompt length. After prefill, the model enters the decode (token generation) phase.
Prefix Caching: A technique where the KV cache for a shared prefix (e.g., a long system prompt) is computed once and reused across multiple requests. Reduces time-to-first-token and compute cost. Supported by vLLM and TGI.
Quantization: Reducing the numerical precision of model weights to compress model size and speed up inference. Common schemes: INT8, INT4, GPTQ, AWQ. A 70B model in FP16 needs ~140 GB VRAM; in INT4 it fits in ~35 GB. Trade-off: slight quality degradation.
RLHF (Reinforcement Learning from Human Feedback): A training technique used to align LLMs with human preferences by training a reward model on human comparisons, then using RL to optimize the base model against it. Used to produce instruction-following models like ChatGPT and Claude.
Speculative Decoding: An inference acceleration technique where a small draft model generates several candidate tokens, and the larger target model verifies them in parallel. Achieves 2-3x speedups on latency-sensitive workloads with no quality loss.
TGI (Text Generation Inference): Hugging Face's production-grade inference server. Supports continuous batching, tensor parallelism, Flash Attention, and quantization. A common choice for self-hosted LLM serving on GPU cloud.
Throughput (tokens/sec): The number of tokens a model generates per second across all concurrent requests - the primary measure of inference serving capacity. Throughput and latency trade off against each other: larger batch sizes increase throughput but add per-request latency. For batch workloads and high-volume APIs, throughput is the right optimization target. For interactive applications, TTFT and TPOT matter more.
Tokens: The basic units models operate on - roughly 1 token = 0.75 words in English. Models are priced per million input/output tokens. Token Factory charges $0.10/million tokens across supported open-source models.
TPOT (Time Per Output Token): The average time to generate each output token after the first one - measured in milliseconds. TPOT is the inverse of per-stream TPS and determines how fast text streams to the user after the initial response begins. It is primarily constrained by memory bandwidth: each decode step reads all model weights from VRAM, so higher HBM bandwidth directly reduces TPOT. The B200's 8 TB/s HBM3e makes it particularly strong for latency-sensitive inference.
TPS (Tokens Per Second): The rate at which a model generates output tokens - either per individual request stream or in aggregate across all active requests. The two numbers are often confused: a system quoting "3,000 TPS" means total across all users, not per request. Per-stream TPS directly affects user experience - below ~15-20 TPS, streaming responses feel noticeably slow. TPS is set by GPU memory bandwidth, model size, quantization level, and serving framework efficiency.
TTFT (Time to First Token): The latency between sending a request and receiving the first output token. Drives perceived responsiveness in chat and interactive applications. Dominated by prefill time for long prompts.
vLLM: An open-source inference engine with PagedAttention, continuous batching, and tensor parallelism. The most widely deployed open-source LLM serving framework. First-class support for Llama, Mistral, Qwen, DeepSeek, and most open models.
Checkpoint: A saved snapshot of model weights at a specific point during training, written to disk so training can resume if interrupted. Also used to save intermediate states for evaluation or fine-tuning. Checkpoint frequency is a key decision - too infrequent risks losing hours of compute, too frequent creates I/O bottlenecks.
Continuous Batching: An inference serving technique that adds new requests to an in-progress batch as slots free up, rather than waiting for the entire batch to finish. Since requests complete at different lengths, naive batching wastes GPU cycles. Continuous batching keeps GPU utilization high - standard in vLLM, TGI, and TensorRT-LLM.
Data Parallelism: A training strategy where the full model is replicated on multiple GPUs, each processing a different data batch simultaneously. Gradients are averaged across all replicas via AllReduce and weights updated identically. The simplest parallelism approach - scales well when the model fits on a single GPU.
Embedding Model: A neural network that converts text or images into dense numerical vectors capturing semantic meaning. Unlike LLMs, embedding models output fixed-size vectors used for similarity search, retrieval, and clustering. Core workload for RAG pipelines and vector search systems.
Multimodal Model: A model that processes and generates across multiple data types simultaneously - most commonly text and images, but also audio, video, and code. GPT-4o, Gemini, and LLaVA are multimodal models. Require more VRAM than text-only models of similar size due to vision encoder components.
PagedAttention: A memory management technique for LLM inference, introduced by vLLM, that manages KV cache in fixed-size pages rather than pre-allocated contiguous blocks. Borrowed from OS virtual memory, it eliminates fragmentation and enables 2-4x more concurrent requests - the core innovation behind vLLM's throughput gains.
Pipeline Parallelism: A parallelism strategy that splits a model's layers across multiple GPUs sequentially - GPU 1 handles layers 1-10, GPU 2 handles layers 11-20, and so on. Enables models too large for a single node at the cost of pipeline bubbles (idle time waiting for the previous stage).
SLM (Small Language Model): A language model designed to run efficiently with far fewer parameters than frontier LLMs - typically under 7B. Examples: Microsoft Phi-3, Google Gemma, Apple OpenELM. Suited for edge deployment where latency and cost matter more than frontier capability.
TensorRT-LLM: NVIDIA's open-source inference optimization library for deploying LLMs at maximum throughput. Compiles models into optimized CUDA kernels, enabling continuous batching, paged KV cache, and multi-GPU tensor parallelism. Typically the highest-throughput serving framework on NVIDIA hardware, at the cost of more complex setup than vLLM.
AI Agent: A system where an LLM is given tools - web search, code execution, API calls, file access - and autonomously plans and executes multi-step tasks to reach a goal, rather than responding in a single turn. Agents loop: the model reasons, calls a tool, observes the result, and decides the next action. GPU requirements for agents are variable - idle during tool calls, spiking during reasoning steps - making dynamic GPU provisioning a natural fit.
Embedding Generation: The process of running an embedding model on text, images, or other data to produce dense vector representations for use in search, retrieval, or clustering. Computationally lighter than LLM inference but often run at high throughput - billions of documents need embedding before a RAG system can operate. GPU-accelerated embedding generation is orders of magnitude faster than CPU for large-scale pipelines. See Token Factory for packet.ai's inference API.
Function Calling: A feature of LLM APIs where the model outputs structured requests to invoke predefined functions - rather than generating free text - and receives the results back as context. The model decides when and how to call each function based on the user's request. Widely used to connect LLMs to APIs, databases, and external systems. packet.ai's Token Factory supports function calling across compatible open models.
Temperature / Top-P: Two parameters that control the randomness of LLM output sampling. Temperature scales the probability distribution over tokens before sampling - lower values (0.1-0.3) make output more deterministic, higher values (0.8-1.0) introduce more variation. Top-P (nucleus sampling) limits sampling to the smallest set of tokens whose cumulative probability exceeds P. Neither affects GPU compute requirements - they only change which token is selected at the end of each decode step.

Billing & Pricing 9 terms

Bare Metal Tier: Direct access to a physical GPU server - no virtualization overhead, exclusive resources, maximum performance. Suited for large training runs, multi-GPU workloads, or compliance-sensitive deployments. See packet.ai Bare Metal.
Dedicated GPU: A GPU instance reserved exclusively for one tenant. No resource sharing, predictable performance. Critical for production inference workloads where latency consistency matters. See packet.ai Dedicated tier.
Dynamic GPU: An on-demand GPU instance without commitment - spin up, pay by the hour, terminate when done. Best for experimentation, batch jobs, and variable workloads. See packet.ai Dynamic tier.
Egress: Data transferred out of a cloud environment to the public internet or another provider. Many GPU cloud providers charge $0.08-0.12/GB for egress. packet.ai does not charge egress fees.
GPU-as-a-Service (GPUaaS): A cloud delivery model where GPU compute is provisioned on demand via API or web interface, billed by time or usage, without owning hardware. GPUaaS providers differ significantly in what they deliver - bare metal nodes, virtualized GPU instances, or managed inference endpoints. packet.ai is a GPUaaS provider offering on-demand NVIDIA GPU instances across Dedicated, Dynamic, and Bare Metal tiers. GPUaaS.com is the enterprise procurement brand in the same infrastructure family, targeting contract-scale deployments with dedicated SLAs.
On-Demand: A billing model where you pay per hour of GPU use with no upfront commitment. Flexible but typically priced higher than reserved capacity. packet.ai's Dynamic instances are on-demand.
Per-Token Pricing: A billing model where you pay based on tokens processed - input tokens consumed and output tokens generated - rather than by GPU-hour. Quoted as $/million tokens ($/MTok), it aligns cost directly with usage: zero idle waste. At very high volumes, self-hosting on reserved GPU instances typically becomes cheaper than per-token APIs, which carry provider margin. Token Factory charges $0.10/million tokens across supported open models.
Reserved / Committed: A billing model where you commit to a GPU for a fixed period (monthly, annually) in exchange for a lower rate. Suited for stable, predictable workloads. Reduces cost vs. on-demand at the expense of flexibility.
Uptime SLA: A contractual guarantee of minimum service availability, expressed as 9s - 99.9% (8.7 hours downtime/year), 99.99% (52 minutes/year), or 99.999% (5 minutes/year). packet.ai's Dedicated tier carries a 99.99% uptime SLA. The gap between 99.9% and 99.99% is ~8 hours of annual downtime.

Networking 4 terms

InfiniBand: A high-performance network interconnect used to link GPUs and nodes in large training clusters. HDR InfiniBand delivers 200 Gb/s per port; NDR delivers 400 Gb/s. Essential for multi-node distributed training - NCCL collective operations run over InfiniBand.
NCCL (NVIDIA Collective Communications Library): NVIDIA's library for GPU-to-GPU communication during distributed training and inference. Implements collective operations like AllReduce and AllGather. Performance depends heavily on the underlying network (NVLink > InfiniBand > Ethernet).
RDMA (Remote Direct Memory Access): A networking technique that allows one machine to directly access another's memory without involving the OS or CPU. Used with InfiniBand and RoCE to achieve low-latency, high-throughput GPU-to-GPU data transfers in multi-node clusters.
RoCE (RDMA over Converged Ethernet): An alternative to InfiniBand that delivers RDMA capabilities over standard Ethernet infrastructure. Lower cost than InfiniBand but slightly higher latency. Used in some large-scale GPU clusters where Ethernet infrastructure is preferred.

Storage 3 terms

NVMe (Non-Volatile Memory Express): The storage protocol and interface used by modern SSDs. NVMe drives deliver dramatically lower latency and higher throughput than SATA SSDs. Critical for data pipelines where training datasets are read repeatedly from local disk.
Object Storage: A storage model for unstructured data (files, model checkpoints, datasets) accessed via an S3-compatible API. Scales to petabytes, cheap per GB, but higher latency than local NVMe. Common for storing model weights and training datasets.
Persistent Storage: Storage that survives after a compute instance is terminated. Essential for saving model checkpoints, datasets, and outputs between training runs. On packet.ai, persistent volumes can be attached to new instances.

AI Infrastructure 2 terms

Docker / Container: Docker is the dominant tool for packaging applications and dependencies into containers - lightweight, portable, isolated runtime environments. A container bundles application code, libraries, CUDA runtime, and model files without a full OS - and starts in seconds vs. minutes for VMs.
Kubernetes (K8s): An open-source container orchestration platform that automates deploying, scaling, and managing containerized workloads across clusters. For GPU workloads, Kubernetes uses NVIDIA device plugins to schedule GPU resources to containers, enforce quotas, and scale jobs across nodes. Most production LLM inference and training runs on Kubernetes or a managed variant (GKE, EKS, AKS).

packet.ai Products 5 terms

Dedicated Tier: packet.ai's reserved GPU tier - your instance is isolated on dedicated hardware, guaranteed performance, not shared with other tenants. Available for A100, H100, H200, L40S, and other enterprise GPUs. See Dedicated GPU Cloud.
Dynamic Tier: packet.ai's on-demand GPU tier - spin up in minutes, pay per hour, no commitments. Available for B200, H100, H200, RTX 4090, RTX 5090, RTX 6000 Pro, and others. See Dynamic GPU Cloud.
Token Factory: packet.ai's OpenAI-compatible LLM inference API. Per-token billing, scale-to-zero, no infrastructure management. Supports Llama 3, Qwen, DeepSeek, Mistral, Kimi K3, MiniMax M3, and other open models. $0.10/million tokens. See Token Factory.
GPU Cluster: packet.ai's multi-node GPU offering - multiple servers linked with high-speed networking for distributed training and large-scale inference. Available on request. See GPU Clusters.
Pixel Factory: packet.ai's managed image generation API - GPU-backed, pay-per-image, no infrastructure management required. Supports leading image generation models and returns results via a simple REST API. Built on the same GPU infrastructure as Token Factory, allowing teams to run both text and image generation workloads through packet.ai without managing separate GPU clusters. See Pixel Factory.

Ready to put these to use?

Deploy on the GPU that fits your workload. No contracts, no egress fees, no drama.

View Pricing → Talk to us