Your LLM API returns 429. Do you sleep(1) and retry? Add jitter? Cascade to a backup provider? Here is exactly what production-grade rate limit handling looks like, and when it is simpler to eliminate the limit entirely.
Handle LLM API 429 errors with exponential backoff starting at 500ms, doubling each attempt with +/-50% jitter, capped at 7 retries and 30 seconds, then cascade to a fallback provider. For solo developers and indie builders, the faster fix is removing the rate limit entirely: packet.ai's RTX Pro 6000 at $0.66/hr has no RPM ceiling and no TPM quota.
Key takeaways
Rate limiting is not an edge case. At any meaningful production scale, you hit it. OpenAI, Anthropic, Groq, Together AI. Every provider enforces per-minute and per-day limits on both request count and token volume. The first 429 usually comes as a surprise. The second one should not.
If you're on a Tier 1 API plan right now, this guide is for you. OpenAI Tier 1 caps GPT-4o at 500 RPM and 200,000 TPM. That sounds like headroom until you run a document pipeline or an agent making 15 calls per user action. packet.ai's Dynamic PODs give you a GPU with no RPM ceiling and no TPM quota. Just the tokens per second the hardware physically generates, at $0.66/hr for an RTX Pro 6000 (96 GB GDDR7 Blackwell), 68% below RunPod Secure Cloud's $2.09/hr for the same silicon.
This guide covers the mechanics of exponential backoff with jitter, what a production fallback chain looks like, circuit breakers, and how to read rate limit signals before the 429 hits. For understanding how inference speed affects your architecture decisions, see TTFT vs Tokens Per Second: Which LLM Performance Metric Matters for Your App. For a cost-first comparison of managed inference APIs, see Cheapest LLM API Providers in 2026.
LLM rate limits are not a single counter. Most providers enforce four independent dimensions simultaneously, and your request fails the moment you exceed any one of them.
A single 40,000-token document prompt at Anthropic Tier 1 exhausts the entire per-minute input ceiling in one request, regardless of how many RPM remain. You can be well within your request count and still hit a 429 because one large request consumed the entire token window.
OpenAI's rate limit system operates on a token bucket model per API key. Tokens refill at a fixed rate per second. When the bucket empties, the API returns 429 until enough tokens accumulate. The provider does not always tell you how long to wait.
Most common mistake
Building retry logic that only watches RPM. TPM is the first limit hit on any workload processing documents longer than a few hundred words. Check your actual TPM consumption before assuming you understand your headroom.
Exponential backoff means waiting progressively longer between retry attempts. Each failure doubles the wait before the next attempt. Jitter adds a random offset to that wait. Both parts are required. Dropping either one makes 429 recovery slower in practice.
Without jitter, every client that hit the rate limit at the same moment waits the same duration before retrying. They all fire at once, generating a synchronized burst known as the thundering herd problem, which produces a second wave of 429s larger than the first. Naive retry loops create this reliably.
The backoff formula: delay = min(base_delay x 2^(attempt-1) x jitter_factor, max_delay) where jitter_factor is a random float between 0.5 and 1.5. Cap maximum delay at 30-60 seconds regardless of attempt count. After 5-7 attempts, stop retrying and surface a clean error upstream.
A 429 from OpenAI sometimes includes a Retry-After header with the number of seconds to wait. When present, use it as the floor for your backoff delay. Do not go below it, but add jitter on top. Many providers do not send this header, so your logic must work without it.
Here is a minimal Python implementation you can drop into any project:
import time
import random
def call_with_backoff(fn, max_attempts=7, base_delay=0.5, max_delay=30):
for attempt in range(1, max_attempts + 1):
try:
return fn()
except RateLimitError as e:
if attempt == max_attempts:
raise
retry_after = getattr(e, 'retry_after', None)
base = max(retry_after or 0, base_delay * (2 ** (attempt - 1)))
jitter = random.uniform(0.5, 1.5)
delay = min(base * jitter, max_delay)
time.sleep(delay)
Read headers on every response to throttle proactively before the 429 hits:
def check_headroom(response_headers):
remaining = int(response_headers.get('x-ratelimit-remaining-tokens', 9999))
limit = int(response_headers.get('x-ratelimit-limit-tokens', 9999))
if limit > 0 and remaining / limit < 0.10:
time.sleep(1) # back off before the next call
packet.ai's RTX Pro 6000 at $0.66/hr delivers 96 GB GDDR7 Blackwell with no RPM ceiling and no TPM quota. Just the tokens per second the GPU physically generates. At 68% below RunPod Secure Cloud's $2.09/hr for the same silicon, it is the most cost-efficient single GPU for 30B-70B open-weight inference available to solo developers today.
Not every error from an LLM API should trigger a retry. Retrying on the wrong status codes wastes time and can make things worse.
Retry these
Fail immediately on these
A 400 means your request is malformed. Retrying it returns the same 400 every time. A 429 means the infrastructure was temporarily overwhelmed. The exact same request will succeed once the rate window resets.
Exponential backoff handles transient rate limits. When your RPM or TPM allocation is genuinely exhausted for the window, you need a different approach: routing to a different provider or model tier entirely.
A fallback chain is a prioritised list of inference targets. When the primary target returns a rate limit error that backoff cannot resolve within your latency budget, the router moves to the next target in the list.
Primary: your preferred provider at your preferred model tier
Best quality and cost for your workload. Attempt backoff here first before cascading.
Secondary: alternative provider with equivalent capability
Activated after 2-3 failed retries at the primary. OpenAI-compatible endpoints make this a config change, not a rewrite.
Tertiary: smaller, faster model on a different provider
Lower quality but always available. Good for classification or extraction tasks where output quality is less sensitive.
Graceful degradation: cached response, simplified output, or queue for later
Only when the task can tolerate latency or reduced fidelity. Never silently fail. Surface the degraded state to the user.
OpenAI-compatible APIs make provider fallback practical without rewrites. packet.ai's Token Factory is an OpenAI-compatible inference endpoint serving open-weight models, built to slot into the secondary or tertiary position of any fallback chain. For solo developers, it removes the per-provider rate limit ceiling at open-model cost. For a full comparison of how Token Factory stacks up against Groq and Together AI on price, see packet.ai Token Factory vs Groq vs Together AI: LLM API Pricing Compared. For a deeper look at what Token Factory is built for, see Who Is Token Factory For?
A circuit breaker is a failure-rate monitor. When requests to a provider start failing above a set threshold, the circuit opens and routes all new requests around that provider without attempting a connection. After a cooldown period, a single test request determines whether to restore normal traffic.
Three states: Closed (normal, count failures), Open (provider bypassed, no retry attempts), Half-open (one test request after cooldown; close if it succeeds, reopen if it fails).
The circuit breaker prevents your retry logic from spending its full budget on a provider whose daily token quota is exhausted and will not recover until midnight. Without it, your app fires thousands of retries against a wall. Add one when your workload can exhaust a provider's daily limit.
LLM providers expose rate limit state through response headers on every successful request, not only on 429s. Reading these gives you a real-time view of remaining headroom so you can throttle proactively instead of reacting to errors.
When x-ratelimit-remaining-tokens drops below 10% of your limit, start spacing requests further apart before hitting zero. For batch workloads, stay below 70-80% of your TPM ceiling at all times. Proactive throttling prevents the 429 cycle entirely on predictable workloads.
Rate limits vary by provider, model, and account tier. These figures reflect 2026 entry-level paid tiers. Verify against each provider's current documentation before finalising architecture decisions. These numbers change.
Anthropic Tier 1 enforces four independent counters: RPM, input TPM, output TPM, and a daily dollar cap. Hit any one and you get a 429. This is why document pipelines on Anthropic hit limits faster than their RPM headroom suggests. Input tokens run out first.
OpenAI Tier 5 reaches 10,000 RPM for GPT-4o. Most solo developers and small teams never get there. Getting from Tier 1 to Tier 5 requires cumulative spend, not just time. A pipeline sending 50 requests per minute with 4,000 tokens each burns 200,000 TPM, right at the Tier 1 ceiling for GPT-4o.
Every strategy above manages rate limits from a third-party provider. The most direct solution for a cost-sensitive solo developer: run your own inference stack and remove the per-minute limits from the architecture entirely.
On a dedicated GPU, the only constraint is hardware throughput. Tokens per second the GPU can generate. No RPM ceiling. No TPM quota. No daily request limit. packet.ai's RTX Pro 6000 at $0.66/hr on Dynamic PODs handles 30B-70B models in BF16 precision, 68% below RunPod Secure Cloud's $2.09/hr for the same GPU. For teams whose API bills are growing faster than their revenue, the break-even on switching to self-hosted inference is almost always below 50 million output tokens per month on models in the 30-70B range.
packet.ai's RTX Pro 6000 at $0.66/hr delivers 96 GB GDDR7 Blackwell performance on a Blackwell workstation GPU with 24,064 CUDA cores. It is the most cost-efficient single GPU for 30B-70B open-weight inference in 2026, at 68% below RunPod Secure Cloud's published rate for the same silicon.
The trade-off is operational: you manage the serving stack, handle model updates, and own the latency SLA. packet.ai's Dedicated PODs give single-tenant GPU access with a 99.99% SLA, removing the shared infrastructure variability that causes rate limiting on managed providers. For understanding the cost math in detail, see LLM Inference Cost in 2026: API Pricing and Cost per Million Tokens Compared.
On Tier 1 and hitting 429s?
packet.ai's RTX Pro 6000 at $0.66/hr removes rate limits entirely.
No RPM ceiling. No TPM quota. No 429. 68% below RunPod Secure Cloud for the same GPU. Spin up in under 5 minutes.
Last reviewed: September 7, 2026. GPU pricing verified from packet.ai/pricing. Rate limit figures from OpenAI and Anthropic documentation, September 2026. RunPod RTX Pro 6000 Secure Cloud rate confirmed at $2.09/hr (Community Cloud $1.69/hr). Provider rate limits change frequently. Verify against each provider's current documentation before finalising architecture decisions.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
