Not every request needs an answer right now. Here is how Batch API inference works, and why providers often price it at 50% of synchronous API rates.
Not every LLM request needs an answer right now, and batch inference is built for exactly those workloads: nightly reclassification jobs, one-time data migrations, bulk document processing, anything where nobody is staring at a loading spinner waiting for the result. "Batch" is overloaded in LLM infrastructure, so it's worth being precise upfront: this guide covers the asynchronous Batch API pattern that major providers offer at a discount for non-urgent work, not GPU-level batch-size tuning inside a live serving engine, which is a different topic covered elsewhere. This guide covers how batch inference actually works, why major providers price it lower, and how to tell which of your workloads actually belong there.
Key takeaways
Batch inference is the practice of submitting a large set of requests together as a single job, rather than sending each one individually through a real-time endpoint and waiting for its response before moving to the next. Instead of a live request-response cycle, a batch api llm workflow typically works as a file submission: you upload a file containing many individual requests to a batch inference api, the provider processes them asynchronously, and you retrieve a results file once the job completes, rather than polling for each request one at a time. Providers commonly expose this as a separate batch processing api alongside their standard synchronous one.
This is a genuinely different interaction pattern from a standard inference api call, not just a bulk version of the same thing. A synchronous request holds a connection open and blocks until the model finishes generating; a batch job accepts the whole set of requests up front and lets the provider schedule and process them on its own timeline, entirely decoupled from when you submitted them.
Two different ideas hiding behind "batch inference"
Batch API: group asynchronous requests into an offline job at the API level, usually for lower-cost processing. This is what this post covers.
GPU batching: process multiple requests together inside a serving system to improve throughput and hardware utilization, a serving-engine technique covered in the packet.ai continuous batching content. They can be used together, but they are not the same optimization.
Batch APIs from major providers commonly offer a 50% discount versus synchronous pricing. OpenAI, for example, currently documents its Batch API at a 50% cost discount compared to its synchronous API, and Google's Gemini Batch API is currently priced at 50% of its standard interactive API cost. This isn't a marketing discount layered on top of the same underlying compute cost; it reflects a genuine advantage on the provider's side. Synchronous requests need to be served immediately, competing for capacity in real time regardless of when they happen to arrive. Because batch requests do not require immediate responses, providers can instead schedule them asynchronously and optimize the workload for throughput rather than interactive latency, which is the same underlying reason off-peak electricity or off-peak cloud compute tends to be cheaper generally.
Batch processing does not inherently use a lower-quality model. It runs the same model asynchronously, with the tradeoff being delayed results rather than reduced model capability. It's also worth noting that the documented provider discount applies to API token usage specifically; the total cost of a batch workload can still include preprocessing, orchestration, storage, and engineering time around it, so "half the API cost" is not automatically "half the total workload cost."
⚡ The stated turnaround SLA and typical real turnaround are different numbers
Batch APIs commonly quote a target turnaround around 24 hours as their upper bound. OpenAI's own documentation describes batches as completing "within 24 hours (and often more quickly)," and Google's Gemini Batch API documentation states that "in the majority of cases, it is much quicker" than the 24-hour target. Neither provider commits to a specific faster number, and actual speed depends on job size and current system load, but the pattern across both providers' own documentation is consistent: the 24-hour figure is a worst-case bound, not the typical experience. Don't rule out batch inference for a workload just because the stated SLA sounds slow; check actual observed turnaround for your own jobs before assuming it won't fit your timeline.
The determining question for whether a workload belongs on batch inference isn't its size, it's whether anyone or anything is waiting on the response in real time. A surprisingly large share of production LLM traffic is latency-tolerant without teams realizing it: nightly reclassification jobs, bulk document tagging, content moderation sweeps run on a schedule rather than inline, periodic evaluation runs, one-time data migrations, and report generation that runs overnight rather than on demand.
The common mistake is treating every request an application makes as if it were user-facing and latency-sensitive by default, when a meaningful share of it is actually backend work that could tolerate an asynchronous turnaround without anyone noticing the difference. Triaging workloads explicitly into interactive (a user is actively waiting), and batch-eligible (nothing is waiting right now) is a more useful framing than defaulting everything to the synchronous, more expensive path simply because that's the endpoint already wired up.
Batch inference, as covered here, is an API-level pattern for submitting asynchronous, non-real-time workloads. Continuous batching is a serving-engine technique for combining live requests to improve GPU efficiency while still returning each request in real time. Both use the word "batch," but one is about how work is submitted and scheduled, while the other is about how a serving engine processes live traffic. Confusing the two when researching either topic leads to reading the wrong content entirely.
Since batch and real-time inference solve genuinely different problems, a production system may need both: a synchronous, latency-optimized path for interactive traffic, and a batch-eligible path for everything that doesn't need an immediate answer. Treating this as a routing decision at the application level, sending each request down the path that actually matches its urgency, captures savings on the batch-eligible share without touching the latency-sensitive path at all. packet.ai's Token Factory is being built as a single OpenAI-compatible endpoint for managed inference, with batch-style processing for non-urgent workloads planned as part of that offering. It's currently in private preview, with the specific model catalog and pricing still being finalized.
Join the waitlist for early access once it opens.
Last reviewed: September 16, 2026. Batch API discount structures and turnaround times vary by provider and change over time; verify current terms against your specific provider's documentation. For the broader mechanics of API-based inference, see the packet.ai What Is an LLM API guide.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
