An LLM API exposes model inference. Here is what that means, how OpenAI-compatible and pricing actually work, and when a managed API beats self-hosting.
An LLM API is a network endpoint that exposes model inference. Most people mean a hosted, managed version when they say it, one where a provider runs the model for you and bills per token, but the term itself just describes the interface, not who's running the infrastructure behind it. This is inference api explained from the ground up: what an llm api actually is, how it differs from self-hosting, what "OpenAI-compatible" means in practice, and when a managed API beats running your own GPU infrastructure.
Key takeaways
This post assumes you already know what inference itself is, the process of running a trained model to generate output; that's covered in the packet.ai What Is LLM Inference guide. What's specific here is the API layer: how you actually access inference without running the infrastructure yourself.
An llm api is a network endpoint you send requests to, a prompt, some parameters, and receive a model's generated output back. Api llm access works the same way regardless of which provider you're calling: a request goes out, inference happens somewhere, a response comes back. What differs is who runs that "somewhere." When people say "llm api" in practice, they usually mean a managed one: the provider has already deployed the model, provisioned the hardware, and built the serving software, and your job is just to call the endpoint without touching any of that infrastructure yourself. This is llm api meaning at its most basic: an interface to inference, whoever happens to be running it.
For example, calling a frontier model through a major provider's API and calling an open-weight model through your own vLLM server are both, technically, LLM API requests. The difference isn't the interface, it's that only one of them requires you to manage GPUs. Training versus inference is a separate distinction entirely: training versus inference covers that split. An llm endpoint always runs inference, never training. The model was already trained by someone, elsewhere, before it ever reached the API.
An openai compatible inference api implements the same request and response structure OpenAI's own API uses, most commonly the /v1/chat/completions endpoint shape: a list of messages in, a generated message out, with the same field names and the same optional parameters (temperature, max tokens, streaming, tool calling) behaving the same way. This matters practically because it's become a de facto standard, not because OpenAI mandates it. Many providers other than OpenAI itself implement this same shape specifically so that existing code, and existing tooling built around that shape, works against their llm rest api with minimal changes.
In practice, this usually means changing a base URL and an llm api key, and little else. Code written against one OpenAI-compatible provider commonly runs against another with those two changes, though model names and some provider-specific parameters still need adjusting. Most providers' llm api documentation includes a working llm api example using this exact request shape, which is often the fastest way to confirm compatibility before writing integration code. This is a meaningfully lower switching cost than a fully custom API shape would require, and it's a large part of why the format has spread beyond OpenAI's own service.
⚡ Compatible doesn't mean identical
Sharing a request format doesn't mean two providers behave identically. Rate limits, exact latency characteristics, which optional parameters are actually supported, and pricing all vary between OpenAI-compatible providers even when the wire format matches. Compatibility gets you a fast integration; it doesn't guarantee identical behavior once you're running against it.
The inference api vs self hosting decision comes down to who's responsible for the infrastructure underneath.
The core tradeoff is control versus operational overhead. Self-hosting, typically on a serving engine like vLLM, gives you full control over which model runs, how it's configured, and exactly what hardware it runs on, at the cost of managing all of that yourself. A managed llm api removes that operational burden entirely, at the cost of being limited to whatever models and configurations the provider offers.
Utilization is the variable that decides this more than any other factor. A GPU you're renting continues costing money whether it's processing requests or sitting idle; a managed inference API only bills for tokens actually processed. This means the math favors different approaches depending on how consistently busy your workload actually is, not on company size or how "serious" the use case is.
Bursty, unpredictable, or low-to-moderate volume traffic tends to favor a managed API, since the alternative is paying for GPU capacity that sits idle during quiet periods. Consistently high-volume traffic running at high GPU utilization around the clock tends to favor self-hosting, since at that point you're not paying for meaningful idle time either way, and self-hosting removes the provider's margin from every token. For the detailed cost math and where that crossover point actually sits, see the packet.ai LLM inference cost guide.
There's also a model-availability dimension separate from cost. There's no single best llm api for every use case; the right one depends on which models a provider's catalog actually includes. If a managed provider's catalog happens to include the specific open-weight model you need, an open-source llm api route can be the faster path to production regardless of the cost comparison. If your model isn't in any provider's catalog, or you need a specific fine-tune, self-hosting becomes the only option, cost aside.
Nearly every llm api pricing model bills per token, with input and output tokens typically priced separately, since generating output requires more computation per token than processing input. This is a fundamentally different billing model from renting a GPU by the hour: a per-token bill scales with actual usage, while a per-hour GPU bill is fixed regardless of how busy that GPU actually is during the rental period.
A free llm api tier is worth understanding for what it actually is: nearly every major provider offers one, typically a fixed credit amount or a rate-capped free quota rather than unlimited free access. This is genuinely useful for evaluating whether a provider's output quality and integration fit your use case before committing to paid usage, but it's not representative of production-scale pricing, and no free tier scales to real traffic volume without eventually converting to paid.
Some providers layer a credit system on top of the underlying per-token rate rather than billing per token directly; understanding whether that abstraction is transparent or opaque matters for predicting your actual spend. The packet.ai Tokens vs Credits guide covers exactly how to evaluate that distinction. Separately, if your prompts share significant repeated content across requests, system prompts, retrieved documents, a provider's Prompt Caching Cost Savings guide covers a discount that can meaningfully change the effective per-token rate you actually pay, and it's worth reading alongside this section specifically, since caching often has the single biggest effect on a real LLM API bill.
If you're weighing a managed API against self-hosting specifically because you don't want to commit to one model or one piece of infrastructure, that's the gap packet.ai's Token Factory is being built to fill: an OpenAI-compatible endpoint, per-token billing, and no GPU operations required on your end. It's currently in private preview, with the specific model catalog and pricing still being finalized.
Join the waitlist for early access once it opens.
Last reviewed: August 31, 2026. For the fundamentals of inference itself, see the What Is LLM Inference guide. For the full cost comparison between self-hosting and managed APIs, see the LLM inference cost guide.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
