A hosted API hides the parser-matching work. Here is what self-hosted tool calling actually requires, and where it can go quietly wrong.
Tool calling on a self-hosted model isn't automatically the same experience as calling a hosted API and getting a clean function call back. The mechanism is the same underneath, constrained generation producing a structured call, but a self-hosted setup makes you configure the piece that a managed API abstracts away: telling the serving engine which parser understands your specific model's tool-call format. This guide covers how llm function calling and agent tool calling actually work on self-hosted infrastructure, what you have to configure that you wouldn't with a hosted API, and where the responsibility for reliable llm tool use genuinely shifts to you.
Key takeaways
Tool calling is fundamentally the same mechanism as structured output generally: the model's response is constrained to match a schema, in this case the schema of a specific tool's parameters, rather than being free-form text. The packet.ai guide to constrained decoding covers this mechanism directly, and the same guarantee and the same limitation both apply here: a function calling json schema call's arguments are guaranteed to be structurally valid against the tool's schema, not guaranteed to be the right tool or the right arguments for the situation. This is the core of tool use llm reliability: structural correctness, not judgment.
This is true whether the model runs behind a hosted API or a self-hosted serving engine. What differs isn't the underlying mechanism of function calling llm systems, it's how much of the surrounding configuration a hosted API handles for you versus what you have to specify yourself when self-hosting.
Different model families can emit tool calls in genuinely different formats. Some produce a JSON object matching a particular structure, others output something closer to a Python function-call syntax, and others use an XML-style wrapper around the call. A hosted API abstracts all of this away, returning one consistent response format regardless of which underlying model handled the request.
Self-hosting removes much of that abstraction layer. With serving engines such as vLLM, you need to explicitly configure tool calling and select a tool-call parser matched to your model's actual output format, since the engine doesn't universally auto-detect this. Getting the parser wrong for your model family means api calling tools either fail to parse or silently return malformed arguments, not a subtle degradation you might miss. Practically, this means checking which parser your specific model family expects (documented parser names exist for common families like Llama, Qwen, Mistral, and several others, so llama function calling and similar family-specific setups each have their own expected parser) before assuming local llm function calling will simply work out of the box.
⚡ A parsable call is not the same as a good call
Serving engine documentation states this distinction plainly: enabling tool calling guarantees a validly-parsable function call, not a high-quality one. Constraining the output to match a schema fixes the format problem entirely; it does nothing to guarantee the model chose the right tool or supplied sensible arguments for the situation. Specifying the expected format directly in the prompt, alongside the schema constraint itself, generally improves how well the model's actual choices align with what the schema forces it to produce.
Beyond the parser that reads a model's output, self-hosting tool calling also requires a chat template that correctly formats the input side: how a tool definition gets presented to the model, and critically, how a previous tool call and its result get represented back to the model in a multi-turn conversation. A chat template that doesn't handle tool-role and assistant-role messages containing prior tool calls correctly will produce a model that either can't see its own prior tool use or misinterprets the conversation structure entirely.
Most serving engines ship example templates for well-known model families specifically because getting this formatting right by hand is genuinely fiddly and easy to get subtly wrong. Using the template that matches your specific model, rather than a generic one, is worth treating as a requirement rather than an optional nicety.
The concept of tool choice, auto (the model decides whether to call a tool), required (the model must call one), and none (tool calling disabled for this request), is consistent across hosted and self-hosted setups conceptually. What varies is implementation: self-hosted support for each mode depends on both which version of the serving engine you're running and whether the specific parser handling your model implements that mode. Required tool calling in particular has been a more recently added capability in some serving engines, so verifying it's supported for your specific engine version and parser combination is worth doing before building a workflow that depends on it.
The parser isn't the only model-specific piece of this puzzle; reasoning-capable models can introduce another layer. A reasoning-capable model can think through a problem before deciding which tool to call and with what arguments, and on a self-hosted stack that reasoning generally needs to be kept separate from the tool call's own structured output rather than colliding with it. In serving stacks that expose reasoning and tool-call parsing separately, this can mean configuring a reasoning parser alongside the tool-call parser, so the model's thinking arrives in its own field distinct from the tool call itself; some specific models need both configured together to work correctly, while others don't require this at all, so the requirement is genuinely model and serving-stack dependent rather than a universal step. Without the right configuration for a model that does need it, reasoning content can end up mixed into the tool call's output, which is exactly the kind of format collision a self-hosted setup has to actively prevent rather than getting handled invisibly the way it typically is behind a hosted API.
Since correctly matching a tool-call parser and chat template to a specific model is exactly the kind of configuration work a managed inference layer is meant to absorb, this is a genuine dividing line between self-hosting and using a tool calling api that already handles it: one requires you to get this configuration right yourself, the other doesn't. This matters as much for a single-call setup as it does for llm agent tool calling, where a whole workflow depends on the parser reliably extracting calls across many turns. packet.ai's Token Factory is being built as an OpenAI-compatible endpoint, meaning open source llm tool calling and closed-model tool calling both work through the same familiar interface, without needing to identify and configure a matching parser. It's currently in private preview, with the specific model catalog and pricing still being finalized.
Join the waitlist for early access once it opens.
Last reviewed: September 21, 2026. Serving engine tool-call parser support and available parser names change as new model families are added; verify current support against your specific engine's documentation. For the underlying constrained-decoding mechanism, see the packet.ai Structured Output guide.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
