Why We Built Token Factory

Token Factory is packet.ai's OpenAI-compatible inference API - serving Llama 3, DeepSeek, Qwen, and Mistral on the same GPU infrastructure behind packet.ai's compute cloud, billed per token instead of per hour.

Why We Built It

The standard path to open model inference in 2024 was: provision a GPU, SSH in, install vLLM, download the model, figure out the right quantisation for your VRAM budget, write a serving script, handle batching, manage restarts. For teams that just want to call an API and get tokens back, this is too much friction.

Token Factory removes the infrastructure layer entirely. You get an OpenAI-compatible endpoint, a list of supported models, and per-token pricing. The GPU provisioning, model loading, batching, and scaling happen on our end.

We built it on the same GPU infrastructure behind packet.ai's compute cloud because we already had the hardware utilisation problem solved. Dynamic placement means we can run shared inference efficiently - workloads scheduled across a well-utilised fleet, priced at wholesale rates, with per-token billing that does not charge you for idle time.

The OpenAI Compatibility Layer

Token Factory implements the OpenAI Chat Completions and Embeddings API. To switch from OpenAI to Token Factory:

import openai

client = openai.OpenAI(
    api_key="your-packet-ai-api-key",
    base_url="https://api.packet.ai/v1"
)

response = client.chat.completions.create(
    model="llama-3.1-70b-instruct",
    messages=[{"role": "user", "content": "Hello"}]
)

No other code changes required. Streaming works. Function calling works on supported models. The response format is identical to the OpenAI API response format.

Supported Models

Token Factory currently supports:

• Llama 3.1 8B Instruct - fast, low latency, good for high-volume classification and extraction tasks
• Llama 3.1 70B Instruct - strong reasoning and instruction following, best general-purpose open model in the 70B class
• DeepSeek R1 - chain-of-thought reasoning, strong on math and coding tasks with explicit reasoning traces
• DeepSeek V3 - large context, strong on longer documents and multi-step reasoning
• Qwen 2.5 7B - efficient, multilingual, good on code generation
• Qwen 2.5 72B - strongest open multilingual model, competitive with Llama 70B on most benchmarks
• Mistral 7B Instruct - fast inference, good for latency-sensitive applications where 7B quality is sufficient

Model additions are driven by usage patterns and requests. If you need a specific model not listed, email help@packet.ai.

Pricing Model

Token Factory bills per token - input tokens and output tokens priced separately, consistent with OpenAI's pricing structure. No minimum spend, no monthly commitment, no idle cost between requests.

Per-token rates for Token Factory are set to be meaningfully below OpenAI's equivalent model pricing for comparable quality. The specific rates are on the packet.ai pricing page and are updated as model availability and infrastructure costs change.

For teams doing high-volume inference - millions of tokens per day - raw GPU instances with vLLM remain more cost-efficient than any API. Token Factory is optimised for teams that want per-token economics without managing the infrastructure to get there.

Token Factory vs Raw GPU Instances

The two options are complementary, not competing:

Use Token Factory when:

• You want inference without GPU management overhead
• You have variable or unpredictable traffic (Token Factory scales to zero, raw instances do not)
• You want to prototype with multiple models before committing to a serving stack
• You are currently using the OpenAI API and want to switch to open models without changing your integration

Use raw GPU instances when:

• You have a fine-tuned model that is not on the Token Factory model list
• You need a custom quantisation, context length, or inference configuration
• You are running high-volume, steady-state inference where per-hour GPU pricing is more efficient than per-token pricing
• You need specific hardware (a particular GPU SKU, multi-GPU for large models)

Implementation Details

Token Factory runs on vLLM behind the OpenAI-compatible API layer. Continuous batching handles request queuing. The API is stateless - each request is independent, no conversation history is stored server-side. Clients manage conversation state by passing the full message history in each request, consistent with the OpenAI Chat Completions format.

Context window limits match the deployed model's native context length. Output token limits are configurable via the max_tokens parameter. Rate limits are generous and designed for production use cases - contact support if you need higher limits for a specific workload.

Getting Access

Token Factory is in limited access during the current launch period. Sign up at packet.ai, and Token Factory API access is available through the dashboard once your account is approved. The same API key used for GPU instance provisioning works for Token Factory - no separate credentials.