Token Factory: How We Built a 98% Cheaper OpenAI Alternative

Token Factory is packet.ai's managed LLM inference API: OpenAI-compatible, $0.10/M tokens real-time and $0.05/M batch, with LoRA fine-tuning and function calling. Here is the complete technical specification.

API Compatibility

Token Factory implements the OpenAI Chat Completions API v1 and the OpenAI Batch API. Switch to Token Factory by changing two lines:

# Before
client = OpenAI(api_key="sk-...")

# After
client = OpenAI(
  api_key="pk_...",  # Your packet.ai API key
  base_url="https://api.packet.ai/v1"
)

Everything else - completion calls, streaming, function calling, message history - remains unchanged.

Supported Models

All models billed on actual token count, not rounded to the nearest 1K. Batch API rates are 50% of real-time rates.

Function Calling

Token Factory supports OpenAI-compatible function calling on all models that natively support it (all models except deepseek-r1). The request format is identical to the OpenAI API:

tools = [{
  "type": "function",
  "function": {
    "name": "get_weather",
    "description": "Get current weather for a location",
    "parameters": {
      "type": "object",
      "properties": {
        "location": {"type": "string"},
        "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
      },
      "required": ["location"]
    }
  }
}]

response = client.chat.completions.create(
  model="llama-3.1-70b-instruct",
  messages=[{"role": "user", "content": "What's the weather in Berlin?"}],
  tools=tools,
  tool_choice="auto"
)

Streaming

Streaming is available on all Token Factory models using the standard OpenAI streaming interface:

stream = client.chat.completions.create(
  model="llama-3.1-70b-instruct",
  messages=[{"role": "user", "content": "Explain transformers"}],
  stream=True
)

for chunk in stream:
  if chunk.choices[0].delta.content:
    print(chunk.choices[0].delta.content, end="", flush=True)

Batch API

The Batch API processes requests asynchronously at 50% of real-time pricing. Input format matches the OpenAI Batch API:

# Create a JSONL file with requests
requests = [
  {"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions",
   "body": {"model": "llama-3.1-70b-instruct",
            "messages": [{"role": "user", "content": "Classify sentiment: great product!"}]}},
  # ... more requests
]

# Submit batch
batch = client.batches.create(
  input_file_id=file_id,  # Upload JSONL first
  endpoint="/v1/chat/completions",
  completion_window="24h"
)

# Poll for completion
batch_status = client.batches.retrieve(batch.id)

Batch jobs complete within 24 hours. Typical completion time for batches under 10K requests is 1-4 hours depending on queue depth.

LoRA Fine-Tuning Integration

Token Factory supports serving LoRA adapters on top of base models. The workflow:

1. Train your adapter on a packet.ai GPU instance

# Train with PEFT/LoRA
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"])
model = get_peft_model(base_model, config)
# ... training loop
model.save_pretrained("/workspace/my-adapter")

2. Upload the adapter weights

# Upload via packet.ai API
curl -X POST https://api.packet.ai/v1/adapters \
  -H "Authorization: Bearer $PACKET_API_KEY" \
  -F "file=@/workspace/my-adapter" \
  -F "base_model=llama-3.1-70b-instruct"
# Returns: {"adapter_id": "ada_xyz123"}

3. Reference the adapter in completions

response = client.chat.completions.create(
  model="llama-3.1-70b-instruct/ada_xyz123",  # base_model/adapter_id
  messages=[{"role": "user", "content": "Your prompt here"}]
)

LoRA serving uses the same vLLM infrastructure as base model serving. Adapter weights are cached on the serving node after first use, so subsequent requests with the same adapter_id have no additional loading latency.

Rate Limits and Quotas

Default limits:

• 10 concurrent requests per API key
• 100K tokens per minute (input + output combined)
• 1M tokens per hour
• No daily or monthly cap

Limits are per API key, not per model. Higher limits are available - contact help@packet.ai with your use case. Batch API requests do not count against real-time rate limits.

Latency Benchmarks

Measured on the Token Factory production cluster, median values:

• Llama 3.1 8B - TTFT: 180ms, throughput: 140 tok/s
• Llama 3.1 70B - TTFT: 420ms, throughput: 45 tok/s
• DeepSeek R1 - TTFT: 600ms, throughput: 28 tok/s (reasoning trace adds tokens)
• Qwen 2.5 72B - TTFT: 450ms, throughput: 40 tok/s

TTFT (time to first token) measured from request receipt to first streamed token. Throughput measured at single-request load. Under batch load, throughput increases due to continuous batching efficiency gains.