I have five AI agents hitting LLM APIs all day. For months, each one connected directly to whatever provider it was configured for. OpenAI here, DeepSeek there, Z.ai Coding Plan over there. Each agent had its own API key, its own base URL, its own error handling. When something broke, I had to check five different places. When I wanted to switch a model, I edited five config files. When I wanted to know how much I was spending, I logged into three different provider dashboards and did mental math.
That system was fragile, opaque, and expensive. The fix was simpler than I expected: put a gateway between your agents and your providers. One process. One config. One place to look.
What a local LLM gateway actually does
An LLM gateway is a proxy that sits between your applications and your LLM providers. Your agents talk to the gateway. The gateway talks to the providers. It speaks the OpenAI-compatible API on both sides, so your agents don’t need to know anything changed.
This is what you get out of the box.
Per-request logs
This is the big one. When you call a provider directly, you get whatever logs the provider decides to give you. Some are generous. OpenAI’s dashboard shows token counts and costs. DeepSeek’s usage page is decent. But many providers, especially subscription-based ones, give you nothing. Z.ai’s Coding Plan, OpenCode Go, various Chinese provider subscriptions. Zero token breakdowns. Zero cache statistics. Zero per-request cost. You pay a flat fee and hope you’re getting your money’s worth.
A gateway fixes this. Every request flows through it. It logs the provider, model, prompt tokens, completion tokens, cached tokens, latency, cost, and the full attempt trail if it fell back between providers. You get the same level of detail for every provider, even the ones that don’t give you a dashboard at all.
Without per-request logs, you can’t optimize. You don’t know your cache hit rate. You don’t know which agent is burning the most tokens. You don’t know which provider is slowest. You’re guessing. With logs, you’re measuring.
Automatic fallback
Providers go down. Rate limits get hit. API keys expire. Without a gateway, each of your agents needs its own retry logic and fallback handling. With a gateway, you define a fallback chain once. Provider A fails, the gateway tries Provider B. Provider B is rate-limited, it tries Provider C. Your agents just see a successful response. They never know anything went wrong.
This is especially useful if you’re mixing direct API access with subscriptions. Your primary provider might be a cheap subscription. When that hits its rate limit, the gateway falls through to a pay-per-token API. You get the cheap rate when you can and the reliable rate when you can’t.
Single configuration point
Add a new provider. Swap a model. Change a fallback order. Rotate an API key. You do it once, in one place. Every agent picks it up immediately. No editing five config files. No redeploying five services. The gateway is the only thing that knows about your API keys and provider URLs.
Cost tracking
The gateway calculates cost for every request based on the provider’s pricing and the actual token usage, including cached tokens at their discounted rate. You don’t estimate. You don’t wait for the bill. You see the cost in real time, per request, per agent, per provider.
Budget governance
Most gateways let you create virtual keys with their own spending caps and rate limits. Give agent A a $50 monthly budget and agent B a $200 budget. If agent A blows through its allocation, it stops getting responses. You find out before the bill arrives, not after.
Cache visibility
Prompt caching is the single biggest cost lever for LLM workloads. Cached tokens are billed at 10x to 20x less than fresh tokens. But you can only optimize cache hit rate if you can see it. A gateway logs cached tokens per request, so you can identify which agents and prompts have poor cache performance and fix them.
Why self-host instead of using a cloud gateway?
There are cloud-hosted LLM gateways. OpenRouter is one. Portkey is another. They work. But self-hosting gives you three things they can’t.
First, data privacy. Your prompts, your conversation history, your system prompts. All of it stays on your infrastructure. Nothing routes through a third party. If you’re working with sensitive data or just don’t want your prompts sitting in someone else’s database, self-hosting is the answer.
Second, no vendor lock-in. A self-hosted gateway is a process you control. If the gateway project shuts down or changes pricing, you keep running the version you have. You’re not dependent on someone else’s uptime or business model.
Third, full customization. Most self-hosted gateways support plugins. You can write custom logic that modifies requests in flight, injects headers, rewrites prompts, or adds fields to the request body. Try doing that with a cloud gateway.
Bifrost: the gateway I use
I went with Bifrost. It’s open source, Apache 2.0 licensed, built in Go, and the official Docker image works out of the box. It supports 23+ providers including OpenAI, Anthropic, AWS Bedrock, Google Vertex, DeepSeek, Z.ai, and OpenRouter. The web UI handles all configuration. The logging plugin runs asynchronously with sub-millisecond overhead. It’s fast enough that you forget it’s there.
The rest of this post is a hands-on guide to getting it running.
Installing Bifrost with Docker
You need Docker and Docker Compose. That’s it. No database to provision, no config files to write, no dependencies to install. The official image is maximhq/bifrost.
Docker Compose setup
Create a directory for Bifrost and add a docker-compose.yml:
services:
bifrost:
image: maximhq/bifrost:latest
container_name: bifrost
ports:
- "8080:8080"
volumes:
- ./data:/app/data
environment:
- APP_PORT=8080
- APP_HOST=0.0.0.0
- LOG_LEVEL=info
restart: unless-stoppedThe volume mount is the one part that matters. Bifrost stores everything in /app/data inside the container: the config database (config.db), request logs (logs.db), and optional vector store for semantic caching. Without the mount, all of it disappears on container restart. With it, your configuration and logs survive redeploys.
The ./data:/app/data pattern uses a bind mount to a local data/ directory. You can see the files, back them up, copy the logs database out for querying. No opaque named volumes.
Start it:
docker compose up -dBifrost launches with zero configuration. No config file needed. Everything is set up through the web UI after startup.
First launch
Open http://localhost:8080 in your browser. The web UI is the primary interface for Bifrost. You can:
- Add provider API keys with a few clicks
- Configure routing rules and fallback chains
- Set up virtual keys with budgets for governance
- View live request logs with token counts, cache stats, and costs
- Monitor per-provider, per-model metrics
Changes apply in real time. No restart needed. Add a provider, and the next request can use it. Remove a key, and requests stop going to that provider immediately.
Adding your first provider
In the web UI, go to the providers section and add a provider. Say OpenAI. Paste your API key. Bifrost validates it and shows you the models available on that key. You’re done. Any request to Bifrost with the model name gpt-4o-mini now routes to OpenAI automatically.
Bifrost resolves providers from the model name. openai/gpt-4o-mini explicitly pins to OpenAI. A bare gpt-4o-mini gets resolved through Bifrost’s model catalog, which maps model names to providers. If multiple providers serve the same model, Bifrost picks one based on your routing rules.
If you prefer declarative setup or want to version-control your configuration, you can bootstrap providers from a config.json file. Bifrost loads it on first startup, seeds the database, then the web UI takes over for runtime changes. Here is a multi-provider setup with OpenAI, DeepSeek, and Z.ai (GLM) all configured at once:
{
"$schema": "https://www.getbifrost.ai/schema",
"providers": {
"openai": {
"keys": [
{
"name": "openai-key-1",
"value": "env.OPENAI_API_KEY",
"models": ["*"],
"weight": 1.0
}
]
},
"deepseek": {
"keys": [
{
"name": "deepseek-key-1",
"value": "env.DEEPSEEK_API_KEY",
"models": ["*"],
"weight": 1.0
}
]
},
"zai": {
"keys": [
{
"name": "zai-coding-plan-1",
"value": "env.ZAI_API_KEY",
"models": ["glm-5.2", "glm-5-turbo"],
"weight": 1.0
}
],
"network_config": {
"base_url": "https://api.z.ai/api/coding/paas/v4"
}
}
},
"config_store": {
"enabled": true,
"type": "sqlite",
"config": {
"path": "./config.db"
}
}
}The env. prefix tells Bifrost to read the key from an environment variable. You can also paste the key value directly, but environment variables are safer for version control. The config_store block enables the web UI. Without it, Bifrost runs in file-only mode and the dashboard is disabled.
Setting up a fallback chain
Add a second provider. Say DeepSeek. Now configure a routing rule: if OpenAI fails or hits a rate limit, fall through to DeepSeek. Bifrost handles this automatically. Your agents never see the failure. They just get a response.
You can chain as many providers as you want. The attempt_trail field in the logs shows you exactly which providers were tried, which ones failed, and why. So when something goes wrong, you can trace the full path the request took.
Real-world routing example: GLM-5.2 with subscription fallback
This is the setup I actually run. GLM-5.2 is my primary model. Z.ai’s Coding Plan is the cheapest source, but it has rate limits. When it hits the limit, I fall through to OpenCode Go accounts (each registered as a separate provider), then to OpenRouter as a final fallback.
Bifrost’s routing uses Virtual Keys with provider configurations. Each provider in the virtual key gets a weight, and Bifrost sorts them by weight to build the fallback chain. Here is the config:
{
"governance": {
"virtual_keys": [
{
"id": "vk-glm-main",
"provider_configs": [
{
"provider": "zai",
"allowed_models": ["glm-5.2"],
"weight": 1.0
},
{
"provider": "opencode-go-1",
"allowed_models": ["glm-5.2"],
"weight": 0.8
},
{
"provider": "opencode-go-2",
"allowed_models": ["glm-5.2"],
"weight": 0.6
},
{
"provider": "opencode-go-3",
"allowed_models": ["glm-5.2"],
"weight": 0.4
},
{
"provider": "openrouter",
"allowed_models": ["glm-5.2"],
"weight": 0.2
}
]
}
]
}
}When a request comes in with the model glm-5.2 and the virtual key vk-glm-main, Bifrost tries Z.ai first. If Z.ai fails or hits a rate limit, it falls through to opencode-go-1. If that hits its rate limit, opencode-go-2 takes over. Then opencode-go-3. Then OpenRouter as the last resort. Each provider gets its own full retry budget before Bifrost moves to the next one.
Your agents just send requests to Bifrost with the x-bf-vk header set to vk-glm-main. They don’t know about Z.ai, OpenCode Go, or OpenRouter. They don’t know about rate limits or fallbacks. They just get a response.
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "x-bf-vk: vk-glm-main" \
-d '{
"model": "glm-5.2",
"messages": [{"role": "user", "content": "Write a Python function to reverse a linked list."}]
}'Configuring retries
By default, Bifrost does not retry failed requests. You need to enable retries per provider in the network config. This is the difference between a single transient error killing your request and Bifrost quietly retrying with exponential backoff:
{
"providers": {
"zai": {
"keys": [
{
"name": "zai-coding-plan-1",
"value": "env.ZAI_API_KEY",
"models": ["glm-5.2"],
"weight": 1.0
}
],
"network_config": {
"max_retries": 3,
"retry_backoff_initial": 500,
"retry_backoff_max": 5000
}
}
}
}With max_retries: 3, Bifrost tries the request up to 4 times (1 initial + 3 retries). Backoff starts at 500ms and caps at 5 seconds with jitter. If the failure is a rate limit (429), Bifrost rotates to a different API key if you have multiple configured. If the failure is an auth error (401/403), it marks the key as permanently dead for that request and moves on immediately with no backoff.
Weighted load balancing
If you have multiple providers serving the same model and want to split traffic between them, assign weights. Bifrost distributes requests proportionally:
{
"governance": {
"virtual_keys": [
{
"id": "vk-balanced",
"provider_configs": [
{
"provider": "openai",
"allowed_models": ["gpt-4o"],
"weight": 0.3
},
{
"provider": "azure",
"allowed_models": ["gpt-4o"],
"weight": 0.7
}
]
}
]
}
}70% of gpt-4o requests go to Azure, 30% to OpenAI. If Azure fails, Bifrost automatically falls through to OpenAI. The weights are normalized, so you can use 7 and 3 or 0.7 and 0.3 with the same result.
Manual fallbacks in the request
If you don’t want to use virtual keys, you can specify fallbacks directly in the request body. This is useful for one-off routing or testing:
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "zai/glm-5.2",
"messages": [{"role": "user", "content": "Hello!"}],
"fallbacks": ["deepseek/deepseek-chat", "openai/gpt-4o-mini"]
}'If Z.ai fails, Bifrost tries DeepSeek. If DeepSeek fails, it tries OpenAI. The attempt_trail in the logs shows which providers were tried and why each one failed.
Stacking subscription accounts
One trick saves serious money if you’re using subscription-based providers. Some providers sell monthly subscriptions with per-account rate limits. OpenCode Go, for instance, charges $10/month per account. Each account gets $60 worth of GLM-5.2 credits but only $15 worth of Kimi K3 credits. Rate limits are $12 per 5 hours and $30 per week per account. One account caps out fast if you’re running multiple agents.
To stack accounts, you register each one as a separate provider in Bifrost (opencode-go-1, opencode-go-2, opencode-go-3) and configure a fallback chain across them. When provider 1 hits its rate limit, Bifrost falls through to provider 2. Provider 2 exhausts, provider 3 takes over. Your agents never see the rate limit. They just see requests succeeding.
Three OpenCode Go accounts cost $30/month total. For GLM-5.2, that’s $180 in effective usage credits. The effective rate limit becomes $36 per 5 hours and $90 per week. That’s 3x the throughput of a single account for 3x the price, which sounds linear until you realize the alternative is hitting a wall and having your agents fail mid-task. The fallback chain turns three small pipes into one big pipe without any application-level changes.
Pointing your agents at Bifrost
Bifrost speaks the OpenAI-compatible API. Your agents don’t need a new SDK or a different client. Just change the base URL:
# Before - direct to provider
base_url = "https://api.openai.com/v1"
# After - through Bifrost
base_url = "http://your-bifrost-host:8080/v1"That’s it. Every request from every agent now flows through Bifrost. You get logs, fallbacks, cost tracking, and a single configuration point. Your agents don’t know the difference.
Testing your first request
curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini",
"messages": [{"role": "user", "content": "Hello, Bifrost!"}]
}'If you have an OpenAI key configured, you’ll get a response. Check the web UI and you’ll see the request in the logs with full token counts and cost.
Querying the logs database
The logs database is SQLite by default, stored at /app/data/logs.db inside the container. To query it directly:
docker cp bifrost:/app/data/logs.db /tmp/logs.db
sqlite3 /tmp/logs.db "SELECT provider, model, prompt_tokens, cached_read_tokens, cost FROM logs ORDER BY timestamp DESC LIMIT 20;"The logs table includes prompt_tokens, completion_tokens, cached_read_tokens, latency, cost, provider, model, status, and the full attempt_trail showing which providers were tried and why they failed. This is the data you need to actually optimize your setup.
For higher volume or analytics workloads, Bifrost also supports PostgreSQL for the log store. But SQLite is fine for most homelab and small-team setups.
Configuration modes
Bifrost has two configuration modes, and you should understand the difference.
Web UI mode is the default. Configuration lives in a SQLite database. You manage everything from the browser. Changes apply immediately. This is what most people want. If a config.json file exists alongside the database, Bifrost uses it to seed the database on first startup, then the database takes over for runtime changes.
File-only mode is for GitOps workflows or advanced setups where you want declarative configuration. Set config_store.enabled to false in your config.json, and Bifrost loads everything from the file at startup. The web UI is disabled. Changes require a restart. This is useful if you want to version-control your configuration or deploy Bifrost as part of an infrastructure-as-code pipeline.
For most people, web UI mode is the right choice. You get the convenience of a dashboard plus the option to bootstrap from a file if you want.
Going further
Once Bifrost is running and all your traffic flows through it, the interesting work begins. You can start optimizing based on what the logs tell you.
If your cached_read_tokens column is low, your prompts probably change between requests. Fix your prompt structure so the prefix stays stable. If one provider has high latency, add a faster fallback. If one agent is burning through tokens, cap its budget with a virtual key. If you’re using OpenRouter, inject a session_id to fix cache fragmentation. If you’re using Chinese providers, learn their peak hours and shift heavy workloads to off-peak windows.
The gateway is the foundation. The optimization is where the actual savings happen. But you can’t optimize what you can’t see, and you can’t see anything without a gateway.