Articles about LLM gateways all tell the same half of the story: one endpoint, a hundred-plus providers, virtual keys, cost tracking. All of it is true. What you rarely read is the other side — because the day you deploy a gateway, you are not adding a feature, you are adding a critical service to operate.
This is not an argument against. On heterogeneous estates a gateway is often the right answer, and it is what I deploy for clients who aren't on AWS. It is an argument for doing the arithmetic first, because there is a threshold below which the move loses money — and that threshold sits higher than most people assume.
This article uses LiteLLM as the reference, since it is the most widely deployed open-source implementation. The reasoning applies to any gateway.
What a gateway actually gives you
Start with the upside, honestly. A gateway sits between your applications and the model providers, and solves five problems most teams handle badly.
- The provider key stops circulating. It stays on the gateway. Applications, agents and developers get a virtual key that can be revoked individually. Someone leaves the team, you disable one key instead of rotating everything.
- Spend is attributed. Each virtual key carries its own budget, rate limits, allowed-model list and metadata. You finally know which team and which service consume what.
- Caps are enforced before the call, not discovered on next month's invoice. Per-team budgets, requests and tokens per minute.
- Everything is logged in one place, whichever provider serves the request. That is what makes an audit tractable across a mixed estate.
- Switching provider doesn't touch the machines. This is the property native integrations don't have: the change happens in the gateway's configuration, not in your deployments.
On the client side, integration is trivial. LiteLLM notably exposes a /v1/messages endpoint in Anthropic format, so an existing agent can be routed through it without rewriting anything — see the article on autonomous agents in production.
The other side: an inventory of what you now operate
A gateway is not a binary you launch and forget. LiteLLM's production documentation is explicit about this, to the project's credit. Here is what it asks for.
A PostgreSQL database, non-negotiable
Virtual keys, budgets and spend logs live in Postgres — that is, exactly the features you deployed the gateway for. SQLite holds up for a laptop demo and no further: under concurrent requests, budget decrements race and your caps become approximate. You also need to set a salt key before registering a single model, otherwise provider credentials aren't encrypted at rest.
A Redis from the second instance onward
And this is the most expensive trap of all. Without shared Redis, each instance enforces limits on its own: two instances, and a team capped at 100 requests per minute gets 200. You think you have budgets; you have budgets multiplied by your replica count. And high availability requires at least two instances. In other words, the moment the gateway becomes reliable, Redis stops being optional.
One worker per pod, and real memory
The recommendation is a single process per pod with horizontal scaling — several workers in one pod make latency unpredictable and throw off autoscaling. Budget roughly one core and 4 GB of memory per worker. This is not a sidecar: it is a full application service, with its own resources and probes.
Periodic restarts
The documentation recommends setting a maximum number of requests before the process restarts, to absorb memory leaks under sustained load. That is an honest admission from the project, and a real operational constraint: your gateway has to be built to be restarted often, without downtime.
The setting that bites first: the request timeout defaults to a very high value. Left alone, one upstream request that hangs ties up a connection for hours. It is exactly the kind of default that never shows up in staging and saturates the gateway in production.
In the same vein: spend writes happen per request until you batch them, and past roughly a thousand requests per second they become the bottleneck for the whole gateway. There is a transaction buffer to avoid database deadlocks at that load — these are settings to know beforehand, not afterwards.
The calculation nobody does
Here is the heart of it, in one sentence: a gateway's cost is fixed, while the line item it optimizes is variable.
Take the inventory above and turn it into invoice lines. In high availability, a gateway is at minimum:
- two compute instances, roughly one core and 4 GB each;
- a managed, backed-up PostgreSQL database;
- a managed Redis;
- a load balancer, monitoring, alerting;
- and the human time to keep all of it current.
Let's price it for real. Here is a properly redundant gateway on AWS in the Paris region, at on-demand rates pulled from the AWS pricing API on 22 July 2026.
| Component | Configuration | USD / month |
|---|---|---|
| Compute (Fargate) | 2 tasks, 1 vCPU + 4 GB each | 101.91 |
| PostgreSQL (RDS) | db.t4g.small, Multi-AZ | 52.56 |
| Redis (ElastiCache) | cache.t4g.micro | 21.17 |
| Load balancer | ALB + 1 average LCU | 25.45 |
| Total | ≈ 201 | |
Roughly 200 USD a month before the first model call. And that is a floor: the total excludes egress traffic, database storage and backups, monitoring, and above all engineering time, which does not stop at deployment.
A frugal variant — single-AZ database, and Valkey instead of Redis — lands around 151 USD. Worth noting in passing, because it is exactly the kind of trade-off a cost audit surfaces: on the same instance class, ElastiCache bills Valkey at roughly half the price of Redis, for a compatible protocol.
The threshold, expressed in tokens. At 3 USD per million input tokens and 15 USD per million output, a 200 USD gateway is worth about 11 million tokens a month (assuming equal input and output volumes). If your consumption sits well below that, your cost-control mechanism costs more than the line item it controls.
This is exactly the reasoning of a FinOps audit: you don't deploy a control mechanism that costs more than the line item it controls. Redo the calculation with your own rates, region and availability target — the order of magnitude, though, barely moves.
The decision rule
Don't deploy one if
- • you only have one model provider;
- • a single team consumes and nobody charges back;
- • your monthly spend is the same order as the running cost;
- • you have no on-call for one more critical service.
Deploy one if
- • several teams consume and you need chargeback;
- • you span several providers or several clouds;
- • you must prove who called what, uniformly;
- • provider independence is a stated objective.
Notice that three of the four good reasons are not financial. That is normal, and it is the point: a gateway is an organizational tool before it is a cost-saving one. Sold as a way to cut the bill, it nearly always disappoints.
If you're below the threshold
The good news is that the properties you're after often already exist without an extra service, as long as you have a single provider.
On AWS, Amazon Bedrock gives you IAM-role authentication instead of a static key, per-call traceability in CloudTrail, per-tag cost tracking in Cost Explorer, and caps through AWS Budgets. That is four of the five properties from section one, with nothing extra to operate. The fifth — switching provider without touching the machines — is missing, and it is precisely what a gateway buys you.
Other providers have their equivalents. The reasoning is the same: if one of them covers your estate, its native governance is almost always the best result-to-effort ratio.
An intermediate step that is often enough: before deploying a gateway to get cost attribution, try one key (or role) per service and per environment, with a naming convention and tags. It's free, it takes a day, and it answers "who consumes what" in a large share of cases. You'll then know whether the rest justifies a gateway.
If you go ahead: what not to discover in production
A short list, drawn from the settings that aren't the defaults and the mistakes that show up most often.
- Redis before the second instance, not after. Otherwise your limits and budgets are wrong by a factor equal to your replica count, silently.
- Lower the request timeout on the very first deployment. The default is measured in hours.
- Batch the spend writes instead of writing per request, and cap database connections based on instances multiplied by workers.
- Treat the gateway as a single point of failure, because it is one. If it goes down, it isn't one application losing model access, it's all of them. Redundancy, probes, and a written fallback plan.
- Budget for the cost of keeping it current. Clients move fast; a gateway that fails to forward a new capability breaks the corresponding feature on the application side, often without a clear error.
- Encrypt credentials at rest with a salt key set before you register the first model, and back that key up like the critical secret it is.
Frequently asked questions
Does an LLM gateway add latency?
Yes, mechanically: one more network hop, plus the proxy's processing time. In practice it's negligible next to inference itself. The real risk isn't average latency, it's availability: without redundancy, the gateway cuts model access for all your applications at once.
Can you run LiteLLM without a database?
For plain routing, yes. As soon as you want virtual keys, budgets or spend logs — the features that justify the gateway — PostgreSQL becomes necessary. SQLite is fine for a local demo, not for production.
Gateway or Bedrock: do you have to choose?
No, they aren't competitors. Bedrock serves the models, the gateway governs access. The common enterprise setup puts the gateway in front of Bedrock, to unify governance across an estate that also includes other providers. The real question isn't "which one" but "do I need the extra layer".
Self-host or use a managed offering?
If the break-even is tight, a managed offering often pushes it negative — but it removes the most underestimated line: your time. Self-hosting only makes sense if you already have a Kubernetes platform, an on-call rotation and operational habits. Otherwise you're adding a critical service to a team that didn't ask for one.
How do you migrate without breaking things?
Service by service, not all at once. Move one application, check that logging and caps behave as expected, then move the next. On the client side the switch is usually a single base-URL environment variable, so it's instantly reversible — keep that path open for the duration of the migration.
Does it replace observability?
No. A gateway gives you cost, volume and throughput — counting. It doesn't tell you whether the answers are good, whether prompts have drifted, or why an agent is looping. Those are two separate projects, and conflating them is a classic disappointment after rollout.
Read also
LLM gateway: worth it for you, or not?
I run the numbers with you: current spend, how many teams and providers, compliance constraints. Then we either deploy the gateway or use your cloud's native governance — whichever the arithmetic points to.
Let's talk about your projectReply within 24h - First conversation free and with no commitment