LLM gateway: what it does, and what it misses
An LLM gateway sits between your application and model providers, unifying traffic across vendors. It's useful infrastructure — but routing requests is not the same as deciding whether a request should run, or proving what happened afterward.
Most teams adopt an LLM gateway to tame multi-provider sprawl: one endpoint, consistent retries, centralized keys. That solves a transport problem. It leaves the governance problem — authorization, budget enforcement, and verifiable evidence — for a different layer.
What an LLM gateway does well
- Provider abstraction and routing across multiple models
- Rate limiting, retries, and transport-level reliability
- Credential handling and request normalization
- Centralized ingress and basic usage logging
What an LLM gateway misses
A gateway can move and shape traffic. It generally can't answer the questions that surface once AI is in production and someone — finance, security, an auditor — starts asking:
- Should this request run at all, under current policy and budget?
- What was the agent authorized to do, before it acted?
- How does the decision become a durable record, not just a log line?
- Can a third party verify that record without trusting the vendor?
Gateway vs control plane
Keel uses the permit — a pre-execution authorization decision — as the canonical seam. The decision exists as its own artifact before execution, not reconstructed from logs later.
| Question | LLM gateway | AI control plane (Keel) |
|---|---|---|
| Primary unit | Gatewayed request path | Permit / decision artifact |
| Main job | Route, proxy, normalize | Authorize, constrain, and prove |
| Cost posture | Usage logging, downstream alerts | Pre-execution deny / throttle / cap |
| Audit posture | Logs plus surrounding stack | Decision-bound, independently verifiable evidence |
You often want both
This isn't an argument against gateways. Many teams run a gateway for provider access and Keel for authorization and proof. The mistake is treating the transport layer as if it already solved the governance layer. The gateway gets your request to the model; the control plane decides whether it should go, and proves what happened.
Frequently asked questions
What is an LLM gateway?
An LLM gateway is middleware between your application and model providers that handles routing, provider abstraction, rate limiting, and request logging. It centralizes traffic but doesn't, by itself, decide whether a request should run or produce verifiable evidence of what happened.
Is an LLM gateway the same as an AI control plane?
No. A gateway handles transport — routing, proxying, normalizing requests. A control plane handles the decision: authorize the action before it runs, enforce policy and budget, and produce a verifiable record. Many teams run both: the gateway for provider access, the control plane for authorization and proof.
Do I need an LLM gateway and Keel?
Often yes. The gateway moves and shapes traffic; Keel decides whether each action is allowed before execution and records tamper-evident, independently verifiable evidence. They sit at different layers and compose.