Architecture pattern
Centralised AI gateway
One governed entry point for every model call, so teams can use several LLM providers without each one rebuilding keys, quotas, routing and logging.
- Caller
- Gateway
- Component you run
- Outside your boundary
- Request
- Event or background
How it flows
- Chat assistant to AI gatewaySends a request with its own app credential, never a provider key.
- Agent runtime to AI gatewayUses the same entry point, with its own quota and model allow-list.
- Batch jobs to AI gatewaySends lower-priority traffic under a separate budget.
- AI gateway to Semantic cacheChecks for a stored answer to a similar question first.
- AI gateway to Usage and audit (asynchronous)Records tokens, cost, caller and model for every request.
- AI gateway to Hosted model ARoutes to the primary model by default.
- AI gateway to Hosted model BFails over on errors, timeouts or rate limits.
- AI gateway to Self-hosted modelRoutes sensitive or high-volume work to private inference.
The problem
Once more than one team uses large language models, the same questions come up in every codebase. Where do provider keys live? Who is spending what? What happens when a provider slows down or rate limits us? Can sensitive prompts go to a public endpoint at all?
When each application calls providers directly, every answer is reimplemented slightly differently. Keys get copied into config files, costs only show up on the monthly invoice, and switching provider means changing every client.
When to use it
- Several teams or applications call LLMs, or soon will.
- You use, or want the option to use, more than one model provider, including self-hosted models.
- You need cost attribution per team, application or customer.
- Security or compliance wants a single place to apply prompt checks, redaction and logging.
- You want failover between providers without client changes.
When not to
- A single prototype calling a single model. A direct SDK call is simpler, and you can add a gateway when a second consumer appears.
- Workloads that depend on provider-specific features the gateway cannot pass through cleanly. Check streaming, tool calling and multimodal support before routing them.
- When nobody will own the platform. A shared gateway without an owner becomes a bottleneck instead of a shortcut.
Key components
- AI gateway. Authenticates callers with their own credentials, holds provider keys centrally, and applies per-consumer quotas and token budgets.
- Routing and failover. Chooses a model by policy (cost, latency, data sensitivity) and retries on another provider when one fails or rate limits.
- Semantic cache. Optional reuse of answers to similar questions, scoped carefully by tenant, model and prompt version.
- Usage and audit pipeline. Records tokens, cost, caller and model for every request, feeding dashboards and chargeback.
- Guardrails. Prompt and response checks, PII redaction and content policies applied in one place.
Products such as Kong AI Gateway implement this pattern as plugins on an API gateway; the same shape can be built with other gateways or a dedicated proxy.
Trade-offs
- Another hop. The gateway adds a small amount of latency and must handle long-lived streaming responses well. Measure it against the provider's own response time, which is usually far larger.
- Lowest common denominator risk. A unified API can hide useful provider features. Keep a pass-through option for teams that need them.
- Shared dependency. The gateway becomes critical infrastructure, so it needs high availability, capacity planning and clear ownership.
- Policy versus speed. Central guardrails protect everyone but can slow experimentation. Offer a sandbox route with lighter policy for prototypes.
Explore further
Lab: A demo is the beginning.→Explore what changes when an AI application meets the real world.
Lab: One request, eight plugins.→Follow a single AI request through an ordered chain of gateway plugins, switch any of them off and see what changes.
Lab: Price the workload before the invoice does.→Estimate monthly AI token cost, see what caching, routing, compression and output caps save, and check peak traffic against rate limits.
Lab: Same question. Different words.→See how exact-match and semantic caching decide when an earlier answer can be reused.
Read: API Gateways for AI Workloads→How API gateways are evolving to handle the unique demands of LLM traffic — streaming, token limits, and semantic caching.
Read: Semantic Caching for LLM Applications→Beyond exact-match caching: using embeddings to reduce API costs and latency for LLM-powered applications.