API Gateways for AI Workloads

API gateways have traditionally been optimised for request-response patterns with predictable payloads. LLM traffic introduces new challenges that require rethinking some fundamental assumptions.

The Challenge

When you send a request to an LLM API, several things are different from traditional API traffic:

1. Response streaming — Most LLM APIs stream responses token-by-token using Server-Sent Events
2. Variable latency — Response times can range from hundreds of milliseconds to tens of seconds
3. Token-based billing — Costs are based on input and output tokens, not request count
4. Context windows — Request payloads can be extremely large (100K+ tokens)

These characteristics don't break traditional API gateways, but they do expose gaps in how we think about rate limiting, caching, and observability.

Rate Limiting Rethinking

Traditional rate limiting counts requests per time window. With LLM traffic, this doesn't capture actual resource consumption. A simple "hello" prompt and a 50K token document analysis both count as one request, but have vastly different costs and resource implications.

// Traditional rate limiting
const requestsPerMinute = 100;

// Token-aware rate limiting
const tokensPerMinute = 100000;
const estimatedTokens = countTokens(request.body.messages);

if (currentWindowTokens + estimatedTokens > tokensPerMinute) {
  return rateLimitResponse();
}

The solution is token-aware rate limiting that:

• Estimates input tokens before forwarding requests
• Tracks output tokens from streaming responses
• Maintains per-client token budgets
• Provides visibility into token consumption

This is more complex than counting requests, but it's the only way to fairly allocate resources when requests have such variable costs.

Caching Considerations

Exact-match caching rarely helps with LLM traffic because prompts are typically unique. Even similar queries often have slight variations that prevent cache hits.

Semantic caching using embeddings can identify similar prompts and serve cached responses. The approach:

1. Generate an embedding for the incoming prompt
2. Search for similar cached prompts within a threshold
3. If found, return the cached response
4. If not, forward to the LLM and cache the result

const embedding = await generateEmbedding(prompt);
const cached = await searchSimilar(embedding, threshold: 0.95);

if (cached) {
  return cached.response;
}

const response = await forwardToLLM(request);
await cacheWithEmbedding(prompt, embedding, response);
return response;

The complexity comes from choosing the right similarity threshold. Too high and you miss valid cache hits. Too low and you serve incorrect responses.

Streaming and Timeouts

Traditional HTTP timeouts don't work well with streaming LLM responses. A response might take 30 seconds to complete, but it's actively streaming data the entire time.

Gateways need to distinguish between:

• Connection timeout — How long to wait for the first byte
• Inter-chunk timeout — How long to wait between streamed chunks
• Total timeout — Maximum time for the complete response

Most LLM APIs send the first chunk quickly, then stream at a consistent pace. Setting a short inter-chunk timeout (e.g., 5 seconds) catches stalled connections without killing active streams.

Observability Changes

LLM observability needs different metrics than traditional APIs:

The most useful LLM dashboard I've built tracks:

• Token consumption per client over time
• Cost per request and per client
• Time to first token (user-perceived latency)
• Cache hit rate (for semantic caching)
• Model routing decisions

Practical Recommendations

If you're adapting an API gateway for LLM traffic:

1. Implement token-based rate limiting alongside request-based limits
2. Consider semantic caching for high-volume, similar query patterns
3. Set appropriate timeouts for streaming responses (inter-chunk, not just total)
4. Monitor token consumption per client for cost allocation and anomaly detection
5. Plan for large request bodies in your infrastructure (some prompts are megabytes)
6. Add content filtering at the gateway layer for safety and compliance

The API gateway layer is well-positioned to handle these concerns, keeping your application code focused on business logic rather than LLM infrastructure patterns.

What's Next

AI gateways are becoming a distinct product category. Kong, AWS, and others are building purpose-built solutions. The question is whether this becomes a feature of existing API gateways or a separate layer in the stack.

My current thinking: for most enterprises, a unified gateway that handles both traditional and AI traffic will be simpler to operate than separate systems. The patterns are different, but the operational concerns (security, observability, governance) are the same.