Semantic Caching for LLM Applications

LLM API calls are expensive. At scale, even small optimisations in cache hit rate translate to meaningful cost savings. Semantic caching — using embeddings to match similar queries — is one of the more interesting approaches.

Beyond Exact Match

Traditional caching relies on exact key matching. For APIs, this usually means hashing the request body or URL parameters. If the hash matches, serve the cached response.

This works poorly for LLM traffic because:

• Users phrase the same question differently
• Slight variations in prompts produce identical outputs
• System prompts change while user queries stay the same

Consider these queries:

• "What is the capital of France?"
• "Tell me the capital city of France"
• "France's capital is?"

All should return "Paris." But exact-match caching treats them as three separate requests.

How Semantic Caching Works

Semantic caching uses vector embeddings to measure query similarity:

┌─────────────┐     ┌──────────────┐     ┌─────────────┐
│   Query     │────▶│  Embedding   │────▶│   Vector    │
│             │     │   Model      │     │   Search    │
└─────────────┘     └──────────────┘     └─────────────┘
                                               │
                    ┌──────────────────────────┘
                    ▼
            ┌───────────────┐
            │ Similar found │───Yes───▶ Return cached response
            │   (> 0.95)?   │
            └───────────────┘
                    │
                   No
                    │
                    ▼
            ┌───────────────┐
            │  Forward to   │───────────▶ Cache response
            │     LLM       │             with embedding
            └───────────────┘

The implementation:

async function semanticCache(query: string): Promise<string> {
  // Generate embedding for the query
  const embedding = await embed(query);

  // Search for similar cached queries
  const results = await vectorStore.search({
    vector: embedding,
    topK: 1,
    threshold: 0.95
  });

  if (results.length > 0) {
    metrics.cacheHit();
    return results[0].response;
  }

  // Cache miss - call the LLM
  const response = await llm.complete(query);

  // Store with embedding for future matches
  await vectorStore.insert({
    query,
    embedding,
    response,
    createdAt: Date.now()
  });

  metrics.cacheMiss();
  return response;
}

Choosing the Similarity Threshold

The threshold is the critical parameter. Setting it wrong causes problems:

Too high (e.g., 0.99)

• Very few cache hits
• Only near-identical queries match
• Minimal cost savings

Too low (e.g., 0.85)

• False matches
• Semantically different queries return wrong responses
• Users receive incorrect information

Sweet spot (typically 0.93–0.96)

• Catches rephrased versions of the same question
• Avoids matching genuinely different queries
• Requires testing with your specific data

The right threshold depends on your use case. Factual Q&A can use lower thresholds (more variations produce the same answer). Creative or context-dependent queries need higher thresholds.

What Not to Cache

Some queries should bypass the cache entirely:

const BYPASS_PATTERNS = [
  /current|today|now|latest/i,  // Time-sensitive
  /my |I |me /i,                // Personal/contextual
  /random|generate new/i,       // Intentionally varied
];

function shouldCache(query: string): boolean {
  return !BYPASS_PATTERNS.some(pattern => pattern.test(query));
}

Categories to exclude:

• Time-sensitive queries — "What's the weather today?"
• Personal context — "What meetings do I have?"
• Creative generation — "Write me a poem about..."
• Multi-turn conversations — Context changes meaning

Cache Invalidation

LLM caches need invalidation strategies:

TTL-based — Expire entries after a fixed period

const CACHE_TTL = 7 * 24 * 60 * 60 * 1000; // 7 days

Version-based — Invalidate when the model or system prompt changes

const cacheKey = hash(modelVersion + systemPrompt + query);

Size-based — Evict oldest entries when storage limits are reached

For most applications, a 7-day TTL with LRU eviction works well.

Implementation Options

At the application layer:

• Full control over caching logic
• Can use any vector database
• More code to maintain

At the gateway layer:

• Transparent to applications
• Centralised cache across services
• Consistent policy enforcement

Managed services:

• Redis with vector search
• Pinecone, Weaviate, or similar
• Less operational overhead

I prefer gateway-layer implementation. It keeps applications simple and provides a single point for cache management and observability.

Measuring Impact

Track these metrics:

• Cache hit rate — What percentage of queries hit the cache?
• Cost savings — Tokens saved × token cost
• Latency improvement — Cached responses are typically 10–100x faster
• False positive rate — How often do cached responses not match intent?

A well-tuned semantic cache should achieve:

• 15–40% hit rate for typical Q&A workloads
• 50–70% hit rate for documentation/knowledge base queries
• Near-zero false positive rate

Practical Advice

1. Start with exact-match caching — It's simpler and catches the easy wins
2. Add semantic caching for high-volume paths — Not everything needs embeddings
3. Monitor false positives aggressively — One wrong cached answer erodes trust
4. Use a separate embedding model — Don't use your expensive LLM for embeddings
5. Consider hybrid approaches — Exact match first, then semantic

Semantic caching isn't magic. It's a trade-off between cost, latency, and accuracy. But for the right workloads, it's a meaningful optimisation that pays for itself quickly.