Architecture pattern
Retrieval-augmented generation with governed retrieval
Ground model answers in your own documents without leaking them, by enforcing each user's permissions at retrieval time and sending prompts through a governed AI gateway.
- Caller
- Gateway
- Component you run
- Outside your boundary
- Request
- Event or background
How it flows
- User to RAG applicationAsks a question. The application keeps the user’s identity with the request.
- RAG application to Retrieval APIRetrieves context on behalf of that user, not with a shared superuser account.
- Retrieval API to Vector indexSearches only chunks the user is allowed to read, filtered at query time.
- Ingestion pipeline to Vector index (asynchronous)Indexes documents with their source permissions and sensitivity labels.
- RAG application to AI gatewaySends the prompt and retrieved context through the AI gateway.
- AI gateway to Language modelApplies prompt checks and redaction, then calls the model and logs the exchange.
The problem
Retrieval-augmented generation (RAG) gives a model relevant passages from your own documents before it answers. Early prototypes often index everything with one service account and search it on behalf of anyone. The model then happily summarises a board paper, a salary file or another customer's contract for a user who should never have seen it.
Retrieved text is also untrusted input. A document can contain instructions that try to steer the model, which is a form of indirect prompt injection.
When to use it
- Assistants that answer from internal knowledge: policies, product documentation, support history, contracts.
- Document access varies by role, team, region or customer.
- You need to show where an answer came from and prove who saw what.
- Prompts and retrieved content may contain personal or confidential data.
When not to
- Public, uniformly accessible content. Plain RAG without per-user filtering is fine, though the gateway controls still help.
- Questions that need precise, current figures from a system of record. Call the system's API as a tool instead of searching stale document chunks.
- When the source permissions themselves are unreliable. Fix access control at the source first; RAG inherits its mistakes.
Key components
- Ingestion pipeline. Classifies, chunks and embeds documents, carrying source permissions and sensitivity labels onto every chunk.
- Vector index. Stores embeddings with those labels as filterable metadata.
- Retrieval API. The only way to query the index. It receives the user's identity and applies permission filters at query time.
- AI gateway. Applies prompt checks, redaction and logging, and routes to an approved model.
- Citations and audit. Answers reference their source documents, and each retrieval is logged against the user.
Trade-offs
- Permission sync. Access changes at the source must reach the index quickly, or revoked users keep seeing content. Plan for re-indexing and deletion.
- Recall versus safety. Strict filtering can hide useful context. That is usually the right failure, but users will notice.
- Latency and cost. Retrieval, guardrails and larger prompts all add time and tokens. Caching helps, but only when cache entries are scoped per permission set.
- Injection is reduced, not removed. Filtering and guardrails lower the risk from malicious documents; tools and actions triggered by the model still need their own limits.
Explore further
Lab: Same question. Different words.→See how exact-match and semantic caching decide when an earlier answer can be reused.
Lab: No single guard stops every injection.→Try curated prompt injection attacks against a mock support agent and see which layered defences stop each one.
Lab: Price the workload before the invoice does.→Estimate monthly AI token cost, see what caching, routing, compression and output caps save, and check peak traffic against rate limits.
Read: API Gateways for AI Workloads→How API gateways are evolving to handle the unique demands of LLM traffic — streaming, token limits, and semantic caching.
Read: Semantic Caching for LLM Applications→Beyond exact-match caching: using embeddings to reduce API costs and latency for LLM-powered applications.