Enterprise RAG Architecture: Building AI Answers Your Business Can Rely On
RAG demos succeed in a week and fail in production for the same reason: retrieval quality is a data engineering problem wearing an AI costume.
Retrieval-augmented generation has become the default architecture for enterprise AI, and for good reason: it grounds answers in your own documents, updates as your documents update, and shows its working through citations. It is also the architecture most likely to look finished at 80% and never reach production, because the last 20% is unglamorous data engineering.
This guide covers what a production RAG system needs beyond the demo.
The pipeline, and where each stage breaks
| Stage | Job | Typical failure |
|---|---|---|
| Ingestion | Extract text from source systems | Tables and PDFs mangled; content silently dropped |
| Chunking | Split documents into retrievable units | Fixed-size splits that sever context mid-sentence |
| Enrichment | Attach metadata and permissions | Omitted, making filtering and citation impossible |
| Indexing | Store vectors plus keyword index | Vector-only, so exact identifiers fail |
| Retrieval | Fetch candidates for the query | Top-k with no re-ranking and no permission filter |
| Generation | Answer from supplied context | No instruction to abstain; hallucinated confidence |
| Evaluation | Prove it works | Absent; replaced by opinion |
Chunking is a content decision, not a parameter
The single highest-leverage improvement in most RAG systems is better chunking. Split on document structure — headings, sections, clauses, table rows — rather than on character counts. Retain the heading path inside each chunk so a retrieved passage carries its own context. Keep tables intact, and store a short generated summary alongside long chunks to improve embedding quality. Overlap helps, but structure helps far more.
Retrieve hybrid, then re-rank
Vector similarity alone fails on exact identifiers, part numbers, policy codes and negation. Keyword search alone fails on paraphrase. Run both, fuse the candidate sets, then re-rank the top candidates with a cross-encoder before passing a small, high-quality context window to the model. Sending twenty mediocre chunks is worse than sending four good ones: it dilutes attention, raises cost and increases the chance the model answers from the wrong passage.
Metadata filtering does the heavy lifting
Most enterprise questions are implicitly scoped: this year, this region, this product, this policy version. Without metadata you cannot express that, and the system will happily cite a superseded 2021 policy. Capture source system, document type, effective dates, version, owner, language and sensitivity classification at ingestion — retrofitting is expensive.
Permissions: the reason pilots never launch
A RAG system that can read everything and is used by everyone is a data breach with a chat interface. The rule is simple and non-negotiable: enforce entitlements inside the retrieval query, using the requesting user's identity, before anything reaches the model. Filtering after generation is not a control — the content has already been used. This requires propagating identity through the whole call chain and keeping the index's permission metadata synchronised with the source systems, including revocations.
Guardrails that measurably reduce harm
- Mandatory grounding. Instruct the model to answer only from supplied context and to say it does not know otherwise. Then test that it complies.
- Citations by construction. Every claim links to the chunk that produced it, and the link resolves to a document the user may open.
- Abstention thresholds. If retrieval confidence is low, return sources without a synthesised answer.
- Prompt injection defence. Retrieved content is untrusted input; it must never be able to change instructions or escalate permissions. The OWASP Top 10 for LLM applications is the practical reference here.
- Output review for regulated domains. Human confirmation before anything customer-facing or contractual.
Evaluation, or you are shipping a vibe
Build the harness before you tune anything. Assemble 200 real questions with human-judged correct answers and correct source documents. Then measure two things separately: whether retrieval found the right passages, and whether the answer was faithful, complete and correctly cited. Separating them is essential, because a bad answer from good retrieval and a bad answer from failed retrieval demand entirely different fixes. Run the harness on every configuration change, and track abstention rate as a feature rather than a failure.
Cost and latency reality
Production RAG costs are dominated by context size and re-ranking, not by the base model. Reduce context aggressively, cache embeddings, cache answers for repeated questions, and use a smaller model for classification and query rewriting while reserving the larger model for synthesis. Set per-team budgets and alert on anomalies — an automated caller in a retry loop can spend a quarter's budget overnight.
A staged path to production
Weeks 1–4: one document domain, one user group, read-only. Build ingestion with structure-aware chunking and full metadata. Stand up the evaluation set.
Weeks 5–8: add hybrid retrieval, re-ranking and permission filtering. Measure, tune retrieval, resist the urge to change models.
Weeks 9–12: add guardrails, citation UX, logging and cost controls. Pilot with real users and track abstention and correction rates.
The systems that succeed are not the ones with the best model. They are the ones where somebody did the metadata properly.
Frequently Asked Questions
What is retrieval-augmented generation?
RAG combines a retrieval system with a language model: relevant documents are fetched at query time and supplied as context, so answers are grounded in your own corpus rather than in model memory. The approach was introduced in research by Lewis et al. in 2020.
Why do enterprise RAG pilots fail in production?
Almost always because of retrieval quality and permissions, not the model. Poor chunking, missing metadata, no evaluation set and no access-control filtering produce confident, wrong, and sometimes unauthorised answers.
Should we fine-tune instead of using RAG?
They solve different problems. RAG supplies facts that change; fine-tuning shapes style, format and task behaviour. Most enterprise use cases need RAG first, and fine-tuning rarely fixes a retrieval problem.
How do we stop a RAG system leaking restricted documents?
Filter by the requesting user's entitlements before retrieval, not after generation. Permissions must be enforced in the retrieval query itself, with the user's identity propagated end to end.
How do we evaluate a RAG system?
Measure retrieval and generation separately. Use a judged question set to score recall and precision of retrieved passages, then score answer faithfulness, completeness and citation correctness against known-good answers.