SSSamarth Sinha

Note

RAG on enterprise data: an architecture checklist

Retrieval-augmented generation works when retrieval is treated as a data problem: permissions, freshness and evaluation first.

Nov 2026 · 1 min read

  • GenAI
  • RAG
  • Vector search
Retrieval-augmented generation with access control and evaluation Documents to Parse and chunk. Parse and chunk to Embeddings. Embeddings to Hybrid index. User question to Retrieve and rerank (question). Hybrid index to Retrieve and rerank. Retrieve and rerank to LLM answer. Structured data to Text-to-SQL tool (metrics). Text-to-SQL tool to LLM answer. LLM answer to Evaluation. question metrics Documents wikis, tickets, PDFs User question with identity Structured data curated tables Parse and chunk keep source and ACLs Embeddings versioned model Hybrid index vector and keyword Retrieve and rerank filtered by access Text-to-SQL tool governed views only LLM answer cites its sources Evaluation groundedness, recall
IllustrativeRetrieval-augmented generation with access control and evaluationScroll →

The idea

Instead of hoping a model remembers your company's facts, retrieve the relevant passages at question time and give them to the model with the question. The answer can only be as good as the retrieval.

Checklist

  1. Permissions travel with the data. Store each chunk's source and access-control list, and filter retrieval by the asking user's identity. An assistant must never answer from a document the user can't open.
  2. Chunk for meaning. Split on structure (headings, sections) rather than fixed sizes alone, and keep titles and metadata with each chunk.
  3. Hybrid retrieval. Combine vector similarity with keyword search: product codes, error messages and names are where pure vector search struggles. Rerank the top results.
  4. Freshness. Re-index on change, not on a calendar, and make deletions remove chunks too.
  5. Numbers go to SQL, not embeddings. "What was revenue last quarter?" belongs to a governed text-to-SQL tool over curated views.
  6. Citations by default, so people can check the answer.
  7. Evaluate before launch and after every change. Keep a question set with expected sources; measure retrieval recall, groundedness and answer quality, and re-run it when the model, prompt or chunking changes.

Trade-offs

  • Bigger context windows reduce the need for precise retrieval but raise cost and latency.
  • Access-filtered retrieval is harder to cache.