RAG Is an Architecture Problem, Not a Vector Database Problem

Retrieval-augmented generation sits at the intersection of search, information management, security, and product design. Reducing that system to an embedding model and a vector store leaves the consequential architecture unanswered.

Decide what counts as an authoritative answer

Start with the questions the system is intended to answer and the sources allowed to support them. A policy repository, a working document, and a chat message do not carry the same authority. Retrieval should preserve those distinctions rather than flatten them into an undifferentiated index.

Define how the system responds when it finds conflicting sources, outdated guidance, or no evidence. An explicit “insufficient information” path is part of the product, not an exception to it.

Keep permissions attached to information

Access checks belong throughout the retrieval path. A user’s ability to ask a question does not grant access to every document that could answer it. Consider entitlement checks during retrieval, caching, response generation, and citation access.

Derived material also needs attention. Summaries, embeddings, evaluation samples, and logs can retain sensitive information after the source has changed or been removed. Assign ownership for updates, revocation, and deletion propagation.

Evaluate the whole response path

A useful evaluation set includes ordinary questions, ambiguous requests, missing evidence, conflicting documents, and unauthorized requests. Test whether the system retrieves the right evidence and whether the generated answer accurately represents it.

Citation presence alone is not proof of grounding. Review whether a cited passage actually supports the associated statement. Track retrieval failures separately from generation failures so the team can make a targeted improvement.

Establish a release boundary

Model, prompt, index, source, and permission changes can all affect behaviour. Record versions and evaluation results together. Agree which changes need human review and what evidence is required before release.

For an initial engagement, a useful output is a system-boundary document paired with a small, representative evaluation set. That provides a better basis for architecture decisions than choosing a vector database first.