RAG: how to build an AI assistant on your own documents

RAG (retrieval-augmented generation) lets an AI assistant answer from your documents without fine-tuning a model: the system finds passages in your knowledge base relevant to the question and passes them to the model along with it. The quality of such an assistant depends less on the model than on how documents are split, indexed and retrieved.

Here is what a RAG system consists of, how to build one and how to test it before clients see it.

How does RAG work?

RAG works in two stages: retrieval, then generation. For a user's question, the system searches the knowledge base for the most relevant passages, and the model then writes an answer based only on them, citing its sources.

Step by step:

1. Preparation: documents are split into chunks, and an embedding — a numeric vector describing the meaning of the text — is computed for each.
2. Indexing: chunks and vectors are stored in a database that supports vector search.
3. Retrieval: the user's question is also turned into a vector, and the database finds the closest chunks by meaning.
4. Generation: the retrieved chunks and the question go to the model with an instruction to answer only from them.
5. Answer: the user gets a response with links to the source documents.

The main advantage over fine-tuning is that knowledge updates without retraining: add a document, compute its vectors, and the assistant already knows it.

What components make up a RAG system?

A RAG system is made of a document ingestion pipeline, an embedding model, a vector store, retrieval with re-ranking and a generation layer with output checks. Each component can be swapped without rewriting the rest.

Component	What it does	Typical options
Document ingestion	Extracts text from PDFs, pages, knowledge bases	Document parsers, exports from CRMs and wikis
Chunking	Splits text into meaningful pieces	By headings and paragraphs, with overlap
Embedding model	Turns text into vectors	Cloud APIs or open models
Vector store	Stores vectors and finds nearest ones	pgvector in PostgreSQL, dedicated vector databases
Re-ranking	Refines the order of retrieved chunks	Re-ranker models, hybrid search
Generation	Writes the answer from retrieved chunks	An LLM behind a gateway with limits

If the project already uses PostgreSQL, the pgvector extension is a convenient start: no new infrastructure, and vectors live next to the rest of your data.

Photo: Unsplash

How do you build a simple RAG in Python?

The simplest working RAG in Python takes a few dozen lines: a table of chunks and vectors, a search function and a model call. Below is the skeleton of retrieval and generation on PostgreSQL with pgvector; embed and llm are your wrappers around the chosen providers.

import psycopg

SEARCH_SQL = """
    SELECT content, source_url
    FROM chunks
    ORDER BY embedding <=> %(query)s::vector
    LIMIT 5
"""

def answer(question: str, conn: psycopg.Connection) -> dict:
    query_vector = embed(question)
    rows = conn.execute(SEARCH_SQL, {"query": str(query_vector)}).fetchall()

    context = "\n\n".join(f"[{i + 1}] {content}" for i, (content, _) in enumerate(rows))
    prompt = (
        "Answer the question using only the passages below. "
        "If the answer is not there, say so. Cite passage numbers.\n\n"
        f"{context}\n\nQuestion: {question}"
    )
    return {"answer": llm(prompt), "sources": [url for _, url in rows]}

The <=> operator in pgvector computes cosine distance. For larger volumes, add an HNSW index to the table so search does not scan every row.

How to chunk documents

Chunking matters more than the choice of model. Chunks that are too large blur the meaning; chunks that are too small lose context. A good start is to split by headings and paragraphs, prefix each chunk with its section heading and document title, and overlap neighbouring chunks slightly.

Why does RAG give wrong answers?

RAG most often answers wrongly because retrieval did not find the right passage, not because the model phrased the answer badly. So quality checks start with retrieval.

Typical causes:

• the information exists, but as a table or scan from which no text was extracted;
• chunks are split so the answer is torn across two pieces;
• the user phrases things differently from the document — hybrid search, vector plus full-text, helps;
• outdated document versions in the index contradict the new ones;
• the model was not told to refuse answering from memory when there are no sources.

To measure quality, collect 50–100 real questions with correct answers and sources. Test two things separately: did retrieval find the right passage, and did the model answer correctly from it.

What should you check before launching RAG for customers?

Before launching for customers, settle data access, cost and what the assistant does when it does not know the answer. An internal assistant for staff forgives more than one on your website.

Pre-launch checklist:

1. Access rights: users must not receive passages from documents they cannot access — permission filtering happens at retrieval.
2. Sources in answers: every claim can be checked via a link.
3. Honest refusal: if the knowledge base has no answer, the assistant says so and offers a human contact.
4. Index updates: changed documents are re-indexed automatically.
5. Prompt injection defences: document text cannot change the model's instructions.
6. Logs and user ratings — the basis for improvements.

We covered the general architecture of AI features — gateway, limits, validation — in how to integrate an LLM into your product.

In short

• RAG answers from your documents without fine-tuning: retrieve passages first, then generate from them.
• Quality depends on chunking, indexing and retrieval, not just the model.
• If you have PostgreSQL, start with pgvector — no new infrastructure needed.
• Test retrieval and generation separately on a set of real questions.
• Before launch, set up access control, cited sources and honest refusals.