RAG Explained

A 5-minute primer for Pascal developers who have not built a retrieval pipeline before.

The problem

Modern LLMs are great at general programming questions and weak at niche frameworks. Ask GPT-4 or Claude how to write a TOrm REST service in mORMot 2 and you get plausible-looking code that mixes Delphi conventions, hallucinated method names, and outdated API patterns from older articles.

The training data is too thin for the model to memorize a 600 k-LOC Pascal framework reliably.

The fix: retrieval at query time

Retrieval Augmented Generation (RAG) sidesteps the memorization problem by giving the model the relevant excerpts of source code at query time, in the prompt itself. The model no longer has to remember the framework — it just has to read what we hand it.

Three steps make it work:

  1. Indexing (offline). Chunk the source code into semantically coherent pieces (one per top-level type, function, or comment block). Compute a numerical fingerprint (an embedding) for each chunk. Store the chunks plus their embeddings in a database that supports nearest-neighbor search.
  2. Retrieval (per-query). Embed the question with the same embedding model used at indexing time. Ask the database for the K chunks whose embeddings are closest to the question's embedding.
  3. Generation (per-query). Hand those K chunks to a generative LLM together with the original question, plus an instruction like “use only the provided context to answer.”

Why embeddings work

An embedding is a fixed-length vector of floats (e.g. 768 numbers) produced by a neural network. The network is trained so that semantically related texts produce vectors that are close together in geometric space.

Two consequences matter:

  • “How do I serialize a TOrm to JSON?” and “TOrm.GetJSONValues serializes ORM rows to JSON output” land near each other in vector space even though they share few literal words.
  • “How do I serialize a TOrm to JSON?” and “FastReport printer configuration on Linux” land far apart, even if both happen to contain the word “serialize” by accident.

Cosine similarity (the angle between the two vectors) ranks the K closest chunks; pgvector's <=> operator computes that for us at SQL-query time.

Why hybrid retrieval

Vector similarity captures meaning. It does not always capture exact identifiers. A query like TSqlDBConnection thread safety benefits from the literal token match in addition to the semantic match, because TSqlDBConnection is a unique symbol.

Postgres ships tsvector (a full-text index) for token matching. mormot2-pgvector blends the two scores per chunk:

combined = α · vector_similarity + (1 − α) · text_rank

Defaults to α = 0.7 (vector-dominant). The query CLI exposes --alpha so you can tune live.

Why pgvector specifically

You could pick a dedicated vector database (Qdrant, Weaviate, Pinecone, Milvus). They all work fine. Three reasons we chose pgvector:

  1. One database. mORMot 2 already has a polished PostgreSQL driver. Adding a second store doubles the operational surface for no functional gain.
  2. Pascal-native pipeline. The codec ports the pgvector-go MIT wire format directly. Bind/scan goes through BindBlob / ColumnUtf8 on the existing TSqlDBPostgresStatement — no extra HTTP hop, no gRPC client.
  3. Hybrid retrieval comes free. The tsvector half of the score formula already has a GIN index next to the HNSW index, in the same row, in the same WHERE clause. Cross-store joins are gone.

Why a Pascal-aware chunker

Naive chunking by character count or line count produces chunks that straddle declaration boundaries. The model then has to reason about half a class definition with its other half missing — exactly the context-loss problem RAG was meant to solve.

The chunker in mormot.ai.chunk.pas walks the source, tracks {} and (* *) comment depth + single-quoted strings, and emits one chunk per top-level declaration (a type block, a procedure / function body, a const / var block, the unit header). Comments immediately preceding a declaration are attached to it; standalone comment blocks become their own chunks.

The result: each retrieved chunk is a self-contained unit of meaning (a class declaration, a procedure body, a documentation block) rather than a 200-line slice that may end mid-begin.

Trade-offs

  • Embedding quality bounds retrieval quality. A weaker model produces vectors with less semantic information; relevant chunks rank lower.
  • Chunking strategy bounds answer fidelity. A method buried inside a 5 000-line class declaration sits in a single chunk together with hundreds of unrelated methods, and the embedding cannot tell them apart. The chunker's SplitThresholdTokens mitigates this by splitting oversized declarations with a configurable line overlap.
  • Generative model bounds final answer quality. The retriever can surface the right code, but a small local model may still miss subtleties. Larger models help; the chat CLI's --llm-model flag makes the choice trivial to swap.