What is RAG (retrieval-augmented generation) and when do I need it?

For answering questions from your documents, RAG is almost always the right starting point:

1. Index: split documents into chunks (a few hundred tokens each, with some overlap), compute embeddings, and store them in a vector store (pgvector, Qdrant…). 2. Retrieve: embed the user's question and fetch the most similar chunks. Hybrid search (vectors + keyword/BM25) retrieves noticeably better. 3. Generate: send the question plus the retrieved chunks to the LLM, with instructions to answer only from the provided context and cite sources.

Why not fine-tuning here: - Fine-tuning teaches style and behavior; it's unreliable for memorizing facts. - With RAG, updating knowledge means re-indexing a document, not retraining. - RAG gives citations, so users can verify answers.

With 2,000 pages, also consider simply putting the relevant docs into a long-context model with prompt caching. It's sometimes simpler than a vector pipeline. Either way, build a small evaluation set of real questions with expected answers before tuning anything.

Practical tip: chunk along the document's structure (headings, sections) rather than fixed character counts, and store the section title with each chunk. Retrieval quality improves a lot.