The core idea
Retrieval-augmented generation gives a language model relevant material at the moment it answers. Instead of relying only on what it learned during training, the system searches a knowledge source, selects passages, and includes them in the prompt.
The appeal is grounding. An answer can be tied to a document, a policy, or a record that the organization controls and can update without retraining the model.
Embeddings and search
Documents are split into chunks, converted into embeddings, and stored in a vector index. A query is embedded the same way, and the nearest chunks are retrieved.
Chunking is an underrated decision. Chunks that are too large dilute relevance; chunks that are too small lose context. The right size depends on the documents and the questions being asked.
Grounding is a behaviour, not a guarantee
Retrieval does not automatically prevent fabrication. A model can ignore the retrieved context, blend it with prior knowledge, or overstate what a passage says. The prompt and the validation layer have to encourage and check grounding.
Useful responses often include citations or identifiers that point back to the source. That makes verification possible and gives the user a way to judge the answer.
Evaluate retrieval and generation separately
When an answer is wrong, the cause may be retrieval, ranking, prompting, or generation. Measuring them separately makes the problem tractable.
Build a small evaluation set of real questions with known answers and expected sources. Track whether the right material was retrieved and whether the final answer used it correctly. That is more useful than a single overall score.
TechieMike
AI Systems Engineer