What is RAG (Retrieval-Augmented Generation)?

RAG is an architecture that lets an LLM answer questions using your own documents - retrieve relevant chunks, inject them into the prompt, then generate an answer grounded in that data.

AILLMRAG

Overview

LLMs have a knowledge cutoff and do not know about your private data. RAG solves this by adding a retrieval step before generation: search your document store for the most relevant chunks, inject those chunks into the prompt as context, then let the LLM answer based on that grounded information. This is how most production AI Q&A and document-chat systems work.

Key Points

•RAG = Retrieval + Generation. Retrieval finds relevant text; generation writes the answer.
•Documents are split into chunks, converted to embeddings (vectors), and stored in a vector database.
•At query time, the question is embedded, similar chunks are retrieved (nearest neighbour search), and stuffed into the prompt.
•RAG reduces hallucination because the model is constrained to reference provided context.
•RAG keeps knowledge fresh - update the vector store without retraining the model.
•The limiting factor is the context window - you can only inject so many retrieved chunks at once.

Examples

Full RAG pipeline (conceptual walkthrough)

typescript

// ── INDEXING PHASE (run once, or on document update) ──────────────────

// 1. Split document into chunks
const chunks = splitIntoChunks(documentText, { maxTokens: 500, overlap: 50 });

// 2. Embed each chunk
const embeddings = await Promise.all(
  chunks.map(chunk => openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: chunk,
  }))
);

// 3. Store in vector database (e.g. Pinecone, pgvector, Chroma)
await vectorDB.upsert(
  chunks.map((chunk, i) => ({
    id: `doc-${i}`,
    values: embeddings[i].data[0].embedding,
    metadata: { text: chunk },
  }))
);

// ── QUERY PHASE (run on each user question) ───────────────────────────

// 4. Embed the user's question
const queryEmbedding = await openai.embeddings.create({
  model: 'text-embedding-3-small',
  input: userQuestion,
});

// 5. Find the top-5 most similar chunks
const results = await vectorDB.query({
  vector: queryEmbedding.data[0].embedding,
  topK: 5,
  includeMetadata: true,
});
const context = results.matches.map(m => m.metadata.text).join('\n\n');

// 6. Generate answer grounded in retrieved context
const answer = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    {
      role: 'system',
      content: 'Answer the question using ONLY the provided context. If the answer is not in the context, say so.',
    },
    {
      role: 'user',
      content: `Context:\n${context}\n\nQuestion: ${userQuestion}`,
    },
  ],
});

The key constraint "answer ONLY using the context" reduces hallucination by forcing the model to stay grounded.

Simple RAG with LangChain (higher-level abstraction)

typescript

import { ChatOpenAI } from '@langchain/openai';
import { OpenAIEmbeddings } from '@langchain/openai';
import { MemoryVectorStore } from 'langchain/vectorstores/memory';
import { createRetrievalChain } from 'langchain/chains/retrieval';

// Index documents
const vectorStore = await MemoryVectorStore.fromTexts(
  documentChunks,
  documentChunks.map((_, i) => ({ id: i })),
  new OpenAIEmbeddings()
);

// Create retrieval chain
const chain = await createRetrievalChain({
  retriever: vectorStore.asRetriever({ k: 5 }),
  combineDocsChain: createStuffDocumentsChain({
    llm: new ChatOpenAI({ model: 'gpt-4o' }),
    prompt: chatPrompt,
  }),
});

const result = await chain.invoke({ input: 'What is our refund policy?' });
console.log(result.answer);

Pro Tip

Chunk size matters enormously. Chunks too large lose precision; too small lose context. A common starting point: 300–500 tokens with 10–15% overlap. Also experiment with "parent-child" retrieval - retrieve small chunks for precision, but inject the larger surrounding section for context.