RAG: Give Your LLM Access to Your Own Documents

LLMs are impressive until you ask them about your company's onboarding process, your product's changelog, or last quarter's revenue. They don't know. Their training data has a cutoff, and it never included your internal docs anyway.

RAG (Retrieval-Augmented Generation) solves this. Instead of asking the model to remember your documents, you retrieve the relevant ones at query time and inject them into the prompt. The model reasons over real context, not memorized patterns.

This is the most practical AI upgrade you can make to most business applications.

How RAG works

User question
     ↓
Embed the question → vector
     ↓
Search your document store for similar vectors
     ↓
Retrieve top-K relevant chunks
     ↓
Build prompt: question + retrieved chunks
     ↓
LLM generates answer grounded in your documents

The key insight: the model doesn't need to know your documents in advance. It just needs to read the right ones at the moment you ask.

Step 1: Chunk your documents

LLMs have context limits. You can't feed in a 200-page manual and ask a question. You need to split documents into chunks that fit in the prompt and are semantically coherent.

interface Chunk {
  id: string
  documentId: string
  text: string
  metadata: {
    source: string
    page?: number
    section?: string
  }
}

function chunkDocument(text: string, source: string, chunkSize = 500, overlap = 50): Chunk[] {
  const words = text.split(/\s+/)
  const chunks: Chunk[] = []
  let i = 0

  while (i < words.length) {
    const chunkWords = words.slice(i, i + chunkSize)
    const chunkText = chunkWords.join(' ')

    chunks.push({
      id: `${source}-${i}`,
      documentId: source,
      text: chunkText,
      metadata: { source }
    })

    i += chunkSize - overlap // overlap prevents context loss at boundaries
  }

  return chunks
}

Chunk size matters. Too small (< 100 words) and chunks lose context. Too large (> 1000 words) and you retrieve too much irrelevant content. 300–600 words is a good starting range.

Step 2: Embed and store chunks

Same embedding approach as semantic search, but at the chunk level.

import OpenAI from 'openai'
import { Pool } from 'pg'

const openai = new OpenAI()
const db = new Pool()

async function ingestDocument(filePath: string) {
  const text = readFileSync(filePath, 'utf-8')
  const chunks = chunkDocument(text, filePath)

  console.log(`Ingesting ${chunks.length} chunks from ${filePath}`)

  for (const chunk of chunks) {
    const { data } = await openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: chunk.text,
    })

    await db.query(`
      INSERT INTO document_chunks (id, document_id, text, metadata, embedding)
      VALUES ($1, $2, $3, $4, $5)
      ON CONFLICT (id) DO UPDATE SET
        text = EXCLUDED.text,
        embedding = EXCLUDED.embedding
    `, [chunk.id, chunk.documentId, chunk.text, JSON.stringify(chunk.metadata), 
        `[${data[0].embedding.join(',')}]`])
  }
}

// Ingest a directory of docs
const docs = readdirSync('./docs').filter(f => f.endsWith('.md') || f.endsWith('.txt'))
for (const doc of docs) {
  await ingestDocument(path.join('./docs', doc))
}

Step 3: Retrieve relevant chunks

async function retrieveChunks(question: string, topK = 5): Promise<Chunk[]> {
  const { data } = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: question,
  })
  const queryVector = data[0].embedding

  const { rows } = await db.query(`
    SELECT id, text, metadata,
           1 - (embedding <=> $1::vector) AS similarity
    FROM document_chunks
    ORDER BY embedding <=> $1::vector
    LIMIT $2
  `, [`[${queryVector.join(',')}]`, topK])

  return rows
}

Step 4: Build the prompt and call the LLM

This is where RAG either works well or falls apart. The prompt design is everything.

async function answerWithRAG(question: string): Promise<string> {
  // 1. Retrieve relevant chunks
  const chunks = await retrieveChunks(question, 5)

  if (chunks.length === 0) {
    return "I don't have information about that in the available documents."
  }

  // 2. Build context from chunks
  const context = chunks
    .map((chunk, i) => `[Document ${i + 1}] (${chunk.metadata.source})\n${chunk.text}`)
    .join('\n\n---\n\n')

  // 3. Build the prompt
  const prompt = `You are a helpful assistant. Answer the question below using ONLY the provided documents.
If the documents don't contain enough information to answer, say so explicitly.
Do not make up information.

Documents:
${context}

Question: ${question}

Answer:`

  // 4. Call the LLM
  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: prompt }],
    temperature: 0.2, // lower temp = more faithful to source material
  })

  return response.choices[0].message.content ?? ""
}

// Usage
const answer = await answerWithRAG("What is our refund policy for enterprise customers?")

The parts most tutorials skip

1. Re-ranking

Vector similarity isn't perfect. Sometimes the 3rd result is actually more relevant than the 1st. A cross-encoder re-ranker reads question + chunk pairs and scores them more accurately.

// Simple re-ranking using the LLM itself (slower but no extra model needed)
async function rerankChunks(question: string, chunks: Chunk[]): Promise<Chunk[]> {
  const scored = await Promise.all(chunks.map(async chunk => {
    const response = await openai.chat.completions.create({
      model: 'gpt-4o-mini', // cheap model for scoring
      messages: [{
        role: 'user',
        content: `Rate how relevant this document chunk is to the question on a scale of 1-10. Reply with only a number.

Question: ${question}

Chunk: ${chunk.text.slice(0, 500)}`
      }],
    })
    const score = parseInt(response.choices[0].message.content ?? '0')
    return { chunk, score }
  }))

  return scored
    .sort((a, b) => b.score - a.score)
    .map(s => s.chunk)
}

2. Handling "I don't know"

Without guardrails, the model will hallucinate an answer when the retrieved chunks don't contain it. Force it to cite sources or admit ignorance.

const prompt = `...
Rules:
- Only answer based on the documents above
- If the answer isn't in the documents, respond with: "I don't have that information in the available documents."
- Cite which document number(s) you used at the end of your answer
...`

3. Keeping your index fresh

Documents change. Add an ingestion trigger whenever content is updated:

// In your content update handler
async function onDocumentUpdated(docId: string, newContent: string) {
  // Delete old chunks
  await db.query('DELETE FROM document_chunks WHERE document_id = $1', [docId])
  // Re-ingest
  await ingestDocument(newContent, docId)
}

When RAG is the right choice

RAG works well when:

• You have a knowledge base users query in natural language (docs, FAQs, policies)
• Your data changes frequently (fine-tuning would be stale immediately)
• You need the model to cite sources
• You have hundreds to millions of documents

Consider alternatives when:

• You need the model to deeply reason over all your data simultaneously (long context models might be better)
• Your data is highly structured (a regular database + SQL is faster and more reliable)
• You need real-time data (RAG retrieves from a snapshot — add a tool for live lookups)

---

Building a knowledge base or internal search tool? This is one of our most common builds. Start a conversation.