How to Fix LLM Hallucinations in Your Production App

Hallucinations cost real money. A customer support bot that invents refund policies, a document summarizer that adds facts, a coding assistant that references non-existent APIs — these aren't edge cases, they're what happens when you deploy an LLM without guardrails.

Here's the framework we use to reduce hallucinations to an acceptable rate in production.

Why LLMs hallucinate

The model doesn't "know" facts the way a database does. It predicts the next token based on patterns in training data. When asked something outside its training or context, it confidently generates plausible-sounding text instead of saying "I don't know."

Four main causes:

1. Question is outside the context — the answer isn't in the prompt
2. Ambiguous question — multiple valid interpretations, model picks one
3. Model overconfidence — high-temperature or poorly calibrated outputs
4. Conflicting context — documents in the prompt contradict each other

Fix 1: Ground every response in retrieved context (RAG)

The single highest-ROI fix. If the model can only answer from documents you provide, it can't invent facts it wasn't given.

def build_grounded_prompt(question: str, retrieved_docs: list[str]) -> str:
    context = "\n\n".join(f"[Doc {i+1}]: {doc}" for i, doc in enumerate(retrieved_docs))
    return f"""Answer the question using ONLY the documents below.
If the documents don't contain enough information, say: "I don't have that information."
Do not add facts not present in the documents.

Documents:
{context}

Question: {question}

Answer:"""

The key instruction: explicitly forbid adding information not in the documents.

Fix 2: Set temperature to 0.0–0.2

Temperature controls randomness. Higher temperature = more creative = more hallucinations. For factual tasks, set it low.

response = client.chat.completions.create(
    model="gpt-4o",
    messages=messages,
    temperature=0.1,  # NOT the default 1.0
)

For creative tasks (writing, brainstorming), higher temperature is fine. For factual retrieval, classification, or extraction — always low.

Fix 3: Force structured output and validate it

When the model returns structured data (JSON, specific fields), use response_format to enforce valid JSON, then validate the schema.

from pydantic import BaseModel, ValidationError

class Answer(BaseModel):
    response: str
    confidence: float  # 0.0 to 1.0
    sources_used: list[int]  # which doc numbers were cited

response = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=messages,
    response_format=Answer,
    temperature=0.1,
)

answer = response.choices[0].message.parsed

# If confidence is low, route to human review
if answer.confidence < 0.7:
    return route_to_human_review(question, answer)

Fix 4: Add a self-check pass

Ask the model to verify its own answer against the source material. Adds latency but catches obvious fabrications.

async def answer_with_verification(question: str, docs: list[str]) -> str:
    # First pass: generate answer
    answer = await generate_answer(question, docs)

    # Second pass: verify
    verification_prompt = f"""
    Question: {question}
    Answer: {answer}
    Source documents: {docs}

    Is every fact in the answer supported by the source documents?
    Reply with JSON: {{"verified": true/false, "issues": ["list any unsupported claims"]}}
    """

    check = await client.chat.completions.create(
        model="gpt-4o-mini",  # cheaper model for verification
        messages=[{"role": "user", "content": verification_prompt}],
        response_format={"type": "json_object"},
        temperature=0,
    )

    result = json.loads(check.choices[0].message.content)
    if not result["verified"]:
        # Re-generate with issues highlighted
        return await regenerate_with_corrections(question, docs, result["issues"])

    return answer

Fix 5: Require citations

Force the model to cite which document each claim comes from. Uncited claims become obvious and easy to catch.

system_prompt = """
When answering, cite your sources using [Doc N] notation after each claim.
Example: "The refund window is 30 days [Doc 2]."
If you make a claim without a citation, that means you're unsure — don't make it.
"""

Fix 6: Implement a human review queue

For high-stakes outputs (legal, medical, financial, customer-facing), don't fully automate. Build a confidence threshold below which answers go to a human.

CONFIDENCE_THRESHOLD = 0.85

async def process_question(question: str) -> dict:
    answer = await generate_answer_with_confidence(question)

    if answer["confidence"] < CONFIDENCE_THRESHOLD:
        await add_to_review_queue(question, answer)
        return {"status": "pending_review", "eta": "2 hours"}

    return {"status": "answered", "response": answer["text"]}

Measuring hallucination rate

Track it. Pick 100 representative questions where you know the ground-truth answer, run them weekly, and measure what % the model gets wrong or fabricates. That's your hallucination rate. Set a target and alert when it degrades.

Most teams don't do this and discover hallucinations from angry customers instead.

Building an LLM-powered product? We've shipped production AI systems with hallucination rates under 2%. Get in touch.