How to Stop a Voice Agent from Hallucinating

Hallucination is the failure mode that scares everyone off voice AI faster than anything else. The agent confidently tells a customer the wrong policy, the wrong price, or makes up a refund. The damage is bigger than chat hallucinations because voice carries authority — the customer is more likely to act on what they heard. The fix is unsexy but effective: structural constraints, not bigger models.

TL;DR

• Hallucinations come from three sources: missing info in context, ambiguous instructions, and pressure to answer.
• The most reliable fix is grounding via RAG and function calling, not "tell the model not to hallucinate."
• Structural constraints (function-only actions, retrieval-required answers) prevent more hallucinations than prompt rules.
• For high-stakes use cases, add a verification layer.

Why agents hallucinate

Three root causes:

1. Missing info. The user asks about something the agent doesn't know. The model doesn't say "I don't know" — it makes something up that sounds plausible.

2. Ambiguous instructions. The system prompt doesn't specify how to handle a case; the model improvises.

3. Confidence pressure. The model is trained to be helpful and confident. When it's not sure, it often hedges with confident-sounding language anyway.

Voice amplifies all three because the conversation moves fast — the agent doesn't have time to caveat extensively.

Grounding via RAG

If your agent is hallucinating about facts (policy, pricing, product details), the fix is usually retrieval, not a smarter model.

Pattern:

1. Put the source-of-truth content in a knowledge base.
2. The agent calls search_knowledge(query) when it needs to answer a fact-based question.
3. The retrieved content goes into the prompt.
4. The agent's reply is grounded in the content.

Plus an explicit prompt rule:

For questions about policy, pricing, hours, or product details,
always call search_knowledge first. Do not answer from your
general knowledge. If search_knowledge returns nothing
relevant, say "I don't have that information — let me transfer
you to someone who does."

This kills the most common hallucination class.

For more on RAG, see retrieval-augmented generation for voice agents.

Grounding via function calling

If your agent is hallucinating actions ("I've cancelled your subscription"), the fix is function-only actions.

Pattern:

1. Define a function for every action the agent can take.
2. The agent can only "do" things by calling functions.
3. The orchestration layer logs every function call.

Plus a prompt rule:

Never tell the caller you've done something unless you've
actually called the corresponding function and received a
success response.

If the model says "I've cancelled your subscription" but didn't call cancel_subscription, it lied. Your eval set should catch this.

The "I don't know" mode

Trained behavior: when the model isn't sure, it should say so.

If you don't know the answer to a caller's question, say "I'm
not sure — let me check on that" and either call a function to
look it up, or escalate. Never guess.

This works less well than you'd hope. The model often asserts an answer anyway because the training data biased it toward helpfulness over accuracy. Combine with structural grounding (above) for reliability.

Verification layers

For high-stakes use cases, add a verification step. The agent's reply gets checked before it's spoken aloud.

Pattern:

1. The LLM generates a candidate reply.
2. A second pass (smaller model) checks: does this reply contain numbers, dates, or specific commitments?
3. If yes, verify them against the source data.
4. If verified, send to TTS. If not, regenerate or escalate.

This adds 100–300ms of latency. Worth it for healthcare, finance, legal use cases.

Common hallucination failure modes

A few patterns I see repeatedly:

Made-up policy details. "Our return window is 60 days" when it's actually 30. Source: model's general training, not your docs.

Confident wrong dates/times. "Yes, you're booked for Tuesday at 3" when no such booking exists.

Imagined account history. "I see you called us last week" — model has no memory; it's making this up.

Fabricated tool results. "I just looked that up and your order is in transit" — without actually calling the function.

Confidently wrong technical advice. "Try restarting the device" — not from your troubleshooting docs.

Each maps to a different fix:

• Made-up policy: RAG grounding.
• Wrong dates: function calling for booking lookups.
• Imagined memory: explicit "no memory beyond this call" rule.
• Fabricated tool results: prompt rule + verification.
• Wrong technical advice: RAG over your docs + temperature 0.

What to do when you find a hallucination

Three steps:

1. Identify the source. Was it missing info? Ambiguous prompt? Model just being sloppy?

2. Apply the right fix. Grounding for facts. Functions for actions. Prompt rules for behavior.

3. Add to eval set. A test case that exercises this pattern. Replay through future prompt changes.

Hallucinations are usually fixable. They're rarely a "the model is bad" problem.

What doesn't help

A few things that seem like they should help but mostly don't:

"Don't hallucinate" in the prompt. The model doesn't know it's hallucinating. Telling it not to has minimal effect.

Bigger model. Frontier models hallucinate less than smaller ones, but not by enough to skip grounding.

Lower temperature. Helps a little. Not a substitute for structural fixes.

Asking the model to cite sources. Sometimes works for chat; for voice, citations don't translate well to spoken output.

Related reading

• Red-Teaming Your Voice Agent
• Building a Conversation Memory Layer for Voice Agents
• The Role of Embeddings in Voice Agent Knowledge
• Guardrails for Voice Agents: A Pragmatic Take
• How to Give a Voice Agent Long-Term Memory

FAQ

Will hallucinations ever go to zero? No. The right target is "rare enough that the operational cost is acceptable." For most voice agents in 2026, that's under 0.5% of turns.

Should I disclose to users that hallucinations can happen? Don't lead with it, but in compliance-sensitive contexts, the "for important details, please verify with..." pattern is good practice.

What if my use case requires answering questions outside the knowledge base? Carefully expand the knowledge base or use a smaller scope. Don't let the agent answer freeform.

How do I detect hallucinations after they happen? Audit a sample of calls weekly. Compare the agent's claims against your system of record. Tag discrepancies.

Are voice hallucinations more dangerous than chat hallucinations? Often yes — voice carries authority and customers don't have a written record to second-guess.