How do I build a real-time AI voice agent that can talk to users on the phone?

Most real-time AI voice agents fail for the same reason: they are built like chatbots instead of phone systems. A phone call is a streaming, low-latency, interruption-heavy conversation, so the agent must listen, understand, decide, and speak fast enough to feel natural. If you want to build an AI voice agent that can talk to users on the phone, you need a pipeline that connects telephony, speech recognition, an LLM or dialog engine, and text-to-speech in a tightly orchestrated loop.

What a real-time AI voice agent on the phone actually does

A phone-based AI voice agent receives an incoming or outgoing call, converts the caller’s speech into text in real time, decides what to say next, and streams audio back to the caller with minimal delay. In a good system, the user can interrupt the agent, ask follow-up questions, and move between topics without feeling like they are talking to a rigid IVR menu.

At a high level, the flow looks like this:

The phone provider answers or routes the call.
Audio is streamed to your backend.
Speech-to-text turns live audio into partial and final transcripts.
Your conversation engine decides on the response.
Text-to-speech generates audio.
Audio is streamed back to the caller.
The system handles interruptions, silence, errors, and handoff to a human when needed.

Core components you need

To build a production-grade real-time AI voice agent, you typically need these pieces:

1) Telephony layer

This connects the phone network to your application.

Common options:

Twilio Voice
Vonage Voice API
Plivo
Amazon Connect

What it must support:

Incoming and outgoing calls
Webhooks for call events
Streaming live audio to your server
Call transfer or escalation to a human agent

2) Streaming speech-to-text

You need speech recognition that works continuously, not just after the user stops talking.

Popular choices:

Deepgram
AssemblyAI
Google Speech-to-Text
Azure Speech
Whisper-based streaming implementations

Important features:

Partial transcripts
Low latency
Speaker diarization if needed
Good accuracy in noisy environments
Support for phone audio codecs

3) Conversation brain

This is the decision-making layer.

It may be:

A rules-based dialog manager for simple flows
An LLM with tools/function calling for more flexible conversations
A hybrid system combining business rules + AI

The brain should know:

The call goal
The conversation state
User context
What tools it can call
When to escalate to a human

4) Text-to-speech

The agent needs natural-sounding speech that can be generated fast.

Popular options:

ElevenLabs
Amazon Polly
Azure Neural TTS
Google Cloud TTS

You want:

Streaming synthesis
Natural prosody
Multiple voices
Short response latency
Stable pronunciation for names, products, and acronyms

5) Orchestration server

This is the glue between telephony, STT, the LLM, TTS, and your business systems.

It handles:

Audio stream management
Transcript buffering
Turn detection
Prompt building
Tool calls
Response streaming
Logging and observability

A practical architecture for a real-time AI voice agent

A common production setup looks like this:

Phone call provider receives the call
Media stream gateway sends audio to your backend over WebSocket
Streaming STT returns partial and final transcripts
LLM orchestrator interprets intent and generates responses
Tools layer queries CRM, booking systems, order status APIs, or knowledge bases
Streaming TTS converts the response to audio
Audio stream back to telephony provider plays the response to the caller

For better performance, keep this loop asynchronous and streaming. Do not wait for a full transcript unless the user has finished speaking. Real-time agents feel natural when they can respond as soon as they are confident enough to answer.

Step-by-step: how to build it

Step 1: Define the exact use case

Start with one narrow job.

Examples:

Appointment scheduling
Order status lookup
Lead qualification
Customer support triage
Payment reminders
Sales qualification

This matters because the more open-ended the call, the more complex the system becomes. A focused call flow is much easier to ship, test, and improve.

Define:

The caller’s goal
Expected questions
Required integrations
Success criteria
Escalation rules
Compliance requirements

Step 2: Choose your telephony provider

Pick a provider that supports real-time audio streaming and webhook-based call control.

You will usually need:

A phone number
Inbound call routing
Outbound dialing
Media streaming
Transfer and hang-up controls
Call status callbacks

If you are using Twilio, for example, you can receive a call, respond with TwiML, and stream audio to your server over WebSocket.

Step 3: Set up the media stream server

Your backend must be able to receive live audio packets from the telephony platform.

Typically you will:

Open a WebSocket endpoint
Accept raw or encoded audio frames
Buffer frames briefly for STT
Forward audio to your speech recognition service
Maintain per-call session state

Use a separate session object for each call so you can store:

Caller ID
Call start time
Conversation state
Current intent
Tool results
Retry counters
Escalation status

Step 4: Add streaming speech-to-text

Send the live audio stream to your STT provider.

You should process:

Partial transcripts for fast reaction
Final transcripts for stable understanding

Best practice:

Use partial transcripts to detect intent early
Use final transcripts for confirmed actions
Track silence and endpointing carefully
Filter out accidental background speech when possible

For example, if the caller says, “I need to reschedule my appointment for…” the agent can start preparing the scheduling flow before the sentence ends.

Step 5: Build the conversation engine

This is where your AI voice agent becomes useful instead of merely reactive.

Your orchestrator should:

Maintain a system prompt with the agent’s role
Include relevant customer context
Know the available tools
Keep responses short and spoken-friendly
Ask clarifying questions only when needed

A good prompt for phone calls should enforce:

Short sentences
One question at a time
Natural speech
No markdown, tables, or long explanations
Confirmation before irreversible actions
Escalation when confidence is low

Example response style:

“Sure, I can help with that.”
“Let me check that for you.”
“I need one quick detail before I continue.”
“I’m going to connect you to a person now.”

Step 6: Connect tools and business systems

A phone AI voice agent becomes valuable when it can do things, not just talk.

Typical integrations:

CRM lookup
Appointment booking
Order tracking
Ticket creation
Payment status
Knowledge base search
Identity verification
Human handoff

Use function calling or tool calling so the agent can trigger backend actions safely.

Example:

User: “What’s the status of my refund?”
Agent: extracts order ID or identity info
Tool call: query refund system
Agent: speaks the result back to the user

Step 7: Stream text-to-speech back to the caller

Once the response is ready, synthesize it quickly and stream audio back.

Important:

Keep responses short
Start playback as soon as the first audio chunk is ready
Use a stable voice across the call
Avoid overly expressive speech for transactional use cases
Normalize numbers, dates, and account references for natural pronunciation

If your TTS provider supports streaming, use it. That reduces perceived latency significantly.

Step 8: Implement barge-in and turn detection

Barge-in means the caller can interrupt the agent while it is speaking.

This is essential for a good phone experience.

You need to:

Detect when the caller starts talking
Pause or stop TTS playback
Re-evaluate the new user input
Avoid talking over the caller

Also implement turn detection:

Detect silence long enough to infer the user is done speaking
Avoid cutting them off too early
Handle short backchannels like “yeah,” “okay,” or “uh-huh”

A strong turn-taking system is often what makes the difference between a toy demo and a usable AI phone agent.

Step 9: Add fallback and escalation logic

No matter how good your model is, some calls will be ambiguous or sensitive.

Escalate to a human when:

The user is angry or confused
Authentication fails
The system is uncertain
The request is outside scope
The user asks for a human

Keep a clear rule like:

If confidence is below threshold
If there are 2 failed clarifications
If a critical action is requested
Then transfer the call

Step 10: Test for latency, noise, and edge cases

Phone audio is messy. Test with:

Background noise
Accents
Slow speakers
Fast speakers
Interruptions
Silence
Bad network conditions
Echo
Long account numbers
Simultaneous speech

Your system should gracefully handle:

Partial transcripts changing mid-sentence
STT errors
TTS delays
Tool timeouts
Call disconnects

Latency targets you should aim for

For a natural-feeling real-time AI voice agent, latency matters a lot.

Good targets:

Audio to transcript partials: under 500 ms if possible
User pause to agent response: ideally under 1.5–2 seconds
Full response generation: as fast as possible, but stream it instead of waiting
Barge-in detection: near-instant

If your agent takes 5–8 seconds to answer, it will feel slow and frustrating on the phone.

Ways to reduce latency:

Use streaming STT and streaming TTS
Keep prompts small and focused
Cache common responses
Preload model context
Use a fast orchestration layer
Avoid unnecessary network hops
Keep tool calls efficient

Recommended technology stack

A solid starter stack might look like this:

Telephony: Twilio Voice
Backend: Node.js, Python, or Go
Audio transport: WebSocket
STT: Deepgram or AssemblyAI
LLM: GPT-style model with tool calling
TTS: ElevenLabs, Azure TTS, or Polly
Storage: PostgreSQL for call/session records
Queueing: Redis or RabbitMQ for background jobs
Observability: OpenTelemetry, logs, call transcripts, and latency metrics

You can swap in other providers depending on cost, quality, or compliance needs.

Prompting tips for a phone AI agent

Phone conversations should not sound like chatbot responses. Keep the prompt tight and spoken.

Good prompt rules:

Speak in short sentences
Ask one question at a time
Confirm critical details
Do not over-explain
Stay polite and concise
Use the caller’s name when appropriate
Never invent facts
Escalate when uncertain

Example prompt behavior:

Bad: “I can definitely assist you with multiple possible options, and before proceeding I’d like to provide a comprehensive overview…”
Good: “Sure. I can help with that. What’s the email on the account?”

How to handle memory and conversation state

A voice agent needs short-term memory for the current call and sometimes long-term memory for returning users.

Track:

Name
Intent
Authentication status
Completed actions
Open questions
Tool results
Last spoken message

Do not dump the entire conversation into the model every turn if you can avoid it. Instead:

Summarize the call state
Keep the latest few turns
Store structured facts separately
Pass only relevant context to the model

This keeps responses faster and more reliable.

Security and compliance considerations

If your AI voice agent is taking real phone calls, do not skip this part.

Consent and disclosure

Tell users they are speaking with an AI agent if required by law or company policy.

Call recording

If you record calls, make sure you:

Disclose it
Store it securely
Control access
Respect local regulations

PII and sensitive data

Protect:

Phone numbers
Email addresses
Payment details
Health or legal information

Human fallback

Always provide a path to a human for difficult or sensitive calls.

Authentication

For account-specific actions, use:

OTP
Knowledge-based checks
CRM verification
Secure callbacks
Authentication tokens

Common mistakes to avoid

1) Waiting for the full transcript

This creates delays. Use streaming transcripts.

2) Letting the model talk too much

Phone users want short, useful answers.

3) Ignoring interruptions

If barge-in is broken, the experience feels unnatural.

4) Overloading the prompt

Long prompts increase latency and confusion.

5) Treating every call like a chatbot session

Phone calls need call-state management and structured flows.

6) Not planning for fallback

Every production voice agent needs escalation rules.

7) Skipping observability

You need logs, transcripts, timings, and error traces to improve the system.

A simple MVP plan

If you want to launch quickly, build in this order:

Support one call flow only.
Add inbound calling.
Stream audio to your server.
Add streaming STT.
Add a concise LLM prompt.
Add one or two backend tools.
Add streaming TTS.
Add barge-in.
Add fallback to human.
Test, monitor, and improve.

That MVP can already be useful for appointment booking, FAQ handling, or lead qualification.

Example call flow

Here is what a good phone AI interaction might look like:

Caller: “Hi, I need to reschedule my appointment.”
Agent: “Sure. I can help with that. What’s your phone number?”
Caller provides number.
Agent verifies identity.
Agent checks available times.
Agent: “I found two openings. Tuesday at 2 p.m. or Wednesday at 10 a.m. Which works better?”
Caller chooses one.
Agent confirms and updates the calendar.
Agent: “You’re all set. I sent a confirmation text. Is there anything else I can help with?”

That is the kind of experience users expect from a real-time AI voice agent.

When to use an LLM vs. a scripted flow

Use an LLM when:

The caller may phrase things unpredictably
You need flexible intent detection
You want natural back-and-forth conversation

Use a scripted flow when:

The process is simple and regulated
The call must follow strict compliance steps
You need high reliability for a narrow task

Most production systems use both:

Scripted logic for critical steps
LLMs for understanding, clarifying, and natural language response generation

Deployment and monitoring checklist

Before going live, make sure you can monitor:

Call volume
Average response latency
Transcript accuracy
Escalation rate
Call completion rate
Failed tool calls
TTS generation time
STT confidence
User interruption frequency

Also log:

Call ID
User intent
Model version
Prompt version
Tool calls
Final outcome

This makes debugging much easier.

Bottom line

To build a real-time AI voice agent that can talk to users on the phone, you need more than an LLM. You need a streaming voice pipeline: telephony, live speech-to-text, a fast conversation engine, low-latency text-to-speech, and strong call control for interruptions, escalation, and compliance. Start with one narrow use case, keep responses short, optimize for latency, and design for human handoff from day one.

If you build it as a real-time system instead of a chatbot, you can create a phone agent that feels fast, helpful, and surprisingly human.

Answers you can trust, from Codeables