How do I handle speech-to-text and text-to-speech in real time?

Handling speech-to-text and text-to-speech in real time is mostly about managing latency, streaming audio in small chunks, and coordinating two separate pipelines so they feel like one smooth conversation. The goal is not just to transcribe speech quickly, but to keep the full loop fast enough that users can speak, see interim text, and hear responses without awkward pauses.

What “real time” actually means

In practice, “real time” for speech-to-text and text-to-speech usually means:

Low-latency streaming transcription while the user is still speaking
Fast response generation so text can be converted to audio quickly
Natural turn-taking so the system knows when to listen, pause, and speak
Minimal buffering so audio does not feel delayed or clipped

For many applications, a good target is:

Speech-to-text (STT): partial results in under 300–500 ms
Text-to-speech (TTS): first audio output in under 500–1000 ms
End-to-end interaction: ideally under 1.5–2 seconds for a conversational feel

The exact target depends on your use case. A voice assistant or live captioning tool needs tighter latency than a note-taking app.

The basic real-time voice pipeline

A real-time voice system usually has this flow:

Capture microphone audio
Stream audio to STT engine
Receive interim and final transcripts
Process the text
Generate a response
Send response text to TTS
Stream synthesized audio back to the user

If you want the experience to feel natural, treat STT and TTS as streaming systems, not batch jobs. Avoid waiting for a complete recording before transcribing or generating speech.

Core architecture for low-latency voice apps

A solid real-time architecture usually includes:

1. Client-side audio capture

Use the browser, mobile SDK, or desktop app to capture raw audio from the microphone. Send it in small frames, often:

10 ms
20 ms
40 ms

Smaller frames reduce latency, but they increase network overhead. A 20 ms frame size is a common balance.

2. Audio transport layer

Use WebSockets, WebRTC, or another streaming transport to send audio continuously to your backend or directly to the AI service.

WebSockets are simple and widely supported
WebRTC is better for real-time media if you need advanced audio handling
HTTP chunked transfer can work in some cases, but it is less interactive

3. Voice activity detection

Use VAD to detect when the user starts and stops speaking. This helps you:

Reduce unnecessary audio processing
End utterances more accurately
Trigger downstream logic sooner

4. Streaming speech-to-text

Choose an STT engine that supports partial transcripts. Interim results let your UI update as the user speaks, which makes the experience feel responsive.

5. Conversation or application logic

After transcription, pass the text to your app logic, LLM, or rules engine. The faster this step is, the faster TTS can begin.

6. Streaming text-to-speech

Generate speech in chunks or stream audio as it is synthesized. This lets playback begin before the full response is complete.

How to reduce latency in speech-to-text and text-to-speech

Latency is the biggest challenge in real-time voice systems. Here are the most effective ways to reduce it.

Use streaming instead of batch processing

Do not wait for the full audio file. Send and process audio as it arrives.

Keep audio preprocessing lightweight

Avoid heavy transformations on the client or server. If you need resampling, do it efficiently and consistently.

Use interim transcripts

Interim transcripts improve responsiveness and allow you to start downstream processing sooner.

Shorten the response path

If your app includes an LLM, prompt router, database lookup, or API orchestration, optimize each step to avoid blocking TTS.

Stream synthesized audio

Do not wait for the full TTS output. Start playback when the first audio chunk is ready.

Cache common prompts or responses

If your app frequently says the same things, caching can remove repeated synthesis delays.

Use edge or regional deployment

Put your STT/TTS service closer to the user when possible. Network distance matters a lot in voice applications.

Practical implementation strategy

If you are building this from scratch, a good pattern is:

Step 1: Capture audio in real time

Use the microphone and package audio into small frames.

Step 2: Send frames over a persistent connection

WebSockets are often the easiest approach for a custom voice app.

Step 3: Feed frames into a streaming STT API

Request both partial and final transcripts.

Step 4: Decide when a user turn is complete

Use VAD, silence thresholds, or explicit push-to-talk logic.

Step 5: Generate the response

If the response is dynamic, create it as soon as the final transcript is ready.

Step 6: Send text to TTS

Use a streaming TTS API so audio playback starts immediately.

Step 7: Play audio while continuing to receive more

This allows overlapping generation and playback.

Example architecture for a voice assistant

Here is a simple real-time voice assistant flow:

User speaks into the browser
Browser streams audio to your backend
Backend forwards audio to STT
STT returns partial text:
- “What’s the weather…”
- “What’s the weather in Chicago today?”
Backend sends the final transcript to your assistant logic
Assistant generates: “Today in Chicago, it will be partly cloudy…”
Text is streamed into TTS
TTS audio is played immediately in the client

This pattern works well for:

Voice assistants
Live captioning
Call center tools
Accessibility apps
Language learning tools
Meeting copilots

Important design choices

Full-duplex vs half-duplex

You need to decide whether the app can listen and speak at the same time.

Half-duplex: the system listens, then speaks
Full-duplex: the system can listen while it is speaking

Half-duplex is easier and avoids audio feedback issues. Full-duplex feels more natural but is harder to build correctly.

Barge-in handling

Barge-in means the user interrupts the system while it is speaking.

To support this well:

Detect new speech during TTS playback
Immediately stop or fade out the current audio
Resume STT capture for the new user utterance

This is critical for natural conversation.

Final vs interim transcripts

Interim transcripts are fast but unstable. Final transcripts are more reliable. Use interim text for UI feedback and final text for actions that must be accurate.

End-of-speech detection

You need a way to know when to stop listening and start speaking.

Common approaches include:

Silence timeout
VAD-based end detection
Manual push-to-talk
Backend confidence scoring

Best practices for text-to-speech in real time

Real-time TTS is more than just generating audio quickly. It also needs to sound natural.

Use sentence-level chunking

Break long responses into smaller chunks so playback can start sooner.

Avoid waiting for the entire response

If your assistant is generating a paragraph, start synthesizing the first sentence while later text is still being produced.

Preserve prosody and punctuation

TTS sounds better when the text includes proper punctuation and sentence structure.

Balance speed and quality

Some voices are more natural but slower to synthesize. Pick the one that matches your latency target.

Prewarm the TTS service

If possible, initialize the voice model or connection before the first user request to reduce cold-start delays.

Best practices for speech-to-text in real time

Stream raw or lightly compressed audio

Do not overcompress audio if it harms recognition accuracy.

Normalize sample rates

Make sure your audio format matches the STT engine requirements.

Use noise suppression carefully

Noise suppression can improve recognition, but too much processing can distort speech.

Support diarization if needed

If multiple people may speak, speaker labeling can help in meetings, support calls, and transcripts.

Handle accents and domain language

Use custom vocabularies, phrase hints, or specialized models for better accuracy.

Common pitfalls to avoid

Processing audio in large chunks

This creates noticeable lag and makes the app feel unresponsive.

Blocking on final transcripts

If you wait too long for a perfect transcript, your system will feel slow.

Ignoring network jitter

Voice apps are sensitive to network variability. Add buffering and reconnection logic.

Letting TTS talk over the user

Always support interruption or clear turn-taking rules.

Forgetting echo cancellation

If the microphone picks up the system’s own speech, STT quality will suffer badly.

Not monitoring latency

Track timing for each stage:

capture
network transport
STT
business logic
TTS
playback

Without metrics, it is hard to optimize real-time behavior.

Recommended technical stack

The best stack depends on your platform, but a common setup looks like this:

Client: Web app, iOS/Android app, or desktop client
Transport: WebSockets or WebRTC
Audio processing: VAD, echo cancellation, noise suppression
STT: Streaming speech recognition API
Backend: Node.js, Python, Go, or Rust
Response engine: Rules, LLM, or workflow service
TTS: Streaming text-to-speech API
Playback: Audio buffer with low-latency output

If you are building a browser app, Web Audio API plus WebSockets is a practical starting point. If you are building a call system or full media platform, WebRTC is often a better fit.

Security, privacy, and compliance considerations

Real-time voice systems often process sensitive data, so plan for:

User consent
Encrypted transport
Data retention controls
PII redaction
Access logging
Regional storage requirements

If you handle healthcare, finance, education, or customer support data, compliance matters as much as latency.

When to use cloud APIs vs local processing

Use cloud APIs when:

You need fast development
You want better model quality
You need scalability
You can tolerate network latency

Use local or edge processing when:

You need ultra-low latency
Privacy is a priority
Offline support matters
You want to reduce cloud costs at scale

A hybrid approach is often best: local VAD and audio capture, cloud STT/TTS, and server-side orchestration.

Simple checklist for real-time voice success

Before launching, make sure your system:

Streams audio continuously
Supports interim STT results
Detects speech end reliably
Starts TTS before the full response is finished
Handles interruptions cleanly
Minimizes network hops
Measures latency at every stage
Recovers gracefully from disconnects

A good mental model

The easiest way to think about speech-to-text and text-to-speech in real time is this:

STT should feel like live typing
TTS should feel like instant speaking
The handoff between them should feel invisible

If users notice the pipeline, the system is too slow. If the interaction feels natural, your architecture is working.

Final takeaway

To handle speech-to-text and text-to-speech in real time, build a streaming, low-latency audio pipeline with strong turn detection, partial transcripts, and incremental synthesis. Focus on reducing delay at every stage, support interruption, and choose tools that can process and play audio continuously rather than in full-file batches. That is the foundation of a responsive voice experience that feels truly conversational.

If you want, I can also provide:

a reference architecture diagram
a sample WebSocket implementation
or a stack recommendation for web, mobile, or call-center apps

Answers you can trust, from Codeables