Real-time voice AI engineer for voice agents
Consulting for real-time voice agents: browser audio over WebRTC, streaming transcription, a text model, speech synthesis and interruptions, with the delay measured in your product.
The challenge
A voice agent must answer fast enough for a natural conversation. Each step adds delay: capture, transcription, the text model and speech synthesis.
People also interrupt. The agent must stop speaking, listen and answer the new question.
Architecture
- Browser audio over WebRTC, with the Opus codec and a separate track for each participant.
- Streaming transcription, so that the agent can start to answer sooner.
- Interruptions: when the person speaks, the agent stops its playback and listens again.
- A delay budget for each step, measured with the product's own audio.
- Capacity planning for the GPUs that run the speech and text models.
In production
I build and operate real-time audio in production: server capture over WebRTC and Opus, transcription on the conversation timeline and recovery after a restart.
Transcripts and audio stay out of logs.
Limits
- The delay depends on the network, the model size and the GPU. The target is measured with your audio; it is not promised in advance.
- Noise, accents and overlapping speech reduce transcription quality. The tests use recordings from your environment.
- A voice agent needs a handoff to a person for cases that it cannot solve.
Hire me for this
- A review of your voice pipeline, with a delay budget for each step.
- Deployment of capture, transcription and speech synthesis, with monitoring and recovery.
- Interruptions and turn-taking tuned to the speech of your users.
Let's assess your project
I reply with scope and a proposal.