Tutorial · Voice
How to set up a voice AI agent
In this tutorial you will give a VegaDūta agent a voice: it will listen through speech-to-text, answer through text-to-speech, and take conversations over a real phone line or directly in the browser. You choose the ear (locally-run Whisper or Sarvam), the voice (ElevenLabs or Sarvam — Indian languages via Sarvam's bulbul voices), and the wire (Twilio for phone calls, LiveKit for in-app voice).
The agent underneath is the same one you built for chat — same knowledge, same guardrails — so voice is a channel decision, not a rebuild.
Step by step
Create the agent
Build the agent as you would for any channel — describe the job in the builder, add knowledge, test in the playground. Voice sits on top of an agent that already works in text, so get the answers right first.
Choose speech-to-text
Pick how the agent listens: locally-run Whisper, or Sarvam's speech-to-text. Sarvam is the natural choice when callers will speak Indian languages; Whisper runs on the platform without an external provider.
Choose text-to-speech
Pick the agent's voice: ElevenLabs, or Sarvam — whose bulbul voices cover Indian languages. Connect the provider's key on the Providers screen if you have not already, then select the voice in the agent's voice settings.
For phone: connect Twilio voice
To put the agent on a phone number, connect Twilio voice in the channel settings. The agent then takes inbound calls, and DTMF menus are supported — callers can press keys to navigate, not just speak.
For browser or in-app voice: use LiveKit rooms
For voice inside a web page or app, the platform uses LiveKit rooms. A caller joins from the browser — no phone number involved — and recording is optional per room if you need a record of the conversation.
Test the call and check guardrails
Place a real test call, on the phone line or in the browser, and listen end to end: transcription accuracy, voice quality, latency. Guardrails apply to voice exactly as to any channel, so actions you gated behind approval still pause for a human even mid-call.
Troubleshooting
Voice adds moving parts; isolate which one is misbehaving before changing settings.
- Poor transcription of Indian languages: switch speech-to-text to Sarvam, which is built for them.
- No audio on the agent's replies: confirm the text-to-speech provider's key is valid on the Providers screen and a voice is selected.
- Phone calls not connecting: verify the Twilio voice connection and the number's configuration on the Twilio side.
Frequently asked questions
Can an AI agent answer real phone calls?
Yes. Connect Twilio voice and the agent takes inbound calls on a real phone number, with DTMF menus supported so callers can press keys as well as speak. Browser and in-app voice runs separately over LiveKit rooms, with no phone number needed.
Does the voice agent support Indian languages?
Yes, through Sarvam: its speech-to-text handles Indian-language callers and its bulbul voices speak Indian languages for replies. For other needs, locally-run Whisper covers transcription and ElevenLabs covers text-to-speech.
Can voice calls be recorded?
In-app and browser conversations run over LiveKit rooms, where recording is optional and set per room. Enable it when you need a record of the conversation; leave it off otherwise.
Do guardrails still apply when the agent is on a call?
Yes. Voice is a channel like any other, so the agent's guardrails and approval rules apply mid-call exactly as they do in chat. An action gated behind human approval still pauses and waits, even while a caller is on the line.
See it working in two minutes
The sandbox provisions a real tenant — describe an agent in one sentence and test it, no account, no card. Or browse ~90 industry workflow recipes to see what teams build.
