How to build an AI calling agent (and when to buy one instead).
Short answer: an AI calling agent is a real-time pipeline. A telephony provider connects the phone call; speech-to-text turns the caller’s audio into text as they speak; a language model decides what to say and which tools to call (your CRM, loan system, payment links); text-to-speech speaks the reply; and everything streams so each turn feels instant. A weekend prototype is very doable. Making it reliable, fast on Indian mobile networks, good at Hinglish and safe to run for a regulated business is the real work.
The architecture
| Layer | What it does | What to watch |
|---|---|---|
| Telephony | Places and receives calls (SIP trunk or a cloud telephony API) and streams audio both ways | Indian numbers, 140/1600 series rules, call quality, recording |
| Voice activity detection | Detects when the caller starts and stops speaking | Too eager and it cuts people off; too slow and it feels laggy |
| Speech-to-text (ASR) | Streams transcription in real time | Code-mixed Hindi-English, accents, numbers, 8 kHz phone audio |
| LLM + dialogue logic | Understands intent, keeps state, decides the next step | Staying on script, not inventing facts, short spoken replies |
| Tools | Calls your systems: look up dues, record promise-to-pay, send links | Authentication, timeouts, safe write access |
| Text-to-speech (TTS) | Speaks the reply, streaming the first words fast | Natural Indian pronunciation of English words and numbers |
| Orchestration | Ties it together: streaming, barge-in, handover, logging | Latency, failures, concurrency |
Open-source frameworks such as Pipecat and LiveKit Agents handle much of the orchestration, so you can plug in the ASR, LLM and TTS providers you choose.
The latency budget
People expect a reply within roughly a second of finishing a sentence. Every layer eats into that: end-of-speech detection, the final transcript, the model’s first token, the first audio from TTS, and network hops in between. What helps:
- Stream everything. Start TTS on the model’s first sentence, not the full reply.
- Keep replies short. Spoken answers should be one or two sentences.
- Host close to the caller. Servers in India cut network time for Indian calls.
- Use fillers carefully (“ek second, check kar rahi hoon”) while a slow tool call runs.
Interruptions (barge-in)
Real callers talk over the agent. The agent must stop speaking the moment the caller does, keep track of what was actually heard, and respond to the interruption. It is one of the biggest differences between a demo and a usable agent.
Indian-language pitfalls
- Code-mixing: pick ASR that is tested on Hinglish phone audio, and test it on your own calls.
- Numbers and dates: “paanch hazaar”, “5,000” and “five thousand” must normalise to the same value before any tool call.
- Pronunciation: TTS must say EMI, KYC, NACH and brand names the way Indians do.
- Script: decide whether the model sees Devanagari, Roman or both, and be consistent.
More on this in Hindi AI voice agents.
Guardrails you need before real customers
- Grounding: the agent quotes amounts and dates only from your systems, never from the model’s guess.
- Scope: it refuses or hands over anything outside its job.
- Handover: a person on request, with the transcript.
- Disclosure: say it’s an AI assistant; RBI’s FREE-AI framework recommends it for financial services.
- Calling rules: number series, consent and DND under TRAI, and for lenders, RBI calling hours and call recording. See the rules for AI calls in India.
- Data: where audio and transcripts are stored, who can access them, and how long you keep them.
Build or buy?
| Build if… | Buy if… |
|---|---|
| Voice is your product, or you have a strong ML and telephony team | Voice is a channel, and you want outcomes this quarter |
| Your use case is unusual and off-the-shelf tools don’t fit | Your calls are standard: reminders, collections, service, onboarding |
| You can own uptime, scaling and on-call | You need audit trails, recording, retention and regulator-ready logs from day one |
Many teams prototype to learn, then buy for production. For banks, NBFCs and insurers, the compliance and integration work is usually larger than the AI work.
Ziya is ZyroAI’s production voice agent for BFSI: streaming ASR and TTS, barge-in, 22+ Indian languages, tools into lending and banking systems, India-hosted, with every call sent to speech analytics. Start with how AI calling agents work if you’re new to the topic.
Quick answers
How do you build an AI calling agent?
Connect a telephony provider to a real-time pipeline: streaming speech-to-text, a language model with tools that call your systems, and streaming text-to-speech, plus voice activity detection, barge-in handling, handover to humans and logging.
What tech stack is used for AI voice agents?
Telephony (SIP or a cloud telephony API), streaming ASR, an LLM with function calling, streaming TTS, and an orchestration layer. Open-source frameworks such as Pipecat and LiveKit Agents handle much of the orchestration.
Is there an open-source AI calling agent?
There are open-source frameworks for building one, such as Pipecat and LiveKit Agents. You still choose and pay for telephony, speech and model providers, and own reliability and compliance.
Should I build or buy an AI calling agent?
Build if voice is your core product and you have the team to run it. Buy if your calls are standard (reminders, collections, service) and you need compliance, audit trails and integrations from day one.