AI phone agent: how to build one that handles real calls (2026 guide)
What an AI phone agent is, the six parts every working one needs, a step-by-step build, a prompt template, real per-minute costs and the failure modes to test.

An AI phone agent is the piece of software that picks up when someone calls your business, or dials out when you need to reach someone, and then actually talks. Not a menu. Not a recording. A conversation that ends with the caller's question answered, a booking made, or the call handed to the right person.
The idea is simple. Building one that holds up on real calls is where most teams get stuck. This guide is the practical version: what the parts are, how to put them together step by step, what it costs per minute, and the specific things that go wrong in the first week.
An AI phone agent needs six things: a phone number, a voice pipeline, a prompt, your knowledge, tools, and a way to get the results out. On Call2Me the wizard sets up the first four in a few minutes, and you start with $5 in free credit after verifying your email, no card needed.
What an AI phone agent is (and what it is not)
The phrase gets used loosely, so it helps to draw the lines.
| What happens when someone calls | Who resolves the call | |
|---|---|---|
| Voicemail | Caller leaves a message, most hang up | You, later, if you call back |
| IVR menu | Caller presses keys to pick a department | Usually a human after the menu |
| Chatbot | Text only, on a website or app | The bot, if the question is simple |
| AI phone agent | Caller speaks normally, the agent replies by voice | The agent for routine calls, a human for the rest |
The last row is the point. An AI phone agent is not trying to route the call somewhere else. It is trying to finish it. When it cannot, it transfers with context instead of leaving the caller to start over.
You will also see the same thing called an AI call assistant, an AI receptionist or a voice agent. The labels overlap. This guide focuses on the build: how the parts fit together so the agent works on a real phone line.
The six parts of a working AI phone agent
Every AI phone agent, on any platform, is made of the same six parts. If one is missing or weak, the whole thing feels broken to the caller.
1. A phone number
The agent needs a number people can dial, or a number it can call from. There are two routes:
- Buy a number from the platform. On Call2Me this takes a few seconds in the dashboard; numbers are available in 21 countries at the time of writing (US and Canada through Telnyx, the rest through Twilio).
- Bring your own carrier over a SIP trunk. You keep your number, your carrier contract and your PBX, and point the traffic at the agent. Our SIP trunk guide walks through it.
2. The voice pipeline
Three models run in a loop on every turn of the conversation:
- Speech-to-text turns the caller's words into text as they speak.
- A language model reads the conversation so far, plus your instructions, and decides what to say or which tool to use.
- Text-to-speech turns the reply into audio and streams it back.
On Call2Me the standard pipeline is Deepgram Nova-3 for speech-to-text, a language model of your choice through OpenRouter, and ElevenLabs Flash v2.5 for the voice. There is also a realtime mode that uses a speech-to-speech model instead of three separate steps. It sounds more fluid and costs an extra $0.25 per minute.
Speed matters more than anything else here. A long pause after the caller stops talking is the clearest sign they are talking to a machine. We wrote about why in the 700ms wall.
3. The prompt
The system prompt is the job description. It tells the agent who it is, what it is allowed to do, what it must never do, and when to hand off. A good voice prompt is short, written for speech rather than for a chat window, and specific about edge cases. Voice prompts are different from chat prompts in ways that matter: no bullet lists read aloud, no long numbers in one breath, one question at a time.
4. Your knowledge
The agent can only answer from what it knows. Hours, prices, service area, policies and FAQs belong in a knowledge base, not stuffed into the prompt. On Call2Me you upload PDF, DOCX, TXT or Markdown files, or point it at a URL, and the agent retrieves the right passage during the call. Retrieval is included in the per-minute rate. More on how this works in from PDF to live phone call.
One rule worth repeating: data that changes every hour (stock levels, today's availability, a customer's order) should not go in the knowledge base. Fetch it live with a tool instead.
5. Tools
Tools are what turn a talking FAQ into an agent that gets things done. On Call2Me there are two kinds:
Built-in functions that need no code:
| Function | What it does on the call |
|---|---|
transfer_call | Hands the caller to a department, a number or another agent |
hold_call | Puts the caller on hold with music while something is checked |
dtmf_input | Collects keypad digits: a PIN, an extension, a menu choice |
end_call | Says a closing line and hangs up |
Custom functions that call your own API mid-call: check an order, look up a booking slot, create a ticket. You define a name, a description and the parameters; the model decides when to call it. Keep these fast. Under 500ms is ideal, and the agent should say something like "let me check that" while it waits. The platform gives up at 8 seconds and hands the model an error, so the prompt should say what to do if a tool fails.
6. A way to get the results out
A call that ends without a record is a call you cannot act on. Each Call2Me
call gets a transcript, an optional recording, and post-call extraction:
you list the fields you want (name, callback number, reason for calling,
interest level) and a model fills them in from the transcript after the call.
Agent webhooks then push call_started, call_ended and call_analyzed
events to your server, so the data can land in your CRM or a spreadsheet.
How to build an AI phone agent, step by step
Here is the order we recommend. It is the same order the problems show up in.
Step 1: Pick one job
Do not start with "answer all our calls". Start with one call type that is frequent, routine and easy to judge: booking appointments, answering after-hours questions, qualifying a new enquiry. You can widen the scope once that one job works.
Step 2: Write the prompt for that job
A structure that works for most phone jobs:
You are the phone assistant for [Business], a [type of business] in [city].
Your job on this call: [the one job].
How to speak:
- Short sentences. One question at a time.
- Read times and prices the way a person would say them.
- If you do not know, say so and offer to take a message.
What you can do:
- Answer questions using the knowledge base.
- Book appointments with the book_slot tool. Confirm day and time back
to the caller before booking.
- Transfer to the front desk with transfer_call if the caller asks for a
person, is upset, or the request is outside this list.
What you must never do:
- Quote prices that are not in the knowledge base.
- Give medical, legal or financial advice.
- Take card numbers over the phone.
Before ending the call, repeat what happens next and ask if there is
anything else.
The "must never" list is the part people skip, and it is the part that saves you from the worst calls.
Step 3: Load your knowledge
Upload the documents a new employee would read on their first day. Then test a few questions you know the answer to. If the agent gets one wrong, the fix is usually in the document (unclear wording, two versions of the same fact), not in the prompt.
Step 4: Choose the voice and language
Call2Me has 26 voices: 16 ElevenLabs voices for standard mode (13 multilingual, 3 native Turkish) and 10 realtime voices. Every voice has a preview button. Pick the one that matches how your business sounds on the phone, and test it in the language your callers actually speak. A voice that sounds great in English can sound flat in another language. Setting the agent to Multilingual lets the same voice follow the caller's language.
Step 5: Add tools
Turn on transfer_call and end_call for every agent. Add dtmf_input if
callers need to enter account numbers or PINs; keypads are still more
reliable than speech for long digit strings. Add custom functions only for
the actions the one job needs.
Step 6: Connect a number
Buy one, or connect your existing line over SIP. For a first test, many businesses forward their main line to the agent only outside business hours. That way nothing changes during the day while you learn.
Step 7: Decide what happens after the call
Define the extraction fields you want on every call, and point a webhook at your system if you have one. If you do not, the Calls tab shows every transcript, recording and extracted field, and exports to CSV.
Step 8: Run twenty test calls before real callers do
Call it yourself and from friends' phones. Try the easy path, then try to break it: interrupt it mid-sentence, ask something off-topic, mumble a name, call from a noisy street, ask for a human. Read every transcript. Fix the prompt, then call again.
Inbound and outbound are different jobs
The same agent can do both, but the design is different.
| Inbound | Outbound | |
|---|---|---|
| Who starts it | The caller | You, from the dashboard, the API or a schedule |
| What the caller expects | Help right now | Nothing, they did not plan this call |
| Opening line | Greeting plus "how can I help?" | Who is calling, why, and permission to continue |
| Typical jobs | Answering, booking, routing, after-hours | Confirmations, reminders, follow-ups the customer asked for |
| Must handle | Any question in scope | Voicemail, wrong person, "not a good time" |
| Price on Call2Me | $0.11/min | $0.14/min |
For outbound, voicemail detection matters. It is off by default; turn it on per agent and choose whether to hang up or leave a short scripted message.
What an AI phone agent costs
Call2Me pricing is per minute with no subscription. These are the live rates from our public pricing endpoint:
| Item | Rate |
|---|---|
| Voice pipeline (STT, LLM, TTS, knowledge base) | $0.10/min |
| Phone line | +$0.01/min |
| Outbound calls | +$0.03/min |
| Recording (optional) | +$0.005/min |
| Realtime voice mode (optional) | +$0.25/min |
| Phone number | Monthly rental, depends on country, shown before you buy |
Two worked examples, using made-up volumes so you can swap in your own:
- Inbound reception: 400 calls a month at an average of 2.5 minutes is 1,000 minutes. At $0.11 that is $110, plus the number rental. With recording on, add $5.
- Outbound reminders: 300 calls at an average of 1.5 minutes is 450 minutes. At $0.14 that is $63.
Ringing, busy and no-answer attempts are not billed. If you want to compare that with what missed calls cost you, our missed call revenue calculator does the other half of the maths. The full rate card is on the pricing page.
Failure modes to test in the first week
These are the problems we see most often on new agents, and where to look.
| What you notice | Usual cause | Fix |
|---|---|---|
| Long silence before each reply | Slow model, slow custom function | Use a faster model; add a filler line before tool calls |
| Agent talks over the caller | Interruption setting too low | Raise interruption sensitivity |
| Confident wrong answers | Fact missing or contradicted in documents | Fix the source document, not the prompt |
| Reads lists and URLs aloud | Prompt written for chat | Rewrite for speech, one item at a time |
| Never transfers | No clear transfer rule | Spell out the exact triggers in the prompt |
| Transfers too often | Rule too broad | Narrow the triggers, add answers to the knowledge base |
| Garbles names and emails | Speech recognition on rare words | Ask callers to spell, confirm back, or use keypad input |
When an AI phone agent is the wrong tool
Be honest about scope. An AI phone agent is a poor fit for:
- Long negotiations or emotional disputes. A person reads tone better.
- Regulated advice. Medical, legal and financial advice should come from a qualified human.
- Taking card payments by voice. Use a payment link or a human with a compliant process.
- Very low call volume where every caller is a VIP. The economics favour a person who knows each client.
For everything routine, frequent and well defined, it works well. Most businesses end up with the agent handling the routine majority and their team handling the rest, with fewer interruptions.
Pre-launch checklist
- One job, written down in one sentence
- Prompt with a "must never" list and clear transfer triggers
- Knowledge base with current hours, prices and policies
- Voice tested in every language your callers use
-
transfer_callandend_callenabled, destinations tested - Post-call fields defined, webhook tested or CSV export checked
- Twenty test calls read end to end
- A plan for who reads transcripts during the first two weeks
Try it on your own number
The fastest way to know if an AI phone agent fits your business is to build one and call it. Set it up for one job, give it your real documents, and ring it from your mobile.
Build your AI phone agent free →
Read next
- AI call assistant: how it works, cost, features: the feature checklist view of the same technology.
- Voice vs chat prompts: 7 patterns that work: how to write the prompt so it sounds right out loud.
- From PDF to live phone call: how the agent answers from your own documents.
- Connect a SIP trunk to an AI voice agent: keep your number and carrier.
- IVR replacement with AI: if your phone line already has a menu tree.
Frequently asked
Q.What is an AI phone agent?
An AI phone agent is software that answers or places phone calls and holds a real spoken conversation. It listens with speech-to-text, decides what to say with a language model, speaks with text-to-speech, and can use tools during the call: look something up, book a slot, transfer to a person or hang up. Unlike an IVR menu it does not ask callers to press keys; unlike voicemail it resolves the call while the caller is still on the line.
Q.How long does it take to build an AI phone agent?
A first working agent on a real number takes minutes on a hosted platform: sign up, run the setup wizard, assign a number and call it. Making it good takes longer. Plan an afternoon to write the prompt properly, load your documents and run about twenty test calls, then a week or two of reading real transcripts and tightening the prompt.
Q.How much does an AI phone agent cost per minute?
On Call2Me the voice pipeline (speech-to-text, language model, text-to-speech and knowledge base) is $0.10 per minute. A phone line adds $0.01 per minute, so an answered inbound call is $0.11 per minute. Outbound calls add another $0.03, for $0.14 per minute. Recording is optional at $0.005 per minute, and the phone number itself has a monthly rental that depends on the country. Unanswered and ringing time is not billed.
Q.Can an AI phone agent transfer a call to a human?
Yes, and it should be able to. On Call2Me transfer_call is a built-in function: you configure the destination numbers, SIP endpoints or other agents, and the system prompt tells the agent when to use them. The agent can also put the caller on hold, collect keypad digits and end the call cleanly.
Q.Can I keep my existing phone number?
Yes. You can buy a new number in the dashboard (21 countries at the time of writing) or keep your current number with your current carrier and connect it over a SIP trunk. SIP works with any provider that speaks SIP over UDP, TCP or TLS.
Keep reading
All posts
IVRIVR replacement with AI: a step-by-step migration guide
How to replace an IVR phone menu with an AI voice agent without breaking your phone line: three migration patterns, an intent map, SIP setup, a pilot plan and rollback.
Nov 17, 202610 min
AI Call AssistantAI Call Assistant: How It Works, Cost, Features
What an AI call assistant does — answering and placing calls, booking, routing, 24/7 — what it costs, the features to look for, and where it falls short.
Jun 8, 20268 min
AI Answering ServiceAI Phone Answering Service: the complete 2026 guide
An AI phone answering service picks up every call 24/7, answers questions, and books appointments for cents per minute. Here is how it works, costs, and limits.
Jun 8, 20269 min