Lessons from my first AI phone agent project

Since spring, I’ve been running my first AI phone agent project in live operation. The client is an owner-run specialty business with consultation-heavy, high-ticket products. The customer base is mostly older. In other words, exactly the audience everyone claims will never talk to an AI. They do anyway. The assistant answers around the clock, handles product questions, books appointments, takes down service requests, and creates callback records in the system.

Along the way, I documented roughly 40 different types of errors — most of them in the test phase, where they belong. The remarkable part: not a single one was “the AI is too dumb.” Almost all were architecture mistakes. In a few places, I had trusted the AI with something it simply cannot do by design.

With this post, I’m starting a series about these experiences. No theory, no promises — real cases, with cause and fix. First up is the error type I have the greatest respect for today: fabricated data.

Case 1: The AI invents a phone number — because of a placeholder

A test call with a withheld caller ID. The assistant doesn’t ask for a callback number — it creates a record, complete with a full, plausible German mobile number. The number exists. It just doesn’t belong to the caller.

The cause was a single misspelled placeholder in the configuration. The system couldn’t substitute it. So the AI effectively saw the text “the number goes here” — and did what an AI does: it filled the gap. Convincingly.

That was the first lesson: an AI has no concept of “missing.” It only knows “I’ll phrase something that fits.” Every gap you leave open will be filled. An empty field you would notice — a fabricated number, nobody notices.

Case 2: A fabricated number hits real customer data — the privacy risk

How serious this can get also showed up, thankfully, in the test phase — before real customers spoke with the system. On calls placed through the website, no caller ID is transmitted at all. Here too, the assistant invented a number. This time, it actually existed in the customer database: the assistant promptly mistook the test caller for a different, real customer — and addressed him by that customer’s name. In a database of 1,600 people, a “plausible” number is no harmless coincidence. The probability of a real match is measurable.

The case had no consequences because it happened inside the protected test environment. That’s exactly what a test phase is for. But it changed how I look at the risk. A hallucination isn’t a quality problem — it’s a data-privacy risk the moment fabricated data reaches into a real customer system. If you’re deploying or buying a phone AI, ask your provider exactly this question: “What happens on your end when the AI makes up a number?” Anyone without a technical answer hasn’t understood the problem yet. I’ve covered the data-privacy basics in a separate post.

Case 3: Withheld caller ID — why the check still failed

The fix for withheld caller IDs was built, tested, all green. Two days later, it broke again in the next test run anyway. The assistant claimed it could see the number. The test caller called it out immediately: “It’s withheld — how would you know it?”

The cause was a technical edge case. A withheld caller ID doesn’t always look the same in the system — depending on the network and call path, there are several variants. Our check knew only two of them. The third slipped through and was treated as a valid number. The lookup on it happened to hit an unrelated person in the database.

The lesson: edge cases like this aren’t in any manual. You only find them through thorough testing. A provider who has never encountered this will discover it with your real customers. So in your conversations, feel free to ask: “What does the system do when someone calls with a withheld number?” The answer quickly tells you how much real-world practice is behind it. Since then, we run on one simple rule: only a complete, real number counts as a number. Everything else is treated as if there were none. The error has never recurred.

Case 4: The AI invents product and person names in CRM fields

Numbers aren’t the only thing that gets fabricated. One caller wanted to book a service appointment and never said which product it was about. The callback ticket afterwards nonetheless contained a specific product name — plausibly picked from the knowledge base. Another time, the assistant asked “Am I speaking with Mr. or Ms. …?” without any lookup ever having run. And from one mumbled word, it produced a last name that never existed.

The pattern is always the same: an AI hates leaving things open. Where there’s an empty field, it fills it — whether the content is right or not. A good system does the opposite: it prefers to leave empty what the caller didn’t say.

The fix was an explicit fabrication ban — not just for what’s spoken, but for what’s written into fields. No product name and no person’s name goes into a record unless the caller said it themselves. And any name that was actively asked for is repeated back and confirmed once before the record is created.

Case 5: Confirmed in the conversation, saved wrong in the CRM

The subtlest case for last. A caller spells out his last name, letter by letter. The assistant reads it back, corrects itself, everything sounds right in the conversation. In the email address, the name is spelled correctly afterwards. In the name field, there’s a different spelling.

What happened? The correction was applied only to the field that was currently “in play” — not to every field holding the same value. To the AI, the correction was just a moment in the conversation — not a change to the data.

That became my most important acceptance rule: never check the conversation. Check the record. I recommend this to anyone signing off on a system like this. In every review, I look at the database first: what was actually saved? How good the conversation sounded comes second.

What helps against fabricated data: Structure, not instructions

The obvious response to all these cases would be: write stricter instructions. “Never fabricate data” — bold, at the very top. I tried that. It helps — but it doesn’t hold reliably. An AI follows instructions with probability, not with guarantees.

What really helped were technical safeguards — things a good provider builds in on their own initiative. Three examples from our project:

The real caller number goes directly from the phone system into the backend — bypassing the AI. A fabricated number is thereby powerless.
A validation step between the AI and the customer database refuses to create a record without a complete, real number — politely, with a path back into the conversation to capture the number.
Fields whose content isn’t backed by anything stay empty. Empty is visible. Fabricated is invisible.

You don’t need to understand how these safeguards are built. But you’re entitled to expect that they exist — and to ask about them.

Why instructions alone aren’t enough is a topic of its own. I’ll dedicate the next post in this series to it.

My take

Fabricated data is not a sign that AI phone assistants don’t work. The assistant this post is about runs reliably today — precisely because these cases were found, understood, and fixed. But these five cases made one thing clear to me: the AI is the part that talks — not the part you entrust with your customer data. Before anything lands in your database, it has to pass through a technical check. Relying on the AI to simply follow its instructions is not enough.

If you ever have a phone agent set up for your business, bring one question into the first conversation with the provider: what ends up in my customer data when a caller doesn’t give their number? Have them show you the answer, not just explain it — ideally in the test phase, before the system talks to real customers. It’s exactly at this inconspicuous spot that a pretty demo parts ways with a system you can trust with your own customer base.