People forgive a chat window for taking three seconds. They do not forgive a phone call. On a call, a pause that long means the line has dropped, and an agent that talks over you is worse than one that never answers.
We have spent much of this year putting Camaan agents on telephone lines. Here is what we learned about the two things that turned out to matter most: taking turns, and rules.
The hardest question: have you finished?
Text arrives finished. Speech does not. A caller reading out an account number pauses in the middle. A caller thinking about a question pauses before answering it. The agent has to tell those pauses from the end of a sentence, within a few hundred milliseconds, and it will sometimes be wrong.
We use two layers. The simple one ends a turn after 700 milliseconds of silence. The smarter one runs a small model that listens to how the speech ended and asks: finished, or pausing? When it is confident, the agent can answer sooner. When it is not, the agent waits.
Most of what we know came from calls that went badly.
- The agent went deaf. The noise gate adapts to the room. It also adapted to the agent's own voice echoing back down the line, and within one reply it had raised the bar higher than the caller could speak. We capped it.
- It answered half a sentence. The model was very sure about short fragments, and twice it replied a quarter of a second into the caller's pause. Now, under a second of speech, the model may not end a turn any earlier than plain silence would.
- It split an account number in two. A transcript that ends on a digit, or without closing punctuation, now waits a little longer, and what the caller says next is joined to it.
- "Are you still there?" talked over the caller. The idle timer started when the reply was written, not when it had finished playing. A long answer plus a thoughtful pause set it off mid-question.
None of these showed up in a demo. All of them showed up within days of real calls.
The same rules in the browser and on the line
The same agent can be spoken to from a web page. At first the browser and the phone gateway each had their own copy of the timing rules, and two of the values drifted apart without anyone noticing. We had to make the same fix three times.
The timings now live in one place, and a shared set of test cases runs against both the phone and the browser. If they ever disagree, a test fails.
Where the time goes
When we broke a turn down, the result was not what the team expected. Audio handling was a small share. The language model's first word dominated. For a voice line, the choice of model matters more than any tuning of the media path, and the fastest reply is the one that needs no model at all: a common question matched with confidence to a scripted answer.
On capacity, one mid-range GPU serves about four simultaneous calls on our speech stack, at a median of 1.1 seconds to first audio. The speech hosts scale on how busy they are and on each customer's calling hours, and outside those hours they scale to nothing.
Who is allowed to dial
An agent that can place calls is an agent that can annoy your customers at scale. So the rules sit in the engine, not in the telephony layer, where nothing can route around them.
- The owner of a line decides. A call the owner places goes straight to the queue. A call a colleague asks for waits for the owner's approval.
- Do-not-call is checked three times. When the call is requested, when it is approved, and again at the moment of dialling. A number added to the list after approval is still never rung.
- A line has a limit. If a line allows one call at a time and four requests race for it, exactly one wins.
One rule had to bend. Our agents normally stop and ask before a sensitive action. On a phone nobody can click "Allow", and an early test call sat in silence for ten minutes waiting for someone to try. Hanging up, collecting keypad digits and transferring to a person are now exempt from that prompt. Nothing else is.
What we would tell anyone starting
Test with real callers early, on a real line, with a bad connection. Measure a whole turn, not the parts. And decide who may dial, and whom they may never dial, before the agent makes its first call.