)
Voice Agent Evaluation:
from simulated calls to continuous control
A dedicated vertical of the AI Evaluation Lab for testing, scoring and monitoring conversational voice agents throughout their lifecycle, so that every release is measured before it ships and every live call stays under control after it does.
The reliability gap in voice AI
Organisations are putting voice agents on the front line of customer contact: answering service calls, booking appointments, handling outages, qualifying leads. Building one has become fast. Proving that it still works after the next change has not.
Like every LLM-based system, a voice agent is non-deterministic: the same request can lead to different outcomes on two consecutive calls. There is no expected output to compare against, so quality has to be scored rather than asserted. A single prompt edit, model swap or new tool can change behaviour across dozens of conversations, silently and in production.
Many voice programmes still rely on a handful of manual test calls before launch and on contact-centre dashboards afterwards.
The result is an agent that is deployed, but not measured.
What a transcript cannot show
Voice adds failure modes that text-based evaluation never sees: latency that makes a caller repeat themselves, an agent that talks over the customer or fails to recover after an interruption, speech that is clipped, rushed or unnatural, a mistranscribed account number, a call that simply drops.
None of these appear in a transcript that reads perfectly. Evaluating voice agents therefore means looking at what was said, how it sounded and how the exchange flowed, and keeping those judgements separate, so that a good conversation on paper cannot hide a poor experience on the line.
One quality model, two moments
The Lab answers two different questions with a single definition of quality. Before release, the question is whether the agent can ship. After deployment, it becomes whether that answer is still true weeks later, under real traffic.
Because test and production share the same indicators, a result measured in the Lab and a signal observed on live calls can be compared directly, and what is learned in operation flows back into the next test cycle.
Test, before release
)
Can we ship it?
Controlled scenarios, written in advance and re-run unchanged
Simulated callers, so every test can be repeated tomorrow
A verdict on each conversation and on the run as a whole
A breached threshold blocks the release
Observe, after deployment
)
Is it still true a month later?
Real callers and real traffic, with no expected answer
The same indicators, measured continuously instead of once
A breached threshold raises an alert instead of failing a build
Slow decline surfaces as a trend, before it becomes a ticket
The same metric definitions drive both phases.
That is what turns two tools into one method.
Simulated callers, real conversations
Every test starts from a written expectation: a scenario with a goal, a caller persona and explicit success criteria. Scenarios can be generated from the agent's own instructions, which makes the method portable to any voice agent, and are complemented by a curated regression set that runs unchanged in every cycle. Coverage spans happy paths, edge cases, adversarial attempts and compliance-sensitive situations.
The caller is played by an AI with its own persona, tone and synthetic voice: impatient, insistent, interrupting, off-script. It does not follow a script; it reacts to what the agent actually says and keeps pushing towards its goal. Conversations run turn by turn against the real agent, by voice or by text, with the agent using its real tools and back-end systems, so every action it takes is recorded and verified.
Impatient
Insistent
Interrupting
Off-script
Short bursts, cuts the agent short
Repeats the same demand after every answer
Talks over the agent mid-sentence
Long requests outside the agent's scope
Transcript judge
)
What was said
A judge model reads the full conversation against the scenario's own success criteria: was the task completed, were instructions followed, were tools used correctly, was anything invented or any compliance line crossed.
Audio judge
)
How it sounded
A multimodal model listens to the agent's spoken turns and assesses naturalness, tone, clarity and audio artefacts that no transcript can reveal.
Conversation dynamics
)
How the exchange flowed
Signals computed directly from timing and turns, with no model involved: response latency, speaking pace, silences, overlaps and recovery after an interruption.
From scores to a release gate
Measurement becomes control when it is tied to a decision. Each key indicator carries a regression threshold: every candidate release is compared with the current baseline, and a breach stops the change from shipping. The gate runs inside the delivery pipeline on a reproducible setup, and every run leaves a complete evidence trail. A guardrail accidentally dropped from a prompt is caught by the pipeline, not by a customer.
Continuous control on live calls
Once real callers arrive, the same indicators keep working. Production calls are traced, so a complaint can be replayed rather than guessed at; release thresholds become alerts; trends across live traffic and scheduled re-runs expose drift weeks before it reaches a ticket queue. Every new failure seen on a live call becomes a new test scenario, so the same defect cannot return unnoticed.
One indicator across successive runs on live traffic: stable, then slipping slowly until the alert threshold is crossed.
Customer-experience indicators
Satisfaction and experience measures for CX programmes.
Compliance panel
Disclosures, identity verification, consent and recording notices for regulated domains.
Adversarial red-teaming
Prompt injection, data leakage and off-script behaviour.
Cross-platform benchmarking
The same scenarios run against several voice providers, to choose the right one.
Evidence that stands up to scrutiny
As voice agents take on customer-facing and regulated interactions, quality has to be demonstrated, not claimed. A documented record of expectations, executions and results for every release supports governance and accountability, including the transparency and risk-management expectations introduced by the EU AI Act, such as making clear to callers that they are speaking with an AI system. Quality engineers, AI specialists, contact-centre owners and compliance functions work from the same shared record of how the agent actually behaves.
Explore Reply's approach to voice agent evaluation
The voice vertical of Concept Quality Reply's AI Evaluation Lab supports organisations that need to put conversational agents in front of customers with measurable quality and continuous control. It works with commercial voice platforms and custom-built agents alike, and connects simulation, multi-lens scoring, release gating and production observability into a single operating model.
The result is a voice agent that is not only deployed, but measured: tested before every release, and watched for as long as it is answering calls.
READY TO PROTECT WHAT POWERS YOUR CUSTOMER CALLS?
Test it before release. Watch it after deployment.
Concept Reply
Concept Reply, part of the Reply Group, specialises in IoT hardware and software innovation, delivering solutions in Smart Infrastructure, Industrial IoT, and Connected Vehicle sectors. Its services encompass the entire development process, from concept and design through to implementation and operational support, with expertise in hardware design, embedded software, edge computing, and cloud applications. In addition to IoT innovation, Concept Reply is recognized as a leader in Testing and Quality Assurance (QA), offering comprehensive product and service validation. Supported by advanced laboratories and a global team of QA specialists, it enables businesses to optimise costs and reduce time-to-market with reliable, high-quality solutions.