Skip to main content
When your agents handle hundreds or thousands of conversations, you can’t read them all to know whether they’re doing a good job. Spot-checking a handful by hand is slow, inconsistent, and impossible to repeat. Evaluation solves that: it measures the quality of your agent conversations consistently and at scale, so you can answer questions like “Did the agent resolve the caller’s task?” or “Did it ever give out wrong information?” across every conversation, not just the few you happened to open. You measure quality two ways, and you can combine them:
  • An AI judge — a top-tier large language model that reads a conversation and answers your quality questions automatically.
  • Human reviewers — people on your team who grade conversations themselves.
Both produce the same kind of result, so you can compare them directly. That comparison is what eventually lets you trust the AI judge to score on its own.

​
Where it lives

Everything below lives in the Analytics section of the dashboard. Inside Analytics you’ll find these tabs:

​
The mental model

The most important thing to learn first is the hierarchy — how the pieces nest. Read this once and the rest of the section will make sense.
Here’s each term, defined the first time you’ll meet it:
A metric expecting false flips the intuition: “false is good.” Throughout the dashboard, the compliant rate is what matters, not the raw share of true answers. Compliant rate = compliant ÷ answered, where answered counts only true and false verdicts — abstain and na are left out of the denominator.

​
AI judge vs. human reviewers

The same conversation can be scored by the AI judge and by a person, and each scoring is its own run. That’s deliberate: when you have an AI run and a human run for the same conversations, you can measure how often they agree. High agreement is your evidence that the AI judge is trustworthy enough to score on its own. Low agreement tells you the question needs sharpening or the judge needs more context. Human grading is designed to be fast. It works “grade-by-exception” — every criterion starts on the no-violation (compliant) answer, and you only flag the exceptions. A human run can also confirm or correct an existing AI run.

​
The end-to-end lifecycle

Putting it together, here’s the path from nothing to trusted, automatic scoring:
1

Define metrics

In Metrics, create the metrics and criteria that capture what “good” means for your conversations. See Creating metrics & criteria.
2

Bundle into an evaluation

In Evaluations, group the metrics you want to score together and add judge instructions (often your agent’s knowledge base, so the judge knows what “correct” means).
3

Run it

From Runs, trigger the evaluation against one or many conversations, a pasted transcript, or an audio URL. Each conversation becomes its own run, processed in parallel.
4

Read the results

In Overview, read compliant rates per metric and drill into the individual verdicts behind any number.
5

Calibrate

In Calibration, compare AI and human runs to ask the real question: is the AI judge trustworthy for this metric?
6

Graduate

Once a metric’s agreement clears the bar, graduate it to turn on automatic AI scoring — explicitly, with a click.
You don’t have to do all six steps before you get value. Defining one metric and running it against a handful of conversations already tells you something. Calibration and graduation are how you scale that up with confidence.

​
Start here

Quickstart

The fastest path: create a metric, run an evaluation, and read your first results.

Core concepts

A deeper tour of metrics, criteria, outcomes, runs, and verdicts.

Creating metrics & criteria

Write good yes/no criteria and choose the right expected value.