How do we create custom evaluation metrics in COVAL (disclosures, escalation handling, KB accuracy, resolution rate)?

Most teams hit a wall when they realize off-the-shelf metrics aren’t enough to govern a production voice agent. You care about disclosures, escalation handling, knowledge base accuracy, and resolution rate because those are the things that actually move risk and revenue. COVAL is built so you can turn those into first-class, custom evaluation metrics and apply them consistently across both simulations and live calls.

Quick Answer: You create custom evaluation metrics in COVAL by defining LLM-as-a-judge prompts (or rule-based checks) that score each conversation against your criteria—like disclosures, escalation handling, KB accuracy, and resolution rate—and then run those metrics across simulated and production calls through the Simulate → Observe → Review loop.

Frequently Asked Questions

How does COVAL handle custom evaluation metrics like disclosures, escalation handling, KB accuracy, and resolution rate?

Short Answer: COVAL lets you define custom LLM-as-a-judge or rule-based metrics, then runs them across simulations and live calls so you can track disclosures, escalation handling, knowledge base accuracy, and resolution rate with the same lens.

Expanded Explanation:
Out of the box, COVAL provides built-in metrics like latency, interruptions, and speech tempo for audio evals. On top of that, you can add custom metrics that mirror your actual success criteria: “Did the agent provide the required disclosure?”, “Did it escalate at the right time?”, “Was the KB answer accurate?”, “Was the issue resolved?” These are implemented as reusable evaluation definitions: a structured prompt + scoring rubric that COVAL applies to each conversation or step.

Once defined, these custom metrics run everywhere—on large-scale simulations before deployment and on production calls via continuous live evals. That gives engineering, QA, product, and ops a single, consistent way to measure behavior over time, catch regressions quickly, and prove performance against the things your risk and CX teams actually care about.

Key Takeaways:

• COVAL supports custom LLM-as-a-judge and rule-based metrics that mirror your real-world requirements.
• The same custom metric definitions power both pre-launch simulations and production monitoring for a unified view.

---

What’s the process to create a custom evaluation metric in COVAL?

Short Answer: You define what “good” looks like, encode it as a metric definition (LLM prompt + scoring rubric or rules), attach it to your Test Sets or live eval configs, and then review the scores and failures in COVAL’s dashboards and queues.

Expanded Explanation:
Creating a custom metric in COVAL starts with clarity: you specify the behavior you want to measure, how it should be scored, and where in the call it applies. For voice-specific checks (like disclosures wording or escalation turns), you can evaluate the full transcript or individual steps. For outcome-based checks (like resolution rate or KB accuracy), you typically evaluate at the conversation level.

Under the hood, you define an evaluation spec—usually an LLM-as-a-judge prompt that takes the conversation transcript, metadata, and your instructions, then returns a structured score (e.g., pass/fail, 0–1, 1–5) and reasoning. COVAL then runs that spec at scale: across thousands of simulated calls in Simulate and continuously on production calls in Observe. Failures and edge cases are routed into Review queues so humans can validate, tune prompts, or adjust policies.

Steps:

1. Define the behavior and rubric: Write down what you want to measure (e.g., “required disclosure within first 2 turns”) and how you’ll score it (pass/fail or graded).
2. Create the metric in COVAL: Configure an evaluation definition using an LLM-as-a-judge prompt (and/or rules) that reads the transcript and returns a structured score plus explanation.
3. Attach and run: Add the metric to Test Sets in Simulate and to live evaluation configs in Observe, then use Review queues to inspect failures and refine.

---

How are custom metrics different from COVAL’s built-in metrics?

Short Answer: Built-in metrics track generic behavior (latency, interruptions, speech tempo), while custom metrics encode your domain-specific requirements like disclosures, escalation handling, KB accuracy, and resolution rate.

Expanded Explanation:
Built-in metrics are consistent across every agent: latency, interruptions per call, speech tempo, turn count, and other objective measures. They’re crucial for understanding overall performance and voice realism under load, but they don’t know your compliance rules or your definition of a “good” resolve.

Custom metrics fill that gap. They let you express your playbooks, policies, and business logic as evaluators. For example, a compliance team’s disclosure rule and a support team’s resolution criteria can both live as separate metrics and be applied to the same calls. When you run simulations or monitor production, you see both: the raw operational metrics and the high-level success/failure outcomes.

Comparison Snapshot:

• Option A: Built-in metrics
  • Latency, interruptions, speech tempo, turn count, etc.
  • Always available; no setup required.
• Option B: Custom metrics
  • Disclosures, escalation handling, KB accuracy, resolution rate, instruction-following, repetition, and more.
  • Defined by you as LLM-as-a-judge or rule-based checks.
• Best for: Teams that need both operational insight and policy-/outcome-level visibility on the same dashboard.

---

How do we implement specific custom metrics like disclosures, escalation handling, KB accuracy, and resolution rate in COVAL?

Short Answer: You implement each metric as a structured evaluator—typically an LLM-as-a-judge prompt—targeted at that behavior (disclosure, escalation, KB answer, resolution) and then run it across calls in Simulate and Observe.

Expanded Explanation:
Each of these metrics maps cleanly to COVAL’s evaluation layer:

• Disclosures: A compliance-focused metric that checks if mandatory language (e.g., “This call may be recorded”) was delivered, delivered in time, and not skipped during interruptions. The evaluator inspects the transcript, identifies where the disclosure should have occurred, and returns a pass/fail plus explanation.
• Escalation Handling: A metric that verifies whether the agent escalated to a human or fallback flow under predefined conditions (customer frustration, certain intents, repeated failures) and whether it did so gracefully. The evaluator can look at both intent signals and the tone/content of the conversation.
• Knowledge Base (KB) Accuracy: A metric that determines whether an answer derived from your KB is factually correct and aligned with the relevant document(s). Often this evaluator takes the transcript and optional ground-truth context and scores the answer on accuracy and hallucination risk.
• Resolution Rate: A metric that judges whether the customer’s core issue was actually resolved—based on the transcript, closing turns, and (optionally) any downstream system results. This is usually a conversation-level pass/fail or graded score.

These evaluators plug into your workflow: you attach them to Test Sets representing scenarios where these behaviors matter (e.g., financial disclosures, high-risk escalations) and to your production call pipelines, so you can track pass/fail trends and route failures into Review queues.

What You Need:

• Clear definitions and examples of “pass” vs. “fail” for each metric (disclosures, escalation, KB accuracy, resolution).
• Access to COVAL’s evaluation configuration to define LLM-as-a-judge prompts and attach them to simulations and live evals.

---

How should we think strategically about custom metrics for voice agents in COVAL?

Short Answer: Treat custom metrics as your governance layer: they convert your policies and business goals into measurable outcomes, enabling faster iteration, controlled failstops, and outcome-led vendor and model decisions.

Expanded Explanation:
Most voice AI programs stall because they can’t prove reliability where it matters. Demos look great, but at scale you get missing disclosures, bad escalations, KB drift, and unresolved cases. Custom metrics in COVAL let you operationalize those failure modes: you define them once, then continuously measure them across simulations and production calls.

This turns your evaluation system into a compounding reliability loop. You stress-test agents with voice realism, measure disclosures/escalations/KB accuracy/resolution rate, catch regressions early via continuous live evals and alerts, and then focus human review only on high-risk failures. Over time, your dashboards show concrete improvements, not just more prompts. And when you compare vendors or models, you do it with outcome evidence: which stack actually improves resolution rate while keeping disclosure failures at zero?

Why It Matters:

• Risk control: You can set thresholds on critical metrics (e.g., no missing disclosures, bounded escalation failures) and use alerts and failure-driven queues to create controlled failstops instead of learning from public incidents.
• Faster, evidence-led iteration: Engineering, QA, product, sales, and ops share a single lens on performance—seeing how changes affect resolution rate, KB accuracy, and compliance in one place—so you can ship faster with confidence, not crossed fingers.

---

Quick Recap

Custom evaluation metrics in COVAL are how you turn real-world requirements—disclosures, escalation handling, KB accuracy, and resolution rate—into measurable, repeatable checks that run across both simulations and production calls. You define clear rubrics, implement them as LLM-as-a-judge or rule-based evaluators, and plug them into the Simulate → Observe → Review loop. The result is a single, metric-driven view of agent performance that lets you catch regressions early, govern risk, and scale voice agents with confidence.

Next Step

Get Started