Back to blog

How to Measure Customer-Facing AI Agents in Production: SLOs, Quality KPIs, and Unit Economics

Discover how to reliably monitor production AI agents: balancing engineering SLOs with conversation quality KPIs, detecting silent failures, and tracking cost per resolution.

Nov 18, 2026
7
directions
notes from delivery practice
3
levels
strategy · process · tools
1
library
guides · checklists · cases
See cases

Why Green Tracing Dashboards Aren't Enough for LLM Agents

Traditional software monitoring focuses on standard metrics: uptime, HTTP 200 responses, memory allocation, and database latency. However, customer-facing AI agents require a fundamentally different observability model. An LLM agent can return a lightning-fast response without throwing any backend exceptions, yet deliver completely fabricated advice or commit to an unauthorized refund. In APM dashboards, every span remains green, while the business incurs real financial and reputational damage.

To run AI agents reliably in production, engineering teams must separate evaluation into two distinct tiers: synchronous infrastructure SLOs (Service Level Objectives) derived from execution traces, and conversational KPIs that assess contextual quality and user intent.

Infrastructure SLOs: Observability for Model and Tool Execution

Engineering SLOs should be measured on every single request using standard instrumentation such as OpenTelemetry. Critical technical thresholds include:

  • Time to First Token (TTFT) and Turn Latency (p95): In streaming conversational interfaces, perceived latency is everything. Measure the exact duration from the user's input to the arrival of the first streamed token on the client side, along with total turn completion time across multiple nested API calls.
  • Tool Call Error Rate: When agents orchestrate database lookups or external integrations, failures must be tagged by specific tool names. A low global error rate often masks an individual integration failing in a significant portion of runs.
  • Execution Loop Detection: Track and alert on repetitive cycles where an agent triggers identical tool calls with duplicate parameters or enters circular conversational loops without making progress toward resolution.

Quality KPIs: Evaluating Context and Intent

Understanding whether an agent actually solved a problem requires conversation-level evaluation. This typically relies on secondary LLM judges and business telemetry rather than raw logs:

  • Task Success Rate: Measured against clear operational criteria per intent (e.g., verifying that an address was updated in the backend and confirmed to the customer).
  • Honest Resolution Rate: A session should only be considered resolved if the user achieved their goal without opening a secondary support ticket within 72 hours. Simply abandoning a broken chat should never be counted as containment.
  • Silent Failure Rate: Conversations where no technical error was logged, yet the agent hallucinated policies, failed to invoke required actions, or ignored explicit system instructions.
  • User Frustration Signals: Tracking behavioral indicators such as demands for a human representative, repetitive rephrasing, aggressive punctuation, or sudden mid-task drops.

Tracking Unit Economics: Cost per Resolved Conversation

A sustainable AI implementation requires visibility into unit economics. Measuring average token cost alone is insufficient. Instead, calculate the cost per resolved conversation, which factors in LLM inference, vector retrieval, external API overhead, and the cost of human fallback when handoffs occur.

A smaller, cheaper model may seem cost-efficient on an API invoice, but if its low accuracy triggers frequent human escalations, the total cost per resolution will rise. By instrumenting unified session identifiers across frontend interactions, backend traces, and evaluation workflows, teams can spot quality regressions early and build dependable AI systems.

Services on this topic

Need this for your project?

Development turnkey: analysis → implementation → launch → support. Tell us about your task — we will reply with an estimate.

About the direction

Relevant articles