Skip to main content

AgenticOps Metrics — Agent KPIs for Operations Monitoring

22 min read
AI documentation guide

Reading Time: ~5 minutes

When AI Agents are deployed to production, system health alone cannot determine quality. We must measure Perceived Quality metrics such as "Did it understand user intent correctly?", "Did it call the right tools?", and "Is the answer faithful?". This document covers essential KPI categories and Langfuse·OTel-based instrumentation methods for Agent operations.

1. Why Agent-Specific Metrics are Necessary​

1.1 Limitations of Traditional APM​

Traditional APM (Application Performance Monitoring) is designed around system metrics such as HTTP success rate, response time, and error rate. However, Agents require additional metrics for the following reasons:

Table 1. Gap between traditional APM metrics and Agent quality metrics

Traditional APMAgent Quality MetricGap
HTTP 200 OKCorrect answerRequest success ≠ Result quality
Response time (total)Time to First TokenUser-perceived speed differs in streaming
Error rateHallucination rateLLM errors return HTTP 200, not 500
CPU/MemoryToken costCloud LLMs charge per token
N/ATool-call accuracyWrong tool call is not a system error

1.2 Perceived Quality vs System Metrics​

Figure 1. System-side versus user-side view of the same response

Expand the diagram to read labels and follow connections.
Diagram

Agent quality is ultimately judged by whether it accurately performed the user's desired task, which is independent of system success metrics.

2. Core KPI Categories​

2.1 Task Success​

Measures whether the user's requested task was completed.

Table 2. Task Success metrics: definition and measurement method

MetricDefinitionMeasurement Method
Task success ratePercentage of successful conversation sessionsAutomated eval (goal attainment) + HITL sampling (10%)
Completion time (p50/p95)Time to task completionSession duration (seconds)
Goal attainment scaleUser goal achievement (1-5)Explicit feedback (thumbs up/down) or LLM-as-Judge

Example (Customer Support Agent):

The example below uses the Langfuse Python SDK v2 trace()/score() API. SDK v3 changed the API to OpenTelemetry-based spans and create_score(), so pin the SDK version you use and confirm its API before applying this. Also, the task_success value (0.0/1.0) is a score recorded after a separate evaluation step (rule-based or LLM-as-Judge) judges the session outcome; the SDK does not compute it automatically.

# Langfuse automated evaluation example (SDK v2 API)
from langfuse import Langfuse
langfuse = Langfuse()

trace = langfuse.trace(
    name="customer-support-session",
    session_id="sess_abc123",
    metadata={"intent": "refund_request", "channel": "web"}
)

# At session end, a separate evaluation step judges and records success
trace.score(
    name="task_success",
    value=1.0,  # 0.0 = failure, 1.0 = success (from a separate evaluation)
    comment="Refund processed and confirmation sent"
)

2.2 Tool Use Accuracy​

Measures whether the Agent correctly calls the right tools.

Table 3. Tool Use accuracy metrics: definition and measurement method

MetricDefinitionMeasurement Method
Tool-call accuracyPercentage of correct tool calls(Correct tool calls) / (Total tool calls)
Tool invocation rateAverage tool calls per sessionSpan hierarchy analysis
Tool failure rateTool call failure percentageHTTP 5xx, Timeout, JSON parsing error

Example:

# Record tool call span
span = trace.span(
    name="tool_call",
    input={"tool": "get_weather", "args": {"location": "Seoul"}},
    metadata={"tool_name": "get_weather", "tool_version": "v1.2"}
)

# Evaluation criteria: intent="weather question" → correct tool="get_weather"
# Wrong example: calling "search_web" instead of "get_weather" → accuracy 0.0
span.score(
    name="tool_call_accuracy",
    value=1.0,  # Correct tool selected
    comment="Correct tool selected for weather intent"
)

2.3 Quality & Safety​

Measures answer quality and safety violations.

Table 4. Quality and safety metrics: definition and measurement method

MetricDefinitionMeasurement Method
Context faithfulnessShare of answers grounded in the provided context (context-adherence measure)Ragas Faithfulness (0.0~1.0)
Factual-error rateShare of answers that contradict external factsFact-check against reference (human or LLM-as-Judge)
Detector-flag rateShare of responses flagged by a hallucination detectorSelfCheckGPT and similar detector flags / total responses
Guardrails violation rateInput/output filter block rateinput/output filter block count
Toxicity incidenceHarmful content generation ratePerspective API / OpenAI Moderation

These three metrics measure different phenomena. Context faithfulness captures whether an answer strays from the given context (RAG grounding), factual-error rate captures disagreement with external facts, and detector-flag rate captures the share flagged as suspicious by a detector. Reporting them as a single "hallucination rate" obscures the cause, so measure them separately.

Hallucination Measurement Example (Ragas Faithfulness):

evaluate() returns an aggregate over the whole dataset, so result["faithfulness"] is the dataset mean, not the score of a single trace. To record per-trace scores, extract each sample's score and bind it to its trace, and record the dataset mean separately at the evaluation-run level.

from ragas.metrics import faithfulness
from ragas import evaluate

# RAG Agent evaluation (dataset level)
result = evaluate(
    dataset=test_dataset,
    metrics=[faithfulness],
    llm=ChatOpenAI(model="gpt-4o-mini")
)

# 1) Record per-sample scores on their traces
scores_df = result.to_pandas()
for row, trace_id in zip(scores_df.itertuples(), trace_ids):
    langfuse.score(
        trace_id=trace_id,
        name="faithfulness",
        value=float(row.faithfulness),  # 0.0~1.0, per sample
    )

# 2) Record the dataset mean at the evaluation-run level
dataset_mean = float(scores_df["faithfulness"].mean())

Guardrails Violation Measurement:

# OpenClaw AI Gateway PII redaction block
if gateway_response.status == "blocked_pii":
    trace.score(
        name="guardrails_violation",
        value=1.0,  # Blocked
        comment="PII detected: email, phone"
    )

2.4 Cost & Efficiency​

Measures Agent operational cost and resource efficiency.

Table 5. Cost and efficiency metrics: definition and measurement method

MetricDefinitionMeasurement Method
Cost per interactionAverage cost per session (USD)Σ(input_tokens × price_in + output_tokens × price_out)
Token efficiencyEffective token ratio(Response tokens) / (Total consumed tokens)
Cache hit rateSemantic cache hit rate(cache hits) / (total queries)

Cost Tracking Example:

# Record tokens and cost in generation span
generation = trace.generation(
    name="llm_call",
    model="example-model",
    input="What is the weather in Seoul?",
    output="The current weather in Seoul is...",
    usage={
        "input": 1200,
        "output": 80,
        "total": 1280,
        "input_cost": 0.012,   # $10 / 1M tokens
        "output_cost": 0.0024,  # $30 / 1M tokens
        "total_cost": 0.0144
    }
)

Cache Hit Rate Measurement:

# When semantic cache hit occurs
if cache_hit:
    trace.event(
        name="cache_hit",
        metadata={"cache_key": cache_key, "latency_saved_ms": 2500}
    )

2.5 User Experience​

Measures user-perceived quality.

Table 6. User experience metrics: definition and measurement method

MetricDefinitionMeasurement Method
Time to First Token (TTFT)Time to first content tokenfirst non-empty content token time - request time
Task-length quartilesQuartile distribution of task duration (or step count)Collect completed-session durations and split at Q1/Q2(median)/Q3 boundaries
Escalation rateHuman handoff ratio(human handoff count) / (total sessions)

Measure TTFT at the first non-empty content token, not the first streaming chunk. Many streaming APIs send a role delta or empty string as the first chunk, so counting it as content makes TTFT look shorter than it is. Task-length quartiles are defined separately from TTFT by splitting the distribution of completed-session durations (or reasoning-step counts) at the Q1, Q2, and Q3 boundaries; the METR Task Standard is referenced as a basis for classifying task types.

TTFT Measurement Example:

import time

request_time = time.time()
# LLM call (streaming)
first_token_time = None

async for chunk in llm_stream():
    # Skip role deltas and empty chunks; measure only the first content token
    if not chunk.content:
        continue
    if first_token_time is None:
        first_token_time = time.time()
        ttft_ms = (first_token_time - request_time) * 1000

        trace.event(
            name="time_to_first_token",
            metadata={"ttft_ms": ttft_ms, "model": "gpt-4o"}
        )

Escalation Rate Measurement:

# Human handoff when Agent detects uncertainty
if confidence_score < 0.7:
    trace.event(
        name="escalation",
        metadata={
            "reason": "low_confidence",
            "confidence": confidence_score,
            "fallback": "human_agent"
        }
    )

2.6 System Reliability​

Measures Agent service stability.

Table 7. System reliability metrics: definition and measurement method

MetricDefinitionMeasurement Method
AvailabilityService uptime ratio(uptime) / (total time)
Error budgetSLO violation allowance consumption1 - (actual SLI / SLO target)
Session continuity rateUninterrupted session completion ratio(Completed sessions) / (Started sessions)
Retry exhaustion rateRetry limit exceeded ratio(max retries exceeded) / (total requests)

SLO Example (Task success rate):

Target SLO: Task success rate ≥ 95% (30-day window, per session)
Error budget: 5% of sessions → at 100k sessions/month, 5,000 failed sessions allowed

This SLO is session- (event-) based, so the error budget is spent in sessions. Converting it to a time-availability budget such as "5% = 36 hours of outage" mixes the two bases. If a time-based budget is needed, define the SLO itself as a time-availability SLO.

3. Langfuse Trace Schema Proposal​

3.1 Span Hierarchy​

Agent execution flow is represented with the following hierarchy:

Figure 2. Span hierarchy of Agent execution

Expand the diagram to read labels and follow connections.
Diagram

3.2 Standard Tags​

Apply the following tags to all traces/spans:

  • agent_name: Agent identifier (e.g., customer-support-agent)
  • model: LLM model name (e.g., example-model)
  • prompt_version: Prompt template version (e.g., v1.2.3)
  • tool: Tool name called (e.g., get_weather)
  • guardrails: Applied guardrails (e.g., pii_redaction,prompt_injection)

3.3 Score Events​

Quality evaluation is recorded as score events:

  • task_success: 0.0~1.0
  • faithfulness: 0.0~1.0 (Ragas)
  • cache_hit: 0.0 (miss) / 1.0 (hit)
  • tool_call_accuracy: 0.0~1.0
  • guardrails_violation: 0.0 (pass) / 1.0 (block)

3.4 JSON Example​

{
  "id": "trace_abc123",
  "name": "customer-support-session",
  "session_id": "sess_xyz789",
  "user_id": "user_456",
  "tags": ["agent_name:support-agent", "environment:production"],
  "metadata": {
    "channel": "web",
    "intent": "refund_request",
    "customer_tier": "premium"
  },
  "spans": [
    {
      "id": "span_001",
      "name": "agent_run",
      "start_time": "2026-04-18T10:00:00Z",
      "end_time": "2026-04-18T10:00:05Z",
      "input": "I want to request a refund for order #12345",
      "output": "I've processed your refund request...",
      "metadata": {
        "reasoning_steps": 3,
        "tools_called": ["get_order", "process_refund", "send_email"]
      }
    },
    {
      "id": "span_002",
      "parent_span_id": "span_001",
      "name": "tool_call",
      "type": "span",
      "start_time": "2026-04-18T10:00:01Z",
      "end_time": "2026-04-18T10:00:02Z",
      "input": {"tool": "get_order", "args": {"order_id": "12345"}},
      "output": {"status": "delivered", "amount": 129.99},
      "metadata": {
        "tool_name": "get_order",
        "tool_version": "v2.1",
        "latency_ms": 850
      }
    },
    {
      "id": "gen_001",
      "parent_span_id": "span_001",
      "name": "llm_generation",
      "type": "generation",
      "model": "example-model",
      "input": [{"role": "system", "content": "You are a support agent..."}, {"role": "user", "content": "I want a refund..."}],
      "output": "Based on your order status...",
      "usage": {
        "input": 1200,
        "output": 80,
        "total": 1280,
        "input_cost": 0.012,
        "output_cost": 0.0024,
        "total_cost": 0.0144
      },
      "metadata": {
        "temperature": 0.7,
        "prompt_version": "v1.2.3"
      }
    }
  ],
  "scores": [
    {
      "name": "task_success",
      "value": 1.0,
      "comment": "Refund processed successfully"
    },
    {
      "name": "faithfulness",
      "value": 0.92,
      "comment": "High context adherence"
    },
    {
      "name": "tool_call_accuracy",
      "value": 1.0,
      "comment": "All tools correctly selected"
    }
  ]
}

4. OpenTelemetry Semantic Conventions​

4.1 GenAI Semantic Conventions (As of 2026-09)​

OpenTelemetry defines LLM instrumentation standards through GenAI Semantic Conventions (semconv 1.44.0, stability: Development).

The GenAI conventions were split into the semantic-conventions-genai repository in May 2026, and all gen_ai.* attributes in the original semantic-conventions repository are now deprecated. The table below follows registry 0c87594.

Core attributes:

Table 8. Core GenAI Semantic Conventions attributes with examples and descriptions

AttributeExampleDescription
gen_ai.provider.nameopenaiGenAI provider (gcp.gen_ai, gcp.vertex_ai, etc.)
gen_ai.request.modelexample-modelModel name
gen_ai.request.temperature0.7Sampling temperature
gen_ai.request.max_tokens2048Max output tokens
gen_ai.usage.input_tokens1200Input token count
gen_ai.usage.output_tokens80Output token count
gen_ai.response.finish_reasons["stop"]Array of termination reasons (string[]) — stop, length, tool_calls

4.2 Span Kind​

  • client: Agent → LLM API call
  • internal: Agent internal reasoning logic

4.3 OTel → Langfuse Bridge​

# OpenTelemetry instrumentation → Automatic Langfuse transmission
import base64

from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Langfuse uses Basic auth (base64-encoded public_key:secret_key)
AUTH = base64.b64encode(b"pk-lf-...:sk-lf-...").decode()

# Langfuse OTLP is an HTTP endpoint (traces path: /api/public/otel/v1/traces)
exporter = OTLPSpanExporter(
    endpoint="https://cloud.langfuse.com/api/public/otel/v1/traces",
    headers={"Authorization": f"Basic {AUTH}"},
)

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)

# Now all OTel traces are sent to Langfuse
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("agent_run") as span:
    span.set_attribute("gen_ai.provider.name", "openai")
    span.set_attribute("gen_ai.request.model", "gpt-4o")
    # ... Agent execution

5. Grafana/CloudWatch Dashboard Examples​

5.1 Top-line Metrics (Executive Level)​

┌─────────────────────────────────────────────────────────────┐
│ Task Success Rate (30d)            │ 96.2% (↑ 1.2% WoW)    │
│ Avg Cost per Interaction           │ $0.12 (↓ $0.03 WoW)  │
│ Hallucination Rate                 │ 2.1% (↑ 0.3% WoW)    │
│ Escalation Rate                    │ 3.5% (→ 0.0% WoW)    │
└─────────────────────────────────────────────────────────────┘

Grafana Panel Configuration:

The langfuse_trace_score_total metric referenced by the PromQL below is not exposed by Langfuse by default. You must first set up a custom exporter (or recording rule) that exports Langfuse score data to Prometheus for this query to work.

# Task success rate (30-day average)
sum(rate(langfuse_trace_score_total{name="task_success", value="1"}[30d]))
/
sum(rate(langfuse_trace_score_total{name="task_success"}[30d]))

5.2 Drill-down Dashboard (Operations Team)​

Tool Call Analysis:

Tool Call Success Rate by Tool
┌──────────────┬──────────┬──────────┐
│ Tool         │ Calls    │ Success  │
├──────────────┼──────────┼──────────┤
│ get_weather  │ 1,234    │ 99.2%    │
│ search_web   │ 892      │ 94.5%    │
│ send_email   │ 456      │ 100%     │
│ get_order    │ 789      │ 98.7%    │
└──────────────┴──────────┴──────────┘

Guardrails Violation Trend:

Guardrails Violation Rate (7d)
┌─────────────────────────────────────────┐
│  5% ┤                                    │
│  4% ┤    ╭╮                              │
│  3% ┤  ╭╯╰╮  ╭╮                          │
│  2% ┤╭╯   ╰╮╭╯╰╮                         │
│  1% ┼╯     ╰╯  ╰─────────────────        │
│  0% ┴────────────────────────────────    │
└─────────────────────────────────────────┘
     Mon  Tue Wed Thu Fri Sat Sun

5.3 SLO Dashboard​

Error Budget Burn Rate (Availability SLO: 95%)
┌────────────────────────────────────────────────────┐
│ Current SLI: 96.2%                                 │
│ Error Budget: 5% of 720h → 36h/month              │
│ Consumed: 12.5h (34.7%)                            │
│ Remaining: 23.5h (65.3%)                           │
│                                                    │
│ ██████████████████░░░░░░░░░░░ 34.7% consumed     │
│                                                    │
│ Status: 🟢 HEALTHY                                 │
│ Estimated Days Until Budget Exhausted: 45 days    │
└────────────────────────────────────────────────────┘

6. Alerting & Anomaly Detection​

6.1 Anomaly Pattern Examples​

Table 9. Detection rules and response actions by anomaly type

Anomaly TypeDetection RuleResponse Action
Guardrails rate spikeExceeds 3σ (rolling 1 hour)PagerDuty P2, prompt review
Cost spikeHourly cost > $100 (baseline $20)Slack alert, activate rate limit
Escalation rate increaseExceeds 10% (baseline 3%)On-call engineer alert, Agent logic review
Tool failure rateSpecific tool > 20% failureAuto circuit breaker, activate fallback

6.2 Baseline Setting and Detection Algorithm​

Rolling window average-based anomaly detection:

# Example: Guardrails violation rate anomaly detection
import numpy as np

def detect_anomaly(current_rate, historical_rates, threshold_sigma=3, min_samples=14):
    """
    Args:
        current_rate: Current time period violation rate
        historical_rates: Same time period rates from history
        threshold_sigma: Standard deviation multiple threshold
        min_samples: Minimum samples required for a statistical decision
    """
    # Do not judge if the baseline is insufficient or variance is zero/non-finite
    if historical_rates is None or len(historical_rates) < min_samples:
        return {"anomaly": False, "reason": "insufficient baseline"}

    baseline_mean = np.mean(historical_rates)
    baseline_std = np.std(historical_rates)

    if not np.isfinite(baseline_std) or baseline_std == 0:
        return {"anomaly": False, "reason": "zero or non-finite std"}

    z_score = (current_rate - baseline_mean) / baseline_std

    if z_score > threshold_sigma:
        return {
            "anomaly": True,
            "severity": "high" if z_score > 5 else "medium",
            "z_score": z_score,
            "baseline": baseline_mean,
            "current": current_rate
        }
    return {"anomaly": False}

# Real-time monitoring example
current_rate = 0.08  # 8% violation rate
historical = [0.02, 0.021, 0.019, 0.022, 0.018, 0.023, 0.020,
              0.021, 0.019, 0.022, 0.020, 0.018, 0.021, 0.019]  # 14 past samples

result = detect_anomaly(current_rate, historical)
if result["anomaly"]:
    print(f"🚨 Anomaly detected: {result['current']:.1%} (baseline {result['baseline']:.1%})")
    # Send PagerDuty alert

6.3 PagerDuty/Slack Integration​

CloudWatch Alarm → SNS → Lambda → PagerDuty:

# Lambda handler: CloudWatch Alarm → PagerDuty
import boto3
import requests

def lambda_handler(event, context):
    # CloudWatch Alarm State Change EventBridge schema
    detail = event["detail"]
    alarm_name = detail["alarmName"]
    state_value = detail["state"]["value"]        # OK | ALARM | INSUFFICIENT_DATA
    state_reason = detail["state"]["reason"]
    # Metric evaluated by the alarm (single-metric alarm)
    metric = detail["configuration"]["metrics"][0]["metricStat"]["metric"]["name"]

    # PagerDuty Events API v2
    payload = {
        "routing_key": "PAGERDUTY_ROUTING_KEY",
        "event_action": "trigger",
        "payload": {
            "summary": f"Agent KPI Anomaly: {alarm_name}",
            "severity": "warning",
            "source": "cloudwatch",
            "custom_details": {
                "metric": metric,
                "state": state_value,
                "reason": state_reason
            }
        }
    }

    response = requests.post(
        "https://events.pagerduty.com/v2/enqueue",
        json=payload
    )
    # Verify delivery before returning success
    response.raise_for_status()
    return {"statusCode": response.status_code, "body": "Alert sent"}

Slack Alert Example:

🚨 Agent Metrics Alert

**Cost Spike Detected**
- Current hourly cost: $142.50 (baseline $18.20)
- Time: 2026-04-18 14:30 UTC
- Agent: customer-support-agent
- Model: example-model

**Probable Cause**: Unusual traffic spike (3.2k requests vs 800 baseline)

Actions:
- Rate limit activated (100 req/min → 50 req/min)
- Fallback to gpt-4o-mini for non-critical queries

📊 Dashboard: https://grafana.example.com/d/agent-cost
📖 Runbook: https://wiki.example.com/agent-cost-spike

7. AIDLC Stage-by-Stage Application​

7.1 Inception: Define Baseline​

Define target KPIs at project initiation.

Table 10. Inception-stage KPI targets and current baseline

KPITarget (after 90d)Baseline (Current)
Task success rate≥ 95%88% (human baseline)
Tool-call accuracy≥ 90%N/A (new)
Hallucination rate≤ 3%12% (initial prototype)
Cost per interaction≤ $0.15$0.32
Escalation rate≤ 5%18%

7.2 Construction: CI Regression Gate​

Automatically detect metric regressions in each PR. Because the good direction differs per metric (success rate and accuracy are better when higher; hallucination and cost are better when lower), specify per-metric thresholds and directions instead of one blanket threshold.

# .github/workflows/agent-quality-gate.yml
name: Agent Quality Gate
on: [pull_request]

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run Ragas evaluation
        run: |
          pytest tests/test_agent_quality.py --ragas

      - name: Check metrics regression
        run: |
          # Declare per-metric tolerance and direction in thresholds.yml
          #   task_success_rate: {direction: higher_is_better, max_drop: 0.03}
          #   hallucination_rate: {direction: lower_is_better, max_increase: 0.02}
          #   cost_per_interaction: {direction: lower_is_better, max_increase: 0.10}
          python scripts/check_regression.py \
            --baseline metrics/baseline.json \
            --current metrics/current.json \
            --thresholds metrics/thresholds.yml

7.3 Operations: Real-time Alerting​

Real-time monitoring after production deployment.

Agent KPI SLO (Production)
┌──────────────────────┬──────────┬──────────┬──────────┐
│ Metric               │ SLO      │ Current  │ Status   │
├──────────────────────┼──────────┼──────────┼──────────┤
│ Task success rate    │ ≥ 95%    │ 96.2%    │ 🟢 OK    │
│ Tool-call accuracy   │ ≥ 90%    │ 93.5%    │ 🟢 OK    │
│ Hallucination rate   │ ≤ 3%     │ 2.1%     │ 🟢 OK    │
│ Cost per interaction │ ≤ $0.15  │ $0.12    │ 🟢 OK    │
│ Escalation rate      │ ≤ 5%     │ 3.5%     │ 🟢 OK    │
│ TTFT (p95)           │ ≤ 2s     │ 1.8s     │ 🟢 OK    │
└──────────────────────┴──────────┴──────────┴──────────┘

8. References​

8.1 Langfuse Documentation​

8.2 OpenTelemetry​

8.3 Evaluation Frameworks​