Observability for Agentic AI: Why Traditional APM Can't Track Autonomous Agents

Traditional APM tools cannot observe agentic AI systems. They lose trace context at agent handoffs, miss reasoning decisions entirely, and collapse hours of autonomous work into a single span. If you are running multi-agent workflows in production, you need an observability stack purpose-built for agents: one that covers trace propagation, reasoning-step logging, drift detection, and cost attribution per execution.

It is 2:47am. Your on-call engineer gets paged because a customer-facing workflow took 47 seconds instead of the usual 8. She opens the APM dashboard. She sees one trace: a single POST request, a 200 OK response, and a latency spike. No errors. No failed spans. What she cannot see is that a multi-agent orchestrator delegated to three sub-agents, one of which retried a tool call 14 times because a third-party API returned ambiguous results, while another agent decided to re-plan its entire approach mid-execution. The APM dashboard is telling the truth. It is also telling her almost nothing useful.

This is the fundamental gap. APM was designed for request-response architectures where a human clicks a button, a server processes a request tree, and a response comes back. Agentic AI systems break every assumption that model relies on.

Your APM Dashboard Is Blind to Half Your Agent's Work

Standard Application Performance Monitoring tools model the world as span trees. One inbound request spawns child spans for database queries, HTTP calls, and cache lookups. Each span has a start time, end time, status code, and parent span ID. This model works brilliantly for microservices.

Agentic systems do not work this way. An agent receives a task, reasons about it, selects tools, executes them, evaluates results, potentially re-plans, and delegates sub-tasks to other agents. The execution path is not a tree. It is a directed acyclic graph (DAG) with potential cycles from retries and re-planning loops. A single "request" can involve dozens of LLM calls, tool invocations, and inter-agent messages spread across minutes or hours.

The practical consequence: your APM dashboard shows latency percentiles and error rates for the outer request. You can see that something took 47 seconds. You cannot see why an agent chose to retry instead of fail, why it selected Tool A over Tool B, or how much of that time was spent on LLM inference versus waiting for external APIs. You are monitoring the envelope, not the letter.

Three Assumptions APM Makes That Agents Violate

Before you can fix your observability, you need to understand exactly where the existing model breaks. APM tools are built on three core assumptions that agentic systems violate constantly.

Assumption 1: Traces are trees. In a standard microservice architecture, each span has exactly one parent. The trace forms a clean tree structure. Agents produce DAGs. Agent A calls Agent B, which calls a tool, gets a result, re-evaluates, calls Agent C in parallel, and then Agent C calls back to Agent A for clarification. OpenTelemetry's span model can technically represent this, but every visualization tool renders it as a tree, which makes cyclic retries and parallel branches unreadable.

Assumption 2: Operations are deterministic. Given the same input, a database query returns the same result. An HTTP call to a stable endpoint returns the same response. APM alert thresholds depend on this predictability. LLM-powered agents are probabilistic. The same prompt can produce different tool selections, different reasoning chains, and different outputs. Your p99 latency is not a fixed number; it is a distribution that shifts with model temperature, prompt construction, and context window contents.

Assumption 3: Execution is short-lived. APM traces typically span milliseconds to low seconds. Agents can maintain state across minutes, hours, or days. A research agent that gathers information over 30 minutes before synthesizing a report does not fit into a trace model designed for sub-second database queries. Most APM backends will time out or discard spans from such long-running executions.

Bolting OpenTelemetry spans onto your agent's tool calls gives you data. It does not give you understanding. You get timestamps and status codes, but you miss the reasoning layer that explains agent behavior.

Trace Propagation Across Agent Handoffs

The most common place observability breaks in multi-agent systems is the handoff. When Agent A delegates a sub-task to Agent B, the trace context is either lost entirely or starts a new root span. This creates orphan traces that cannot be correlated back to the original task.

The fix is straightforward but requires discipline. Every inter-agent message must carry a correlation ID and parent span reference in its envelope. Here is a minimal example:

# agent_message.py
from dataclasses import dataclass, field
from typing import Optional
import uuid

@dataclass
class AgentMessage:
    task: str
    payload: dict
    # Trace propagation fields
    trace_id: str = field(default_factory=lambda: uuid.uuid4().hex)
    parent_span_id: Optional[str] = None
    source_agent: str = ""
    execution_phase: str = "planning"  # planning | executing | evaluating

    def create_child_context(self, child_agent: str, phase: str) -> "AgentMessage":
        """Propagate trace context to a child agent."""
        return AgentMessage(
            task=self.task,
            payload=self.payload,
            trace_id=self.trace_id,       # Same trace
            parent_span_id=self.span_id,  # Link to parent
            source_agent=child_agent,
            execution_phase=phase,
        )

The critical detail: trace_id stays constant across the entire agent chain while parent_span_id creates the linkage between agents. When agents communicate through async queues (SQS, Kafka, Redis Streams), you must serialize these fields into the message body or headers. When agents use webhooks or external APIs as intermediaries, inject trace context into custom HTTP headers following the W3C Trace Context specification [1].

Design your span schema with distinct span types for different agent operations. An LLM invocation span is different from a tool call span, which is different from a reasoning evaluation span. This lets you filter and aggregate by operation type when debugging.

Reasoning-Step Logging: Capturing the Why, Not Just the What

Call logging tells you an agent invoked the search API. Reasoning logging tells you the agent chose search over its local cache because its confidence score for cached data was 0.62, below its configured threshold of 0.7. The difference between these two log entries is the difference between knowing what happened and understanding why.

Here is a structured reasoning log schema that captures decision points:

{
  "trace_id": "a1b2c3d4e5f6",
  "agent_id": "research-agent-v2",
  "timestamp": "2025-06-15T14:32:07Z",
  "decision_point": "data_source_selection",
  "alternatives_considered": [
    {"option": "local_cache", "confidence": 0.62, "reason": "cache age 47min exceeds 30min freshness policy"},
    {"option": "search_api", "confidence": 0.89, "reason": "query matches high-priority topic category"},
    {"option": "skip_retrieval", "confidence": 0.15, "reason": "insufficient context for direct answer"}
  ],
  "selected_action": "search_api",
  "selection_rationale": "highest confidence; cache staleness exceeded threshold",
  "token_budget_remaining": 12400,
  "execution_phase": "planning"
}

Privacy matters here. Reasoning logs can capture user queries, personal data, and sensitive business context. Filter PII before logs reach your observability backend. Apply the same data classification rules you use for application logs. Most teams configure a sanitization layer in the log pipeline that redacts fields matching known PII patterns (emails, phone numbers, account IDs) while preserving the reasoning structure.

Drift Detection: When Agents Go Off-Script

Behavioral drift is when an agent's output distribution shifts from its baseline without any code change. This happens more often than most teams expect, because agents depend on external systems (LLM providers, APIs, knowledge bases) that change independently of your deployment cycle.

Three categories of drift matter for production agents:

- Tool-selection drift: The agent starts choosing different tools for the same class of inputs. This can happen when a tool's response latency changes, altering the agent's cost-benefit calculation.

- Output-quality drift: The agent's outputs pass validation checks but measurably decline in quality. This often follows an upstream model update from your LLM provider.

- Cost-profile drift: The agent's token consumption per task changes significantly. A 2x increase in average tokens signals the agent is taking longer reasoning paths or retrying more frequently.

To detect drift, record behavioral fingerprints: the sequence of tool calls, token counts, and decision confidence scores for each execution. Compare these against a rolling 7-day baseline window using statistical tests (KL divergence for distributions, chi-squared for categorical tool selections).

Here is a real scenario: a team running a customer support agent noticed a gradual UX degradation over two weeks. No code had changed. No errors appeared. Investigation revealed that their LLM provider had adjusted pricing tiers, and the agent's cost-optimization logic had silently shifted from GPT-4o to GPT-4o-mini for 60% of requests. The cheaper model produced adequate but noticeably worse responses. With cost-profile drift detection, this would have triggered an alert within the first day.

Cost Attribution Per Agent Execution

Aggregate LLM spend dashboards are where budget overruns go to hide. Seeing "$14,200 in OpenAI spend this month" tells you nothing about which agent, which workflow, or which execution phase is consuming your token budget. Teams frequently discover that one poorly-tuned agent accounts for 50-60% of total spend, usually because of retry loops or unnecessarily verbose prompts.

Tag every LLM call with three identifiers: agent ID, task ID, and execution phase (planning, executing, evaluating). This enables per-agent cost breakdowns that surface the actual cost drivers.

# cost_tracking_middleware.py
import tiktoken

COST_PER_1K_TOKENS = {
    "gpt-4o": {"input": 0.0025, "output": 0.01},
    "gpt-4o-mini": {"input": 0.00015, "output": 0.0006},
    "claude-sonnet-4": {"input": 0.003, "output": 0.015},
}

def track_llm_cost(response, model: str, agent_id: str, 
                   task_id: str, phase: str) -> dict:
    input_tokens = response.usage.prompt_tokens
    output_tokens = response.usage.completion_tokens
    rates = COST_PER_1K_TOKENS[model]
    cost = (input_tokens * rates["input"] + 
            output_tokens * rates["output"]) / 1000
    
    return {
        "agent_id": agent_id,
        "task_id": task_id,
        "execution_phase": phase,
        "model": model,
        "input_tokens": input_tokens,
        "output_tokens": output_tokens,
        "cost_usd": round(cost, 6),
        "trace_id": get_current_trace_id(),
    }

Set per-execution cost ceilings. When an agent's accumulated cost for a single task exceeds the ceiling, the system should either halt the agent, downgrade it to a cheaper model, or escalate to a human. This prevents a single runaway retry loop from consuming your daily budget.

The Planning Agent here represents 46% of total monthly spend. Without per-agent attribution, this would be invisible in an aggregate dashboard showing "$8,330/month in LLM costs."

Building the Agentic Observability Stack

A complete agentic observability stack has four layers, each connected by a shared trace ID.

Layer 1: Trace collection with agent-aware span schemas. Start with OpenTelemetry but extend the semantic conventions. Define custom span kinds for agent.reasoning, agent.tool_call, agent.llm_invocation, and agent.handoff. This gives you filterable, queryable traces that distinguish between agent thinking and agent doing. Projects like OpenLLMetry [4] and Langfuse [5] provide starting points for LLM-specific instrumentation.

Layer 2: Reasoning log pipeline. Ingest structured reasoning logs (the JSON schema from earlier) into a queryable store. Elasticsearch or ClickHouse work well here because you need both full-text search ("show me every time an agent rejected the cache") and analytical queries ("average confidence score by agent over the last 7 days"). Keep reasoning logs separate from application logs; they have different retention, access control, and query patterns.

Layer 3: Drift detection engine. Run statistical comparisons of behavioral fingerprints against rolling baselines. This can be a batch job (hourly) or a streaming computation, depending on your alert latency requirements. For most teams, an hourly batch job comparing the last hour's fingerprints against the 7-day baseline is sufficient.

Layer 4: Cost attribution and budget enforcement. Aggregate per-call cost records by agent ID, task ID, and time window. Enforce budget ceilings in the agent execution middleware, not in a separate monitoring system. The enforcement must be synchronous (blocking the next LLM call) rather than asynchronous (alerting after the money is spent).

The join key across all four layers is the trace ID. Every reasoning log, every cost record, every drift signal, and every span shares the same trace ID. This means you can start from a cost anomaly alert, find the trace, read the reasoning logs to understand why the agent spent so many tokens, and check whether drift detection flagged the behavior.

For teams tracking engineering quality and cost metrics across multiple agent-powered services, Connectory's Engineering Intelligence Dashboard can surface these signals alongside code quality and deployment metrics, giving engineering leads a single view into both human-written and agent-generated system behavior.

Frequently Asked Questions

Can I use Datadog or New Relic for agentic observability?

You can use them for Layer 1 (trace collection) with custom instrumentation. Both support OpenTelemetry and custom span types. However, neither provides native support for reasoning logs, behavioral drift detection, or per-agent cost attribution. You will need to build Layers 2-4 yourself or use specialized tools like Langfuse, Braintrust, or Arize AI alongside your existing APM.

How much overhead does reasoning logging add?

In our experience, structured reasoning logs add 5-15ms of latency per decision point, mostly from JSON serialization and log shipping. This is negligible compared to LLM inference latency (typically 500ms-3s per call). Storage costs are modest: roughly 3GB per million agent decisions at full verbosity.

When should I start adding agentic observability?

Before your second agent goes to production. One agent with APM is manageable. Two agents with inter-agent communication and no trace propagation becomes a debugging nightmare within the first week.

Start Observing Agents This Week

You do not need all four layers to start making progress. Here is a concrete two-week plan:

This week: Pick your highest-traffic multi-agent workflow. Add a trace_id and parent_span_id to every inter-agent message. Verify that you can follow a single task execution from the entry point through every agent handoff. This is two to four hours of work for a single workflow.

Next sprint: Add structured reasoning logs to your most expensive agent (check your LLM provider dashboard to identify it). Use the JSON schema from this article as a starting point. Log to a separate index in your existing search backend.

The metric to start tracking now: cost-per-successful-task-completion, not cost-per-LLM-call. The first metric tells you how efficiently your agents deliver value. The second tells you how much you are paying OpenAI. These are very different numbers, especially when retry-heavy agents inflate LLM call counts without completing more tasks.

Remember that 2:47am page from the opening? With trace propagation and reasoning logs in place, your on-call engineer would see a decision graph: Agent A delegated to Agent B, which attempted a tool call, received ambiguous results (confidence: 0.31), retried with a refined query 14 times (escalating cost to $3.80), and eventually succeeded. The 47-second latency becomes a readable story instead of a black box. The fix, adding a confidence floor that triggers a fallback agent after 3 retries, becomes obvious.

Your agents are making hundreds of decisions per minute. Start watching what they decide, not just what they return.

References

[1] W3C, "Trace Context Specification," 2024. https://www.w3.org/TR/trace-context/

[2] Braintrust, "Evaluating LLM Applications in Production," 2024. https://www.braintrust.dev/docs

[3] OpenAI, "API Pricing," 2025. https://openai.com/api/pricing/

[4] Traceloop, "OpenLLMetry: Open-source observability for LLM applications," 2025. https://github.com/traceloop/openllmetry

[5] Langfuse, "Open Source LLM Engineering Platform," 2025. https://langfuse.com/docs

[6] OpenTelemetry, "Semantic Conventions for Gen AI Systems," 2025. https://opentelemetry.io/docs/specs/semconv/gen-ai/

[7] Anthropic, "Claude Model Pricing," 2025. https://docs.anthropic.com/en/docs/about-claude/models