LLM Observability and AI Agent Observability Sources

Last verified on Nov 8, 2026.

An AI feature fails differently from ordinary code. The request succeeds, the model answers and the answer is still wrong, slow or expensive. Telemetry from the framework layer is how you see that, and this category covers the sources that supply it.

What these sources record

Each source records one level of the stack.

• LangChain has no OpenTelemetry output of its own. You add an instrumentation package such as OpenInference, which creates spans for chains, tools and model calls. See the LangChain integration.
• Mastra ships an OtelExporter. You add it to the observability config and Mastra batches spans for agent runs, with no hand-built SDK setup. See the Mastra integration.
• OpenRouter pushes one trace span per generation request through its Broadcast feature. Each span names the model and upstream provider and carries token counts and cost. See the OpenRouter integration.
• Respan auto-instruments LLM and agent frameworks and sends spans over OTLP. See the Respan integration.
• Claude Agent SDK emits the same events as Claude Code, covered under Claude Code.

How teams use them together

The usual split is a gateway source plus a framework source. A gateway such as OpenRouter sees every model call from every service, so it answers questions about spend and provider mix. A framework source sees the shape of one agent run, so it answers questions about which tool failed and why the agent looped.

Neither replaces your application logs. The span says a tool call returned an error. The application log says the database row it needed was gone. Search both in one thread and they explain each other.

Example: cost per session

The Claude Agent SDK sends claude_code.api_request events with model, input_tokens, output_tokens, cache_read_tokens, cost_usd and duration_ms. Every event also carries session.id, and app.entrypoint is sdk-py or sdk-ts for agents you build, which separates production agents from interactive sessions.

An example event, with made-up values:

event            = claude_code.api_request
app.entrypoint   = sdk-ts
session.id       = 7c1e0d52
model            = claude-sonnet
input_tokens     = 18400
output_tokens    = 920
cache_read_tokens = 16000
cost_usd         = 0.071
duration_ms      = 6340

Ask for the sum of cost_usd by session.id over the last day. A handful of sessions that cost ten times the median is a runaway loop or an oversized context, and you can open each one to see the sequence of requests. This is a query you run yourself. Sazabi does not alert on spend today.

What to watch across all of them

• Content capture. Prompts often contain your users' input. OpenRouter includes prompt and completion text unless Privacy Mode is on, and the Claude Agent SDK exports raw prompts only when OTEL_LOG_USER_PROMPTS=1. Decide per environment and keep it off in production unless you need it.
• Missing usage. If the provider returns no usage data, spans have no token counts. A cost chart built on those spans undercounts.
• Repeated tool calls. The same tool called many times inside one session is the signature of a loop. Log a session or run ID so you can count calls per run.
• Latency by step. A slow response can come from the model, a tool or a retry. Per-step duration separates them.
• Metrics. Some of these sources send OTLP, and Sazabi drops OTLP metrics sent to a logs and traces endpoint. Use the OpenTelemetry metrics source for those.

Which detections apply

None of the detections Sazabi runs today is tied to AI framework sources, so the cost and loop checks described above are queries and alerts you set up yourself.

For the transport underneath most of these sources, read about OpenTelemetry with Sazabi.