Detection
Detection

LLM Behavior Drift Alerts: An Issue When Model Output Quietly Changes

By Sazabi Team

Last verified on Nov 10, 2026.

What it catches

A model call can succeed at the HTTP level and still go wrong. The reply is cut short, wrapped in prose when you asked for JSON or refused for the first time. Nothing throws. The Sazabi agent sets up sensors for your project and maintains them over time, and LLM behavior drift is one of the signals it writes when your logs show model calls. The shifts it can watch for:

  • Output that fails to parse as the structure you asked for, going from rare to common.
  • Refusals or apologies in a feature that never produced them.
  • Reply length moving, for example a summary that becomes twice as long.
  • Token use per call climbing for the same prompt.
  • Calls that start returning empty content.

Example: structured output starts failing

This is an example, not output from a real project. A ticket-triage service asks a model for JSON with a category and a priority. For months a parse failure was a one-off.

2026-11-08T08:14:02Z INFO  ticket-triage llm ok model=assistant-large prompt=triage-v4 output_tokens=61 parsed=true
2026-11-08T10:31:47Z WARN  ticket-triage llm parse failed model=assistant-large prompt=triage-v4 output_tokens=140 reason=prose_before_json
2026-11-08T10:31:52Z WARN  ticket-triage llm parse failed model=assistant-large prompt=triage-v4 output_tokens=138 reason=prose_before_json

A sensor for this could ask: "Has the share of triage calls that fail to parse gone far above its usual level?" Each model and prompt name is judged on its own, so a change to one prompt shows up without being averaged away by the others.

An example issue:

  • Name: ticket-triage JSON parse failures rising on assistant-large with triage-v4
  • Description: Since 10
    UTC a growing share of replies start with prose before the JSON, and output length per call has roughly doubled. No change to the prompt or code was deployed in that window.
  • Severity: medium

The last sentence of that description is what makes the issue useful, since it sends you to the model side first.

How Sazabi sets it up

You set nothing up and define no metrics. The agent reads your logs for the lines your LLM calls produce, such as the model, the prompt, token counts and parse results, and writes and maintains sensors for the ones that matter. Most ask whether a share, a count or a size has moved far outside its usual range. Generative output is noisy, so ordinary variation in lengths and counts does not count as drift.

It keeps these current as you change models and prompts. If you want something specific, such as an alert on refusals for one feature, you can ask support for it in plain words.

What you see and how you get told

The issue shows what moved, since when and for which model or prompt, with evidence from your logs, and it arrives already investigated. Notifications follow your own settings, with shared-channel rules by severity, component or owning team. Use the thread to ask whether the shift lines up with a model change, a prompt version or a particular customer's traffic. If the fix is in code, such as stricter parsing or a retry with a repair prompt, sazabi issues agent-prompt produces a prompt for a coding agent.

sazabi signals list --type sensor lists the signals behind it.

What it needs

Your logs or telemetry must carry the facts: whether output parsed, token counts, refusals if you track them. Sazabi accepts AI framework telemetry through OpenTelemetry, and LLM observability tools such as Langfuse and LangSmith are among the integrations.

What it does not cover

  • It does not grade correctness, tone or factual accuracy, and it does not run evals.
  • Spend is its own question, with its own signal. Rogue agent behavior is not covered today.
  • It cannot change a model or a prompt for you.
  • It does not read provider release notes.

Provider throttling is a different failure, covered under vendor rate limiting. A provider that answers slowly is vendor degradation. The manual approach is in the guide to LLM behavior drift. See pricing for billing.

What Sazabi detects today

There are no monitors or alert rules to write: the Sazabi agent automatically sets up sensors for your project and maintains them over time. Sazabi opens an issue when one fires.

  • LLM behavior drift: Model outputs change: length, refusals, malformed JSON or token use. Example: Structured output parse failures go from 0.1% to 4%.

Frequently Asked Questions

Does Sazabi judge whether the answers are good?

No. It does not score correctness or run evaluations. It watches measurable things your logs show: how often output fails to parse, how long output is, how often the model refuses and how many tokens calls use.

Which frameworks and providers does this work with?

Anything that writes the relevant lines to logs or telemetry you send. The agent reads what is there. Connecting an AI framework or an observability tool for LLMs gives it more to work with.

Will it notice when a provider silently updates a model?

It notices the effect if the effect is in your logs, for example the parse failure share rising on the same prompt. It does not read model release notes.

Does it replace evals?

No. Evals test quality before release. This looks for changes in production behavior after release, and the two answer different questions.

Sources

  1. Issue contents, severity, status, hand-off to a coding agent (accessed Nov 8, 2026)
  2. Issue triggered notifications and delivery rules (accessed Nov 8, 2026)
  3. Listing sensor signals from the CLI (accessed Nov 8, 2026)
Send your LLM logs to Sazabi