Retry Storm Alerts: An Issue When One Operation Is Retried in a Tight Loop

Last verified on Nov 10, 2026.

What it catches

A single retry is fine. It is how systems survive a blip. A retry storm is when retries stop being a recovery and become the load: one record, message or request is attempted again and again with little or no delay, and the attempts feed each other.

It matters because a storm is quiet at first. Each attempt logs a warning, every warning is individually reasonable and the system looks busy rather than broken. Meanwhile a downstream service takes the hit, a quota runs out or a worker never gets to anything else.

Common causes:

• A job fails permanently, such as bad input, but is classed as transient, so it retries without limit.
• A client and a proxy both retry, multiplying attempts.
• Backoff is missing or has a bug, and attempts land back to back.
• A message is returned to the queue before it can be acknowledged.
• A webhook consumer returns an error for an event it can never process, so the sender keeps redelivering it.

Example: one export job, many attempts

This is an example, not output from a real project. An export-worker retries failed jobs. A normal failed job shows two or three attempts. This one does not stop:

2026-11-08T04:11:02Z WARN export-worker job=exp_8831 attempt=1 error="row 1182 has invalid UTF-8"
2026-11-08T04:11:02Z WARN export-worker job=exp_8831 attempt=2 error="row 1182 has invalid UTF-8"
2026-11-08T04:11:02Z WARN export-worker job=exp_8831 attempt=3 error="row 1182 has invalid UTF-8"
2026-11-08T04:11:03Z WARN export-worker job=exp_8831 attempt=4 error="row 1182 has invalid UTF-8"

The attempt counter keeps climbing and the interval does not grow. Sazabi opens an issue. An example of what it could say:

• Name: export-worker retries job exp_8831 without backoff
• Description: The same job fails on the same row and is retried many times a minute. The error is deterministic bad input, so retrying cannot succeed. The loop is also holding a worker.
• Severity: medium

The fix is in the retry policy, not the job, and the issue says so because the error text never changes between attempts.

How Sazabi sets it up

You set nothing up. The Sazabi agent sets up sensors for your project and maintains them over time, and it fits this one to the jobs, calls and messages that appear in your logs. Sensors are standing questions about your logs, and this one asks: "Is any single job being attempted far more times than usual in a short window?"

Each job is judged on its own, so one runaway job does not hide behind the healthy ones, and the issue names the job. The agent writes, backtests and maintains the sensor as your code changes. A detection starts an investigation that opens an issue when the loop is real.

What you see and how you get told

You get an issue naming the operation, the repeated error and what is likely driving the loop. You can ask the agent follow-ups in the thread, and sazabi issues agent-prompt hands the work to a coding agent. If a storm returns while an issue is open or ignored, it folds into that issue. A resolved issue opens a new one. Issue triggered goes to the in-app feed, plus email, Slack or Teams through delivery rules. See detections with sazabi signals list --type sensor.

What you need

Logs where each attempt is its own line with something that identifies the job or message. Any queue or scheduler is fine. Background on structured logging explains why those identifiers are worth adding. Connect logs through OpenTelemetry or AWS.

What it does not cover

• Retries that never reach a log.
• Duplicate processing of the same message with no retry in the logs. That is not covered today.
• Slow growth in a queue's depth, which is a different question.
• A fixed retry count you wish to enforce as policy.

Related: new error detection for the first appearance of the failure, volume shift for a rate change on any event, and pricing for how issues are metered.