Vendor Monitoring: Detect Third-Party and Upstream Failures

Last verified on Nov 8, 2026.

Your product depends on services you do not run: payment processors, email and SMS providers, identity services, model APIs, storage, maps and every webhook that tells you something happened. When one degrades, your users blame you, and your own dashboards often show nothing wrong because your code is healthy.

Vendor monitoring means measuring each dependency from your side, at every call, so you know about the problem before the vendor's status page does.

How vendors fail

• Errors: 5xx responses, connection resets and timeouts.
• Slowness: latency that rises without errors, which ties up your threads and queues.
• Throttling: rate limit responses when you, or a retry storm, send too much.
• Silence: events you expect, such as webhooks, stop arriving.
• Contract changes: a field disappears or changes type, or a response format shifts, with status codes still 200.
• Behavior drift, which is especially common with model APIs: same request, different answers. See LLM behavior drift.

What to log on every outbound call

One line per attempt, plus one for the final outcome after retries. Include the vendor, endpoint, status, duration, attempt number and a request ID.

2026-11-08T12:05:51Z msg="vendor call" vendor=mailer method=POST endpoint=/v3/send status=503 duration_ms=2004 attempt=1 request_id=r-1042
2026-11-08T12:05:53Z msg="vendor call" vendor=mailer method=POST endpoint=/v3/send status=503 duration_ms=2011 attempt=2 request_id=r-1042
2026-11-08T12:05:53Z msg="vendor call final" vendor=mailer method=POST endpoint=/v3/send outcome=failed attempts=2 request_id=r-1042

OpenTelemetry defines http.client.request.duration as a stable histogram in seconds, so auto-instrumented HTTP clients can already give you latency per destination. Retries are the key detail: if you only record the final outcome, a vendor that succeeds on the third try looks healthy while adding seconds to every request.

Detect it step by step

1. Error ratio per vendor per five minutes, from the attempt lines:

-- Example: vendors with a high failure share over the last 10 minutes
SELECT vendor,
       count(*) AS calls,
       avg(CASE WHEN status >= 500 OR status = 0 THEN 1.0 ELSE 0 END) AS failure_share
FROM vendor_calls
WHERE ts >= now() - interval '10 minutes'
GROUP BY vendor
HAVING count(*) >= 30 AND avg(CASE WHEN status >= 500 OR status = 0 THEN 1.0 ELSE 0 END) > 0.05;

2. Latency per vendor, compared with the same hour last week. Alert on the 95th percentile, not the mean.
3. Rate limiting: count 429 responses. RFC 6585 defines 429 as the user having sent too many requests in a given amount of time, and says responses may include a Retry-After header. Per RFC 9110, Retry-After indicates how long to wait, and on a 503 it indicates how long the service is expected to be unavailable. Log that value and obey it.
4. Silence: compare expected inbound events with actual ones per hour. A payment provider that sends hundreds of events per hour and then sends none is a gap, whatever the vendor's status page says. See webhook gaps.
5. Contract checks: track the null or missing rate of fields you rely on, and the share of responses that fail schema validation. A jump after months of zero is a contract change.
6. Correlate with the vendor's own incidents. Record vendor status events next to your error ratios so the first question in an incident, "is it them or us", is answered by a chart. See vendor degradation.

Thresholds and pitfalls

Use per-vendor thresholds. A vendor called twice a minute needs a longer window than one called every second, and a minimum call count keeps one failure from reading as 100 percent.

• Your own network or DNS problems look like vendor failures. Check whether several unrelated vendors fail at once. If they do, look at your egress first.
• Timeouts you set too low make a healthy vendor look broken. Compare your timeout with the vendor's normal p99.
• Retries hide errors and amplify load. Track attempts per call as its own metric.
• Backoff that ignores Retry-After turns throttling into an outage.
• Seasonality: a vendor that is busy every first of the month may be slow every first of the month.

What to do when it fires

Decide who owns the next step. If the vendor is degraded, switch to a fallback or queue the work, trip a circuit breaker so you stop piling load on them and show a clear message to users. If it is your retry behavior or request volume, fix your side. If the contract changed, patch the parser and ask the vendor what changed.

Afterward, record the vendor, the duration and the user impact. That history makes a decision about replacing a vendor, or adding a second one, a matter of data. To bring vendor events into your telemetry, see the webhooks integration. Typical dependencies worth this treatment include Stripe, Twilio and OpenRouter.