Data Loss Detection: Find Writes That Never Persist

Last verified on Nov 8, 2026.

A lost write is a save that the system acknowledged and then never kept. The user saw a success message, the API returned a 2xx and the next page load shows the old value. Nothing crashed, so nothing alerts.

Error monitoring cannot see this failure, because there is no error. You find it by checking that every acknowledged write has a matching durable outcome.

How writes get lost

The common causes are mundane:

• The handler answers before the work is done. HTTP defines 202 Accepted as a request accepted for processing where the processing has not been completed, and the request might never be acted on. Any route that returns 202 and enqueues a job can lose the write if the job fails.
• A catch-all error handler returns success after a failed insert.
• An UPDATE matches zero rows. PostgreSQL reports this as a command tag of UPDATE 0, and a handler that ignores the count reports success for a write that touched nothing.
• A transaction rolls back after the response was sent.
• The write goes to a cache or a replica and never reaches the primary store.
• Asynchronous commit settings trade durability for speed. PostgreSQL documents that with synchronous_commit set to off, a crash can lose recently reported commits, with a maximum delay of three times wal_writer_delay.

Evidence in your telemetry

A lost write leaves a gap between two kinds of events: the request that was acknowledged and the write that should have followed. Example log lines for one request:

2026-11-08T09:14:02Z msg="request complete" request_id=r-5521 method=PUT route=/api/profile status=200
2026-11-08T09:14:02Z msg="profile update" request_id=r-5521 user_id=88 rows_affected=0

The first line says success. The second shows no row changed. Neither is an error, so the pair is invisible unless you look for it.

User-side evidence is also useful: a form submit followed within seconds by a reload that returns the old value, or support tickets that say "my changes disappeared".

Detect it step by step

1. Log the outcome of every write. Include request_id, the rows affected, and whether the transaction committed. Never sample these lines, because a gap in sampled data looks identical to a lost write.
2. Find acknowledged writes with no persisted outcome. Leave a settling period so in-flight work is not counted:

-- Example: successful mutating requests with no commit logged for the same request
SELECT r.route, count(*) AS unmatched
FROM request_logs r
LEFT JOIN write_logs w
  ON w.request_id = r.request_id AND w.committed = true AND w.rows_affected > 0
WHERE r.method IN ('POST', 'PUT', 'PATCH', 'DELETE')
  AND r.status < 300
  AND r.ts BETWEEN now() - interval '2 hours' AND now() - interval '10 minutes'
  AND w.request_id IS NULL
GROUP BY r.route
ORDER BY unmatched DESC;

3. Add a read-back canary. A scheduled job writes a marker through the public API, waits for the longest normal replication or queue delay, reads it back and logs the result. It tests the whole path, including caches and replicas.
4. Compare counts across stores. Orders accepted by the API against rows in the orders table, per hour, is a cheap reconciliation. The reconciliation mismatches guide goes deeper.

Thresholds and pitfalls

Alert on the ratio, not the raw count: unmatched acknowledged writes divided by all acknowledged writes per route. A healthy system sits near zero, so even a small sustained ratio on a payment or profile route is worth a look.

Watch for these false positives:

• Idempotent writes that legitimately change nothing. PostgreSQL counts matched rows in the UPDATE tag even when values did not change, so UPDATE 0 is a true no-match. Other databases may count differently, so check yours.
• Asynchronous routes. Set the settling period to your slowest normal job time, not to a few seconds.
• Retried requests that reuse a request ID. Match on the idempotency key when you have one.
• Log pipelines that drop lines. If the write log is missing because ingestion lagged, you will see phantom losses. Check ingestion health first, and see the novel events guide for catching new messages that signal a changed code path.

What to do when it fires

Pick one affected request ID and trace it end to end. Check whether the handler returned before the commit, whether the job queue accepted the message and whether the database row exists under another key. If the write path swallows errors, make it fail loudly and return the real status.

For data already lost, restore from the source of truth: the event stream, the queue dead-letter store or a backup. Then add the canary so the next regression is caught within minutes.

The join above only works when request logs and write logs live in one searchable place. OpenTelemetry and Fluent Bit can ship both. Related guides cover stale data and field drift, the null and empty values that often accompany a broken write path.