How to detect API failures before users do

The status board stayed green for nineteen minutes after checkout stopped working. The health check hit GET /health every sixty seconds, got a 200, and recorded another minute of uptime. It was telling the truth. The truth just had nothing to do with whether customers could pay.

POST /charge was also returning 200. The response body, which the monitor never read, contained {"status":"failed"}. A downstream payment provider was timing out, the API was catching the timeout, and it was responding politely with a success code and a failure inside it. Every layer was behaving exactly as written. The customer just could not buy anything.

A 200 is an opinion, not a fact

A response code tells you the server received a request and produced a reply. It does not tell you the reply was correct, or that the work the request asked for actually happened. Health endpoints make this worse: most of them check that the process is alive and maybe that the database accepts a connection. They rarely exercise the dependency chain that the real endpoints rely on.

The failure that hides inside success

Even when you do watch error rates, degradation hides. Clients retry, so a request that fails once and succeeds on the second attempt reports as a slow success, not a failure. Averages bury it: a p50 of 240ms looks healthy while the p99 sits at nine seconds and a tenth of your customers are timing out. And a problem confined to one region or one payment method is mathematically erased by a global success rate.

1. 09:14p95 on /charge climbs from 380ms to 2.9s
2. 09:15GET /health returns 200, uptime board stays green
3. 09:16Client retry rate on /charge doubles
4. 09:21First support ticket: "my card keeps getting declined"
5. 09:33Engineer finds a payment-provider timeout buried in logs

Notice where the nineteen minutes went. The engineers did not look at the payment provider first. They looked at the deploy that went out at 09:02, because that was the thing that had changed and the thing they controlled. They read their own logs, checked their own dashboards, and rolled back a change that had nothing to do with it. The cause was a third party they never instrumented.

The first place engineers look is the thing they shipped, not the thing they depend on.

What a check has to watch instead

Catching this earlier does not require exotic tooling, just measuring the things that actually represent the customer:

• The latency distribution, not the average, so the p99 cannot hide behind the p50
• The meaning of the response, not just its code, so a failure wrapped in a 200 is still a failure
• A real operation, like a synthetic charge, rather than a /health endpoint that proves nothing
• Per-region and per-endpoint segmentation, so one failing market is not averaged away
• The dependencies the endpoint calls, so a provider timeout shows up as a provider timeout

The goal is simple to state and easy to skip: the signal that something is wrong should arrive before the support ticket does, and it should point at the cause rather than the last thing you deployed.