Webhooks fail silently, and that is the whole problem

Most outages announce themselves. A service returns 500s, a queue backs up, a latency graph bends. Somebody’s alert fires and somebody looks.

Inbound webhooks do not work like that. When a webhook integration breaks, the signal you get is nothing arriving — which is byte-for-byte identical to the signal you get when there is genuinely nothing to report. A quiet Tuesday and a broken Stripe subscription look the same from inside your system.

That is why these get found by a customer email.

Four ways it breaks without telling you#

The provider disabled your endpoint. Stripe, GitHub and Shopify all retire webhook subscriptions that keep failing. Your endpoint 500s during a deploy, retries fail too, and at some point the provider stops trying — for good. The failure is in their dashboard, not yours, and nothing in your infrastructure changed at the moment delivery stopped.

The payload changed shape. A provider adds a field, moves one inside a nested object, or starts sending null where it used to send a string. Your handler throws, you return 500, the provider retries a few times and gives up. Your error tracker sees a handful of exceptions and then silence, which reads like a resolved blip.

The signature check started failing. Somebody rotated a secret and updated one of the two places it lives. Every request now fails verification and gets rejected. From your side this is indistinguishable from no traffic; from the provider’s side it is a wall of 401s.

It got slow, not broken. Deliveries still succeed, but at 8 seconds instead of 200ms, because a downstream call in your handler degraded. Nothing errors. Nothing alerts. Events pile up behind the provider’s own concurrency limits and everything runs hours late.

Only the last one shows up in a normal APM tool, and only if you are looking at that endpoint specifically.

Why “we log webhooks” is not the fix#

The standard answer is a webhook logging tool: capture every request, keep the payloads, add a replay button. That is genuinely useful — when you already know something is wrong.

It does not help with any of the four above, because all four are defined by the absence of a record. You cannot notice a missing event in a list of events. Log-and-replay is a debugging tool that assumes you have already been told there is a bug.

What is missing is the thing every other part of your infrastructure already has: a baseline, and an alert when reality departs from it.

What actually needs watching#

Four signals, and none of them are visible in a request log:

• Freshness. When did this integration last deliver, compared to what is normal for it? An endpoint that fires every few minutes and has been silent for two hours is an incident. The same silence on a weekly export is not.
• Success rate over time. Not “did this one fail” but “what share of the last hour failed, and is that unusual for this endpoint”.
• Latency distribution. The p95 of your handler’s response, per endpoint, trended — because the failure mode is drift, not a cliff.
• Payload shape. The set of fields you have been receiving, and an alert when a provider quietly starts sending something else.

Each of these has an obvious baseline and an obvious alarm condition. None of them require you to have already suspected a problem.

The part only aggregate traffic can answer#

There is one question a single-tenant view structurally cannot answer: is it me, or is it them?

When a provider goes quiet, you are looking at your own logs wondering whether your deploy broke something. Checking the provider’s status page tells you what they have already decided to publish, which lags the incident by a good while and often never happens for partial degradations.

Something watching the same provider across many independent customers sees the answer immediately: if every account stopped receiving from Stripe at 14:32, it is not your deploy. That is not a smarter algorithm — it is just a vantage point a single system does not have.

---

This is what we are building Catchook to do. If webhook integrations breaking quietly is a problem you recognise, the waitlist is open.