The Saturday the site went down and nobody noticed

The Saturday the site went down and nobody noticed

A company found out their website had been offline for eleven hours from a customer. Not from a monitoring system, not from an alert — from a polite message that said, "just so you know, your site isn't loading."

Eleven hours, and the customer told us

Eleven hours, on a Saturday, in the middle of an advertising campaign. The money for clicks had been spent in full. The visitors had arrived, met an error page, and left.

The worst part was not the outage. The worst part was that nobody inside the company could have known. The site was "working" — until you tried to open it. There was no alarm, no check, no one whose job it was to notice. The business trusted the fact that the website had always been there.

Monitoring that actually means something

Modern hosting is reliable, which is exactly why nobody prepares for the moment it is not. A server runs out of disk space. A certificate expires. A payment provider changes something on their side and a single function starts failing. None of these announce themselves; they just quietly turn customers away.

We put monitoring in place, and that word means something narrow here: every few minutes, from outside the server, something checks that the site actually opens, that the checkout actually responds, and that the pages actually load in reasonable time. If a check fails twice in a row, a notification goes out — to a phone, not to an email nobody reads on a weekend.

Outages still happen

Then we treated the causes separately. Certificates renew on their own with a warning a month ahead. Disk space is watched. Every external integration is checked as a separate thing, because a site can be perfectly up and still unable to accept an order.

Outages still happen to everyone. The difference is between finding out from a stranger on Saturday afternoon, and finding out from an alert at 3 am while you are asleep — which, for a business, is a very comfortable difference.

All articles