Health checks matter more in the age of AI agents

I recently found out that Home Assistant's Octopus Energy integration had been broken for months. Octopus had changed its API, and the version of the integration I had installed no longer worked with it. Its sensors had gone unavailable, and it only came to light when I went looking for the data to check my energy bills. Fixing it took a newer version of the integration and a new API key.

Nothing told me, because nothing expected that data to keep arriving. I now set things up the other way round: each job and server reports when it has worked, and a service outside the house tells me when those reports stop.

The software around my code is changing faster#

I usually have several coding agents running at once, so I ship a lot of changes. Each one goes through automated tests, several rounds of adversarial review by agents told to find what's wrong, and agents that test the product the way a user would.

None of that covers the software I depend on. Closed-source products and SaaS APIs change without showing me the code, and I hear about a change when the vendor announces it, if I'm paying attention. Open-source projects publish their changes, but someone still has to read them. Either way, new bugs and security vulnerabilities keep arriving in software I didn't write.

Everyone else is shipping with AI too. In Google's 2025 DORA report, 90% of the nearly 5,000 technology professionals surveyed said they use AI at work. AI adoption now has a positive relationship with software delivery throughput, but it "does continue to have a negative relationship with software delivery stability". The report's explanation is that without strong testing and fast feedback loops, "an increase in change volume leads to instability". A survey can't establish cause, but the volume part matches my experience: the software I run changes faster than I can review it. I can't fix that, but I can find out quickly when something I rely on stops working.

Report success and alert on silence#

Health checks are a habit I picked up in 2019 at Capital One, where I went through a phase of building them into everything I built. It meant we heard quickly when something we owned broke. I've largely kept the habit up since, and it matters more now than it did then.

The usual way to alert is to catch an error and send a message, which only works if the code is still running and gets far enough to report. A power cut doesn't raise an exception, and neither does a job that stopped being scheduled after an upgrade. An integration whose errors only reach a log nobody reads is just as quiet.

Healthchecks.io turns this around. It's a dead man's switch for scheduled work: it never probes your service, and waits for your service to check in instead. Each check has a URL that whatever you're monitoring requests when it succeeds. You set a period, how often to expect a ping, and a grace time, how much longer to wait before a late ping counts as missing. If the ping still hasn't arrived when the grace time ends, the check goes down and alerts you. The free plan covers 20 checks.

The trade-off is speed. An error alert can fire as soon as something fails, while a heartbeat has to wait for its deadline. I want both.

Keep the watcher outside the thing it watches. If your monitoring runs on the same servers, region or cloud account as your product, it can go down with it, and the power cut in the demo would never reach anyone.

The ping should only go out after the outcome you care about. If Home Assistant had pinged a check just to say it was running, it would have kept pinging through months of missing Octopus data. For a backup, it's two pings around the work. The start ping means a job that hangs also alerts, and sending the exit status reports a failure straight away:

# Illustrative backup job
URL=https://hc-ping.com/your-check-uuid
curl -fsS -m 10 --retry 5 -o /dev/null "$URL/start"
./run-backup && ./verify-backup
curl -fsS -m 10 --retry 5 -o /dev/null "$URL/$?"

Wiring it into my house#

I use Healthchecks.io with alerts to the ntfy app on my phone, and email as a backup. ntfy is free, doesn't need an account, and Healthchecks.io supports it directly.

Every backup job reports its start and exit status, as in the snippet above. Each of my home servers, a Mac mini and two Raspberry Pis, runs a health script every few minutes that checks its containers, disks and services. It pings when everything passes, and sends a failure signal if a problem is still there on the next run, so a container restarting once doesn't wake anyone.

One script also checks that every security camera is still recording, because the camera software shows a dropped camera in its own interface without alerting anyone. If all three servers stop checking in together, whatever took them offline, Healthchecks.io notices from outside.

In the house, Home Assistant is set up to send a top-priority ntfy alert, naming the sensor's location, as soon as a leak sensor gets wet, and smoke, heat and carbon monoxide alarms go the same way. I get an alert after a minute on the 4G backup, and another when fibre comes back, with how long it was out. If the internet seems slow, I know whether the house is on 4G before I start troubleshooting my own machine.

If you only alert on leaks, an offline leak sensor looks exactly like a dry one until the floor gets wet, because a sensor with a flat battery never reports water. So Home Assistant also watches the sensors themselves. If a leak or smoke sensor stops reporting, I hear about it within a few hours, and it's listed again every morning until it's fixed.

Choosing what to watch in an app#

In a product, the same idea matters most for the few things that have to keep working.

1. Ask what would happen if each part stopped for an hour and nobody noticed. The answers that hurt are usually signing in, paying, the product's core action, saving and backing up data, and the emails or webhooks other people rely on.
2. Mark the parts of those paths you don't control, such as the payment provider, sign-in service, email sender, model API and open-source packages. That's where change arrives without a pull request of yours.
3. Define success at the customer's outcome. "Checkout page loaded" tells you little; "payment confirmed and order marked paid" is the event that has to keep happening.
4. Watch scheduled and rarely used work with heartbeats, and busy paths by counting successes against what's normal for that time.

For example:

A flow that nobody uses overnight can't prove anything on its own, so a synthetic run with a test account goes through it on a schedule. Counting successes on busy paths catches partial breakage, which often never shows up as an error. If a button does nothing in one browser, the request never reaches your servers, so they have nothing to log.

The hard part is the threshold. A tight one fires on normal variation, and a loose one lets a partial break slip underneath. With the settings below, the break is never caught; raise the line to 85% and see what that costs before the deploy.

Requiring two low intervals in a row, as the simulation does, cuts false alarms. It adds at least 15 minutes, and more when a partial drop hovers around the line.

Start with whatever would hurt most if it stopped working tonight. Make it ping a URL when it succeeds, decide how long it can stay silent before that counts as broken, and send the alert to your phone. Then break it on purpose and check that your phone goes off.