SSL Certificates and Backups That Fail Silently: How to Get Alerted
The two failures that hurt a hosting server most are also the quietest. A certificate that doesn't renew works perfectly until the day it expires, and then every visitor gets a security error. A backup that stops running goes unnoticed until the day you need it. The rule is simple: don't trust that something automatic works, check its result. For certificates, look at how many days are left, not whether certbot is installed; for backups, look at when the last successful backup finished, not whether the cron job exists. Here are the checks, ready to copy.
Why nobody warns you anymore
Until 2025 Let's Encrypt emailed you when a certificate was about to expire without being renewed. That service has been shut down: today, if automatic renewal breaks, the only warning you get is a customer who can't open their site. Let's Encrypt has also announced that certificate lifetimes will gradually drop from 90 to 45 days by 2028: renewals will happen more often, and a broken renewal will hurt sooner.
Backups are worse: no backup tool emails you "I didn't run last night". If the cron job disappears after a migration, the storage password changes or the destination disk fills up, the backup simply isn't there.
1. How many days before a certificate expires
The right check is from the outside, the way a browser sees it:
echo | openssl s_client -servername www.example.com -connect www.example.com:443 2>/dev/null \
| openssl x509 -noout -enddate
The answer is a date like notAfter=Dec 24 10:15:02 2026 GMT. To know whether it expires within 14 days, openssl has an option that returns an exit code:
echo | openssl s_client -servername www.example.com -connect www.example.com:443 2>/dev/null \
| openssl x509 -noout -checkend $((14*86400)) || echo "EXPIRES WITHIN 14 DAYS"
Why 14 days? Certbot tries to renew about 30 days before expiry. If the old certificate is still there with 14 days left, renewal has been broken for more than two weeks and you still have time to fix it.
2. One daily check for every site
Put the domains in a file and check them all with a script:
#!/bin/bash
# /usr/local/bin/check-certificates
DAYS=14
while read -r domain; do
[ -z "$domain" ] && continue
end=$(echo | timeout 10 openssl s_client -servername "$domain" -connect "$domain:443" 2>/dev/null \
| openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
if [ -z "$end" ]; then
echo "$domain: no readable certificate"
continue
fi
left=$(( ($(date -d "$end" +%s) - $(date +%s)) / 86400 ))
[ "$left" -lt "$DAYS" ] && echo "$domain: expires in $left days ($end)"
done < /etc/domains-to-check.txt
cron emails a job's output to you if the server can send mail. One line in crontab -e is enough:
MAILTO=you@example.com
30 7 * * * /usr/local/bin/check-certificates
The script prints nothing when all is well, so you only get an email when something is wrong.
3. Finding out why certbot doesn't renew
When a certificate doesn't renew, the checks are always the same:
systemctl list-timers | grep -i certbot # does the timer exist and is it active?
sudo certbot renew --dry-run # would renewal work right now?
sudo tail -50 /var/log/letsencrypt/letsencrypt.log
There are three usual causes: the domain no longer points to this server (the customer changed DNS), port 80 is closed by the firewall or a CDN in front of the site, or the site's nginx configuration was changed and the /.well-known/acme-challenge/ check no longer reaches where certbot expects it.
4. Backups: check the age of the last successful one
The useful check isn't "did the backup start" but "when did the last successful backup finish". With restic:
restic snapshots --latest 1 --json | jq -r '.[-1].time'
And a check that warns you when the last backup is older than 26 hours (one day plus a margin):
#!/bin/bash
# /usr/local/bin/check-backup (RESTIC_REPOSITORY and RESTIC_PASSWORD_FILE in the environment)
last=$(restic snapshots --latest 1 --json | jq -r '.[-1].time // empty')
if [ -z "$last" ]; then echo "No backup in the repository"; exit 1; fi
hours=$(( ($(date +%s) - $(date -d "$last" +%s)) / 3600 ))
[ "$hours" -gt 26 ] && echo "Last successful backup $hours hours ago ($last)"
Once a week, also check that the data in the repository can be read, a rotating part at a time:
restic check --read-data-subset=5%
The full guide to setting up restic backups on S3 storage is in back up websites to S3.
5. The dead man's switch: an alert when something does NOT happen
Check scripts run on the same server as the backups: if the server goes down, they don't warn you. The fix is an external service that waits for a periodic signal and alerts you when the signal doesn't arrive. Services like Healthchecks.io work this way: add a call at the end of the backup, only if it succeeded.
restic backup /var/www /var/backups/mysql && curl -fsS -m 10 --retry 3 https://hc-ping.com/YOUR-UUID
If the signal doesn't arrive for 26 hours (backup failed, cron job gone, server off), you get the alert. It's the most robust check of all, because it doesn't depend on the server it checks.
6. The only real proof: a restore
A backup you've never restored is a hypothesis. Once a month, restore a random site into a temporary folder and check that the files and database are there:
restic restore latest --target /tmp/restore-test --include /var/www/example.com
ls -la /tmp/restore-test/var/www/example.com
Then delete the folder. Ten minutes a month, and you know it will work when you need it.
With Prometheus
If you already run Prometheus, blackbox_exporter checks certificates with the probe_ssl_earliest_cert_expiry metric, and you can check backups by exposing the age of the last snapshot through node_exporter's textfile collector. We explain it in monitor a web hosting server with Prometheus and Grafana.
What Koapanel does
On a server running Koapanel these checks are already there:
- Expiring certificate: if automatic renewal hasn't succeeded with less than 14 days left, the administrators get an email naming the site.
- Failed backup: an email for every failed backup, and the last successful backup of each site is shown in the all-sites list.
- Disk almost full: an email above 90%.
- The same alert arrives at most once a day, so your inbox doesn't fill up (System emails).
Since version 0.27 the same information is also available as Prometheus and Grafana metrics, including a "no successful backup for two days" rule per site (API and webhooks).
What's left to you: an external availability check (the dead man's switch or an uptime service) and an occasional restore test. The panel restores a site in one click from the Backups page, so the test takes a minute.
FAQ
How often should I check certificates?
Once a day is enough. With a 14-day threshold you get thirteen checks before expiry.
The certificate looks valid on the server but the browser says it expired: how?
Usually a CDN or proxy in front of the site serves its own certificate, or the domain points to another server. That's why the check must run from the outside, with openssl s_client on the site name, not by reading the certificate file on disk.
Are my VPS provider's backups enough?
They help if you lose the whole server, but they're usually daily, kept for a few days and restore the whole server. To bring back one site or yesterday's database you need your own per-site backup, off the server.
How often should I test a restore?
At least once a month and after every significant change (new storage, new password, migration).
Try Koapanel
Look around the public demo or install it on a fresh Ubuntu 24.04, free for up to 3 sites:
curl -fsSL https://get.koapanel.app | sudo bash