Monitoring and logging
Monitoring tells you what’s broken and helps you learn why. It’s also one of the most employable skills you can practice in a lab. Start small, and only add what you’ll actually look at.
Where to start
Section titled “Where to start”- Check whether your services are up. Uptime Kuma takes minutes to set up, and sends notifications to your phone.
- Watch the hosts. Beszel and Netdata are quick ways to see CPU, memory, disk use and more.
- Watch your drives. Set up SMART alerts, and ZFS scrub and error alerts. Failing disks are your most likely hardware problem.
- Check that your backups ran. Use an alert such as Healthchecks, which notifies you when an expected job doesn’t report in.
Bigger tools
Section titled “Bigger tools”| Tool | Type | Notes |
|---|---|---|
| Prometheus and Grafana | Metrics and dashboards | The modern standard, particularly with Kubernetes. Well worth learning |
| Zabbix | All-in-one monitoring | Common in businesses, with a lot of built-in templates. Takes some time to set up |
| LibreNMS | Network monitoring | Excellent auto-discovery for switches, routers and other SNMP devices |
| Checkmk | All-in-one monitoring | Has a free community edition with auto-discovery |
| Icinga | Infrastructure monitoring | A modern tool with roots in Nagios |
| Nagios Core | Infrastructure monitoring | The old classic. Still around, but newer tools are often easier to start with |
| Observium | Network monitoring | Has a free community edition with delayed updates |
Logging
Section titled “Logging”Central logging lets you search everything in one place, and it preserves logs if a machine dies or is compromised.
| Tool | Notes |
|---|---|
| Grafana Loki | Lightweight. Indexes labels rather than the full text of each log line. Pairs well with Grafana |
| Graylog | Full-featured log management. Needs more resources |
| OpenSearch or Elastic Stack | Very capable, and common in business. They use a lot of RAM |
| VictoriaLogs | Efficient and relatively new |
To send logs, use Alloy, Fluent Bit, or Vector, or just use plain rsyslog if you prefer to keep it simple.
Alert carefully
Section titled “Alert carefully”Too many alerts soon get ignored. Alert on what you need to act on, like a failing disk, a failed backup, or a service down for more than a few minutes. Use dashboards for everything else. ntfy and Gotify are self-hosted options for push notifications, and most tools can send email or chat messages as well.
If your monitoring runs on the same host that it monitors, it can’t tell you when that host dies. An external check, such as a small monitor on a cheap VPS or a free service, covers that gap.