Hello selfhosters.
We all have bare-metal servres, VPS:es, containers and other things running. Some of them may be exposed openly to the internet, which is populated by autonomous malicious actors, and some may reside on a closed-off network since they contain sensitive data.
And there is a lot of solutions to monitor your servers, since none of us want our resources to be part of a botnet, or mine bitcoins for APTs, or simply have confidential data fall into the wrong hands.
Some of the tools I’ve looked at for this task are check_mk, netmonitor, monit: all of there monitor metrics such as CPU, RAM and network activity. Other tools such as Snort or Falco are designed to particularly detect suspicious activity. And there also are solutions that are hobbled together, like fail2ban actions together with pushover to get notified of intrusion attempts.
So my question to you is - how do you monitor your servers and with what tools? I need some inspiration to know what tooling to settle on to be able that detect unwanted external activity on my resources.
I’m pretty old school, but as I only have 1 server, I just use
ssh
,df
,du
andtop
.Not even htop? That is old school.
Not even btop? That’s middle school.
Not even bottom? That’s elementary school.
Okay priest.
Prometheus.
It’s open source, it’s easy to setup, its agents are available for nearly anything including OpenWrt, it can serve the simplest use case of “is it down” as well as much more complicated ones that stem from its ability to collect data over time.
Personally I’m monitoring:
- Is it up?
- Is the storage array healthy?
- Are the services I care about running?
I used to run it ephemerallly - wiping data on restart. Recently started persisting its data so I can see data over the longer run.
I run Prometheus on a separate cluster, so I plug my servers with node_exporter and scrape metrics. I then alert with grafana. To be honest, the setup is heavier (resource usage-wise) than I would like for my use case, but it’s what I am used to, and scales well to multiple machines.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters DNS Domain Name Service/System SSL Secure Sockets Layer, for transparent encryption VPN Virtual Private Network VPS Virtual Private Server (opposed to shared hosting)
3 acronyms in this thread; the most compressed thread commented on today has 9 acronyms.
[Thread #421 for this sub, first seen 10th Jan 2024, 14:55] [FAQ] [Full list] [Contact] [Source code]
Sometimes I just sit and stare at my apache access logs because I’m bored
GoAccess is pretty nice for a broad overview of Apache logs, also.
For other services I generally just look at them every now and then and if something looks off I investigate. I found a cryptominer on my network once because it was spamming DNS and that shows up in DNS logs.
I’m running checkmk for monitoring but that won’t help you with detection of unwanted logins. For security I’m running crowded.