What are you trying to observe? Just the server and applications? Traffic?

Netata will give you just about every metric you could possibly use for a server. However and anecdotally, netdata seems to spawn a ton of zombie processes (github.com/netdata/netdata/issues/20565) which was quite annoying.

If you’re going for server traffic observability , I recommend ntopng. It’s pretty comprehensive.

Depends on what youre trying to track, but prom can track whatever you want and grafana can display it. You may also want log monitoring as well, which you can also feed in - grafana is just display.

The metrics you want to look at are the big question though.

Im mostly looking for traffic uptime and resourses

It seems to me like you don’t really want historic metrics.

Youre not looking for an observability platform, more a resource monitor like btop. Whether its btop.or one of the other variations of top, there are a lot of them with heavy customization options.

I’d recommend starting there to figure out what you want/need.

What would you recommend for historic metrics? Specifically for podman containers.

They are useful for diagnosing recurring crashes.

Prometheus and Grafana are my typical recommendation there, with the podman exporter. Prometheus for time series data, Grafana to visualize and alert.

One thing I think many folks get wrong is thinking about Grafana (or any other dashboard) as the way to watch for problems, but that requires you to look at them. They should be thought of as a way to evaluate for conditions that there was a problem, then create an alert based on the indicators. This way you can address it before it becomes a problem, or at a minimum be aware that conditions are leading toward a problematic state.

If its repeatable and managed with a container restart or something, you can automate that from the alert, and never have to look again.

If its something you can’t control (for example, availability of a 3rd party service), you can use it as a way to be notified of the status change. I have a custom exporter I wrote for work to keep track of some services that result in a bunch of posts to Teams by people asking if its working for them. So I have an automation that posts to Teams when a service has a problem so my IT guy doesn’t need to respond to all of them individually.

So I’d recommend focusing on good alerts and automations over beautiful dashboards first (though dashboards can be a lot of fun to make!)

I’d suggest VictoriaMetrics over pure Prometheus - better ram footprint is pretty useful nowadays.

They played a good bit fast and loose with their claims some time back (even pulling a few things after an “oopsie - decimal point in the wrong spot!” type issue), and I kind of wrote them off after that. I’ll have to check in on them again.

I’ve used VM in my homelab for a while - it’s genuinely a smaller ram footprint, especially on the ingress path. You can even scale query separately if required.

Hmm, I wonder if I even need a dashboard (other than for fun). Perhaps I should do as you say and first look at what I really need to make sure is working (backups) and then monitor that and set up alerts for it to begin with. Thanks for the insight!

Btop is my favorite top of all the tops. I run it regularly just to scope how things are processing.

Prometheus is mostly for regularly collected statistics and state. what would you recommend for statistics that are only collected when something happens? stats that aren’t available on schedule. as an example, take UPS state changes to battery mode or overvoltage mode. Prometheus can’t collect them when they are short lived.

Depends

To me this is where SNMP traps shine, its quite specifically its purpose. Which means alloy, logstash, snmptrapd + promtail, logstashetc. Which fits best depends on what youre doing, I personally lean simple first. Do you want traps by themselves? Remote logs? Do you have a device thats already handling relays that can be triggered? How does the UPS connect? If its serial or USB over to a dedicated machine, that machine should log and send upstream.

Slow down, punctuate, proofread.

Wonderfully succinct. Thank you for caring about strangers representing themselves best and better.