I have the homepage widget. Basic but does the job of CPU, memory, and storage.

I enabled docker integration by passing docker.sock to the hompage container to see per services memory usage. Nice for spotting memory hogs.

I forgot to look into that! I use glance for my homepage now, and I think they have a Grafana community widget that I could take a look at. I have a bunch of bookmarks of community widgets that I need to look at lol

I mean, I got the whole set of default dashboards, some for networking and one each for backups (k8up) and Postgres (CloudNativePG), but I don’t look at them often, usually only if something’s wrong. I also have logs available via Grafana, but I don’t have any dashboards for that, I just query what I need. If I ever have enough time and motivation, some things I’d put on a kind of overview dashboard would be in addition to your list:

• zpool health (scrub status, IO etc.)
• list of crashing pods
• list of warnings from FluxCD (e.g. application upgrade failures or invalid manifests)
• list of failing persistent volumes

I think aggregating logs would be a good choice, yeah. My homelab is distributed across 5 machines, so it’s been getting a little unruly to ssh into them all the time. I actually haven’t taken a look at premade dashboards - completely forgot those exist. I’ve just been writing the PromQL by hand to learn how it all works before adding premade things

Yeah, for most things, a premade dashboard exists, and if it isn’t fully to your liking, just customize it. Also for more professionally maintained metrics exporters like the node-exporter, there’s usually an official dashboard. But learning PromQL before relying on premade stuff is probably a good idea, so you know how to adapt them to your needs

Right now I’m just using it for an AirGradient dashboard, which is an open hardware / open software air quality monitor. I used claude to rewrite the firmware to send the data to prometheus instead if their own servers, and countless other frustrating changes because they make the whole operation dependent on a connection to their servers…anyway, I didn’t want to pay a subscription for something so simple. I’ve thought about using it for some other projects, but for a lot of stuff I just don’t see the need for a dedicated dash.

Definitely something to add to the list, environment monitoring sounds like fun

I have a series of Tasmota esp32’s with Temp/Humidity/Barometric Pressure sensors around the house, attic, garage and outside. I scrape it into influxdb and have a Grafana dashboard setup.

How much does each sensor wind up costing, esp32 and all? That sounds really interesting. I think I have an esp32 laying around that I haven’t gotten to playing with yet.

I found a pack of 5 on Amazon for $20, for each of the sensors and the esp32’s… so $40 for 5. Probably cheaper in Ali or similar. The only negative is they need some calibration. The ones from Adafruit are better quality and already calibrated, but much more expensive.

Thats really cool.

Is there any reason you chose grafana over home assistant?

I’m just setting uo a similar system myself, but I never even had the idea to use grafana for it

I already had influxdb+grafana setup, so it was easy to pump data to, plus I was able to then match it up with cpu temps and such from my Kubernetes nodes. I like having all my data in one place.

That said, Tasmota has native Matter support and I now have a few ESP32’s controlling sockets, lights, etc., that integrate with Home Assistant.

• Uptime
• Days since last system update
• Days since last CVE scan

Ah, those are good ones. I’m a little too neglectful with updates, haha.

I think it’s pretty cool in beszel you can set it up using docker so easily and pass the disks through for SMART monitoring and not have to have an extra solution.

Beszel is really cool! I was actually planning to set that up at first, but I really wanted to learn Grafana / Prometheus and have the flexibility to customize things how I wanted. That’s definitely my biggest annoyance with using Grafana - I have five machines setup, and each one has 3 containers for monitoring (cAdvisor, smartctl_exporter, node_exporter), so it got pretty tiring to put up so many containers.

Other than the items you mentioned I have dashboards for:

• SSH/SMTP honeypot
• Jellyfin
• Pihole
• Torrents
• Tasmota (smart plugs)

What kind of things do you monitor with the Jellyfin dashboard? Would definitely be cool to put up monitoring for Transmission. I was thinking about putting up an Adguard instance to monitor it lol

Jellyfin exporter has all kind of metrics but the ones that interest me most are active users and media played. I have a nice panel that shows who played what, when and for how long. Would also highly recommend monitoring your torrent server, that has produced all kinds of useful and interesting (to me!) metrics.

I’d add backup age and size, not just status, like alert if the newest dump is older than ~26h or smaller than 1MB, a job can say success and still leave you a useless file. and disk fill with a threshold, not only the graph. learned that one the hard way, a table on my vps was getting a row every minute with no cleanup and one day the disk hit 100% and postgres just died for like 40 min, obviously nobody was looking at the dashboard right then lol. now 80% warns and 90% sends me an email, and uptime kuma checks the cert expiry from outside. I guess the dashboard is nice to look at but the stuff that matters should ping you

Size is a great idea, thanks. The problem with backup age is that Backrest doesn’t expose a metric for when the backup was attempted, so right now it’s just checking for failure status daily. That, of course, fails if backrest doesn’t attempt a backup for whatever reason. Chances of that happening, I don’t know, but I’ll have to look into implementing the metric (or label, I suppose) myself.

What I do for that is turn it around, instead of checking for a failure I alert on the absence of a success. backrest has hooks, so on snapshot success it pings a push monitor in uptime kuma (healthchecks.io works the same), and if no ping arrives in ~26h it alerts. That way it catches the backup that never ran, which is the one that scares me the most I guess

• grafana dashboards github.com/nodiscc/xsrv/tree/master/…/files
• victoriametrics alerting rules github.com/nodiscc/xsrv/tree/master/…/files

ALTNg1yNptacZzyuIC.png

usually add a bunch of random stuff, then forget it exists, only to check it half a year later and discover that 75% of prometheus exporters died for some reason