Observability Stack?

I’m curious as to which tools and technologies you all are using to keep track of all those services you are deploying, whether it be resource tracking, network traffic, logs, traces, or uptime.

As a bonus question, how have you organized your network or your services to reduce the overhead of implementing observability?

I’ve got grafana and Prometheus, but I’m currently working on a generic log pulling script that will then feed to a local ollama instance to summarize the logs and let me know if there are any potential issues I should be aware of. Grafana is great for current state, but I’m a big believer in watching the logs to see trends, but I’m too lazy to do it by hand.

Once that’s up and running and steady, I’ll probably tie it into ntfy to actually alert me in real time