posted in Selfhosted

My Homelab Got Hacked - A Postmortem – Phunky Cafe

Welp. My Forgejo instance got popped with an RCE two days ago by CVE-2026-60004. Luckily, I noticed the following morning and had the day free to figure out what happened. Let’s dive in!

As a homelab enthusiast, I found this a very interesting post. Here are my take aways from the post that I’m implementing myself:

  • Miner detection. I’ve updated monitoring rules to now watch the CPU on my hosts. If the same thing happened to me I would not have been alerted at all as I’m doing simple up / down monitoring. Fixed.
  • Access logging. I turned on access logging for my homelab Caddy instances.
  • Log retention. I have increased the amount and retention of my logging. The hope is this will help me reconstruct what happened after a breach.
  • Logs offsite. The VPS access logs now ride along with the normal backup process, which runs hourly. The homelab side still only gets caught by the weekly VM backup, so that’s next.
  • Closed an open signup. My webtrees instance (genealogy) had self registration enabled, which is the same door this guy got hit through. Oops. Fixed.
  • Built a tool. log-inventory.sh, so “could I actually reconstruct what happened” is a command I run instead of a thing I assume.
phunky.cafe/my-homelab-got-hacked/
enPage

Replying to @⁨Nickelalloy@lemmy.world⁩

Don’t let this discourage you.

You can self host on your local network just fine and never portforward which is what exposes it to the outside.

I don’t need to watch tv or upload pictures to my server outside of my home. Its perfectly useable.

When you are ready and feel more confident you can setup your own wireguard vpn and only expose that.

Then Your device can connected to that vpn tunnels inside and can acces everything like home.

Currently without vpn your device is at risk every time it connects to a network you don’t own. A third party vpn is not a guarantee your data is safe either. Your home vpn though means all your outside the home networking is fully encrypted and outsiders cant even detect your home adresses is running a vpn to hack. (No ping unless you have the key)

Replying to @⁨webghost0101@sopuli.xyz⁩

This! Good advice! I went this path last year so i can give some extra advice for everyone who wants to start:

Before doing anything, buy a new router and put your iot devices and phones in a isolated guest network. Maintain the router updated. After that, you can work, learn and test without fear. Everything stays in your home.

Then like @webghost0101@sopuli.xyz already said, once you get confident, you can run a vpn tunnel to access your lan. This setup is pretty robust, hard to fuck up, and doable even for newbies. (If i can do it, everyone can, i assure you). Lots of guides out there.

Last advice, which you should already be doing, but setup a backup.

Replying to @⁨Nickelalloy@lemmy.world⁩

I have a lot of stuff self-hosted.

Just no outside access. Well, that’s not quite true, I do have a Tailscale VPN for remote management, but only certain machines are allowed to join, joins require my authorization, and access to the account is via two-factor.

The big thing with external access is to never permit it directly through your local connection. Do it though a Virtual Private Server (VPS) - let it get hit with access attempts. And then only allow specific traffic from the VPS to a specific service in your network using a VPN of some sort.

Replying to @⁨gaylord_fartmaster@lemmy.world⁩

It’s fairly safe as long as you add a strong enough form of access control. For example if you put it behind a VPN, or a SSH tunnel, or require mTLS. Even a key in a custom HTTP header or Basic HTTP auth can be good enough if the key is strong enough.

You can further decrease the probability of drive-by bots reaching a publicly exposed service by merely scanning IPs and ports if you use a reverse proxy and hide your service FQDNs and IP.

You can do this by using TLS certs on wildcard domains rather than explicit domains, using explicit CNAMEs for the service subdomains rather than a wildcard domain, and keeping the A/AAAA records on an obfuscated subdomain rather than the base domain. If the bots can’t figure out a FQDN they’re not getting past the reverse proxy even if they find the IP and port.

This is obfuscation not real security but it cuts down tremendously on bot hits.

Replying to @⁨ExperimentalGuy@programming.dev⁩

It’s a tool to manage docker stacks, e.g. by reading them from git repos. You can configure your git forge in a way to ping Komodo when something in a repo changed, Komodo then deploys the new version. You’d need your own git forge, e.g. forgejo. This is a good guide:

nickcunningh.am/…/how-to-automate-version-updates…

You can start without Renovate to get going, but you’ll want it sooner or later to automate checks for updates.

Replying to @⁨IanTwenty@piefed.social⁩

Yeah it’s useful and I’ve been using it for my home lab, but too many of my containers want outbound connections to the Internet. Like Forgejo needs :443 to be able to mirror or remote push repositories, so either I have to identify and allow list github.com, gitlab.com, etc. or end up allowing 0.0.0.0/0:443 which is not very strong protection.

I’d love for some kind of CNI or network plugin that filtered based on domain name.

Replying to @⁨michaelharley@infosec.pub⁩

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters CA (SSL) Certificate Authority DHCP Dynamic Host Configuration Protocol, automates assignment of IPs when connecting to a network DNS Domain Name Service/System Git Popular version control system, primarily for code SSH Secure Shell for remote terminal access TLS Transport Layer Security, supersedes SSL VPN Virtual Private Network VPS Virtual Private Server (opposed to shared hosting)

[Thread #97 for this comm, first seen 7th Sep 2026, 20:40] [FAQ] [Full list] [Contact] [Source code]

Hachyderm.ioImran Nazar ~ عمران نزر (@Two9A@hachyderm.io)2.93K Posts, 306 Following, 268 Followers · Front-end #typescript developer and general keyboard tapper; author of Internet RFC 7168 (Hypertext Coffeepot Control Protocol for teapots); occasional plumber, more than occasional #c64 #retrocomputing enthusiast, terrible at classical #piano. Header photo is a verdant scene looking over the landscape near Buxton, England, taken from a single-track country road.

Replying to @⁨michaelharley@infosec.pub⁩

I just recently went through a much more benign, but scary nonetheless version of this.

I realized that my Ansible directory, that I had made public on GitHub to share as an example to some folks, had secrets committed and pushed.

It was the direct URL and credentials of an app I developed to store non-PII customer data. Now, it wouldn’t be the end of the world if someone noticed this and scraped the data, but it wouldn’t be good, either.

Luckily, I have Caddy access logs, and it appears no one ever accessed it.

So I pulled the secrets out of the Ansible directory and made the repo private, I rotated the credentials, and installed Crowdsec to monitor Caddy access logs and ban bad actors.

I only noticed the secrets because I had just setup Authelia as an OAuth2 provider for my homelab, and I was adding it to my backup scripts.

Replying to @⁨michaelharley@infosec.pub⁩

My server got hacked once and was mining crypto! I noticed it when I heard the fans were at 100% when walking passed the room lol. Turns out, qbittorrent-nox used UPNP for its web admin endpoint enabled by default and qbittorrent can do many things like run custom scripts. My mistake was not changing its login credentials. Exposed my full system. Live and learn.

Replying to @⁨michaelharley@infosec.pub⁩

About the only recent even I’ve had was when I was sitting at my desk reading an article. I noticed the mouse slightly move, but I hand no hand on the mouse. I sit there watching it, not manually moving the mouse or typing on the keyboard. It really freaked me out. Turns out, my old desk pad had a slight curl towards the edge the mouse was at, and the curl would make the mouse sense movement ever so slightly. I deployed a new desk pad, and things are back to normal. Whew!

Replying to @⁨michaelharley@infosec.pub⁩

Yiiiiiikes. This is why I’m gonna be stuck on tailscale or another VPN only access for quite a while. I know it’s not perfect, but exposing ports on today’s Internet is scary.

I know I don’t have the mental overhead to secure literally everything, and have a solid contingency for when some botnet armed with a zero-day inevitably breaks it.

Really glad this person wrote this write up about how that happens and what it looks like though. It’s REALLY interesting and I learned a lot!