My setup consists of an x86 “network appliance”. They’re low-cost, low-power, sometime headless units that can run most any open-source software solution. Even with RAM as bad as it is, you can find these for less than $200 from multiple sources if you’re okay with 2.5Gbps. If you did need more than that, you’d have to build something custom likely, unless it’s a small-business class device.
If you’re just getting into the network software side of things, I recommend experimenting off-Internet for a while with a few different operating systems. Find which one has a user-interface you enjoy and can wrap your head around since in a personal homelab, they almost all meet the requirements. My personal choice for software ended up being OPNSense.
I run all of my other services that aren’t network related on separate hardware via a managed switch just to keep things isolated and the rest is up to you.