Just use Tailscale for your VPN. If you want to be 100% FOSS self hosted, set up Headscale as your control server somewhere. It’s simple, secure, and has excellent performance.
EDIT: And also check what your upload speed is from your server. It sounds like you’re running off your home internet connection and many ISPs have terrible upload speeds, particularly for sustained traffic.