I’m still having trouble with it and I’m currently traveling. My analysis so far points to my vps provider being at fault.
I tried doing long term diagnostics, which effectively pinged through the tunnel every 5s and that make it work constantly and perfectly. So maybe some energy saving sleep stuff, which ends up breaking the tunnel?