Any external NPU that can be hooked to a proxmox miniPC?

Hi! So I’m considering…maybe having an NPU or something similar to be hooked to my proxmox server, which runs in a mini PC. It’s a EliteDesk 800 micro form factor. It has a Core i5 8500 CPU, which at the moment of purchase was good enough for live encoding HEVC video on Jellyfin…that was my main concern back then. But I’d like to consider the possibility of hooking maybe some docker instances or other containers to some local-only AI acceleration. Is there any NPU or cheap GPU I could hook on USB to this proxmox server to run? Has it been done before?

Thanks!

Do you want to run TensorFlow Lite / LiteRT models? PyTorch Mobile? TensorRT? onnx? YOLO? vLLM? Something else? The recommendations will vary based on your use case.

Google Coral was decent for TensorFlow Lite, but it’s EOL (end of life) now. I’ve got the dual TPU Mini PCIe version in my home server, via a PCIe adapter board. I use it for object detection with Blue Iris + CodeProject AI and it works pretty well for that use case.

Hailo-8 is supposed to be like a more powerful version of the Coral, but I don’t have experience with it. It supports a bunch of frameworks: TensorFlow, TensorFlow Lite, Keras, PyTorch and ONNX. I’d be interested in hearing other people’s thoughts on it.

I don’t know if any of these work over USB though. They’re usually internal devices. Google marketed the Coral USB as being for development and testing only, pointing people to the M.2 and PCIe versions for production usage.

As for something totally different… There’s the Nvidia Jetson single board computer which supports TensorRT, but I don’t have experience with it either. I also think it’s a bit older too. You could also consider getting a newer mini PC with a AMD Ryzen AI processor in it, or an Nvidia DGX Spark.

Google’s latest TPUs are only available in their cloud - they’re not selling the hardware to end users any more.

Thanks! I was considering using it perhaps to have a diffusion model? Or running ollama or similar without a RAM hit on my laptop or phone. Also whisper comes to mind, for Bazarr or other tools to use.

Aren’t diffusion models and LLMs (ollama) too big for an external NPU? As far as I know something like a Coral runs specific models only. And it’s limited to the 1 or maybe 2GB of memory on it. It’d do tasks like voice recognition, or image classification. But not generate images or text.

If you want to run arbitrary AI models and generative AI, I think you should be looking for a graphics card?!