I have a deep learning box that runs various computations on both CPU and GPU. The ability to off-load expensive computations from my laptop is fantastic.
I deploy my deep lerning box using NixOps, running Nix 16.09 and monitor it with prometheus.
I created a small tool, nvidia-smi-prometheus which runs as a service and reads the NVidia system stats every 10 seconds, and then exports them for prometheus.