Even without sparsification, AVX-512 CPUs are far more cost-effective for inference.
To get your money's worth, you must saturate the processor. It's easy, in practice, with an AVX-512 CPU (e.g., see https://NN-512.com or Fabrice Bellard's https://bellard.org/libnc), and almost impossible with a GPU.
A GPU cloud instance costs $1000+ per month (vs. $10 per month for an AVX-512 CPU). A bargain GPU instance (e.g., Linode) costs $1.50 per hour (and far more on AWS) but an AVX-512 CPU costs maybe $0.02 per hour.
That said, GPUs are essential for training.
Note that Neural Magic's engine is completely closed-source, and their python library communicates with the Neural Magic servers. If you use their engine, your project survives only as long as Neural Magic's business survives.
> A GPU cloud instance costs $1000+ per month (vs. $10 per month for an AVX-512 CPU). A bargain GPU instance (e.g., Linode) costs $1.50 per hour (and far more on AWS) but an AVX-512 CPU costs maybe $0.02 per hour.
This is a little confused. A T4 on GCP is $.35/hr at on-demand rates. A single thread of a Skylake+ CPU is in the 2-3c/hr range [2] (so $15-20/month w/o any committed or sustained use discounts. Your $10 is close enough).
So roughly the T4 GPU itself is ~10 threads. Both of these are before adding memory, storage, etc., but the T4 is a great inference part and hard to beat.
Comparing a single thread of a CPU to training-optimized GPU parts (like the A100 or V100) is sort of apples and oranges.
If you're not holding it for a full month, it's $0.015 per hour.
Here are the $1000 per month ($1.50 per hour) cloud GPUs I use, the cheapest that Linode provides: https://www.linode.com/pricing/
I would like to see a comparison of Linode vs Google Cloud, taking into account network bandwidth costs, etc. Maybe the Google Cloud T4's total cost of ownership is lower? I doubt it: for example, the Linode plan includes 16 terabytes of fast network traffic at no additional cost, and Google Compute charges $0.085 per gigabyte.
And the T4 is still 23x more expensive than an AVX-512 core, hourly.
boulos is 100% right that you are choosing to compare to particularly expensive GPUs instead of cheaper GPU instances that are actually intended for inference at low cost.
> Note that Neural Magic's engine is completely closed-source, and their python library communicates with the Neural Magic servers. If you use their engine, your project survives only as long as Neural Magic's business survives.
Could be totally off here, but have a feeling this team is going to get scooped up by one of the big co's. Having deja vu of XNOR.ai (purchased by Apple), when partners got the short end of the stick (see Wyze cam saga: https://www.theverge.com/2019/11/27/20985527/wyze-person-det...)
Hi 37ef_ced3, AVX-512 has certainly helped close the gap for CPUs vs GPUs. Even then, though, most inference engines on CPUs are still very compute bound for networks. Unstructured sparsity enables us to cut the compute requirements down for CPUs leading to most layers becoming memory bound. This then allows us to focus on the unique cache architectures within the CPUs to remove the memory bottlenecks through proprietary techniques. The combination is what's truly unique for Neural Magic and enables the DeepSparse engine to compete with GPUs while leveraging the flexibility and deployability of highly available CPU servers.
Also as boulos said, the numbers quoted here are a bit lopsided. There are more affordable GPUs and GCP has enabled some very cost effective T4 instances. Our performance blog on YOLOv3 walks through a direct comparison to T4s and V100s on GCP for performance and cost:
https://neuralmagic.com/blog/benchmark-yolov3-on-cpus-with-d...
Additionally, I'd like to clarify that our open sourced products do not communicate with any of our servers currently. These are stand alone products that do not require Neural Magic to be in the loop for use.
Agreed - I priced this out for a specific distributed inference task a few months ago and the T4 was cheaper and faster than GPU.
On 26 million images using a pytorch model that had 41 million parameters, T4 instances were about 32% cheaper than CPU instances, and took about 45% of the time even after accounting for extra GPU startup time.
Many businesses/services can't saturate the hardware you describe. It's just too much compute power. With CPUs you can scale down to fit your actual needs: all the way down to a single AVX-512 core doing maybe 24 inferences per second (costing a few dollars PER MONTH).
Also, your cost/inference results will change if you use a fast CPU inference engine, instead of something slow like PyTorch (which you appear to be using).
Thanks - this is something I wasn’t familiar with. Do you have any pointers for CPU inference engines that you’ve had good experience with or that I can look into further?
To get your money's worth, you must saturate the processor. It's easy, in practice, with an AVX-512 CPU (e.g., see https://NN-512.com or Fabrice Bellard's https://bellard.org/libnc), and almost impossible with a GPU.
A GPU cloud instance costs $1000+ per month (vs. $10 per month for an AVX-512 CPU). A bargain GPU instance (e.g., Linode) costs $1.50 per hour (and far more on AWS) but an AVX-512 CPU costs maybe $0.02 per hour.
That said, GPUs are essential for training.
Note that Neural Magic's engine is completely closed-source, and their python library communicates with the Neural Magic servers. If you use their engine, your project survives only as long as Neural Magic's business survives.