Neural Magic Blog

vLLM supports MLA for Deepseek models as of 3 weeks ago. 3x higher generation throughput and 10x token memory capacity. https://github.com/vllm-project/vllm/releases/tag/v0.7.1 MHA is still faster in low QPS regime apparently. https://neuralmagic.com/blog/enhancing-deepseek-models-with-mla-and-fp… Also published this month was theoretical proof showing that for the same KV Cache overhead, MLA consistently offers greater expressive power than GQA. Furthermore, widely used GQA-based pre-trained models (e.g. LLaMA, Qwen, Mixtral) can be converted into MLA-based models. https://arxiv.org/pdf/2502.07864

Look into sparsification of ML models: https://neuralmagic.com/ If your cpu has avx512 it can make it even faster.

I asked the author to compare the NN-512 dense convolver (https://NN-512.com) in his benchmark, as an alternative to Intel's closed source OpenVINO Neural Magic (https://neuralmagic.com/) is also doing very sophisticated sparse convolution (closed source) on CPUs, and they benchmark against NN-512 Neural Magic achieves astonishing performance by exploiting sparsity

Disclosure: I work for Neural Magic. Hi ml_hardware, we report results for both throughput and latency in the blog. As you noted, the throughput performance for GPUs does beat out our implementations by a bit, but we did improve the throughput performance on CPUs by over 10x. Our goal is to enable better flexibility and performance for deployments through more commonly available CPU servers. For throughput costs, this flexibility becomes essential. The user could scale down to even one core if they wanted to, with a much more significant increase in the cost performance. We walk through these comparisons in more depth in our YOLOv3 blog: https://neuralmagic.com/blog/benchmark-yolov3-on-cpus-with-deepsparse/ INT8 wasn't run on GPUs because we have issues with operator support on the conversion from PyTorch graphs to TensorRT (PyTorch currently doesn't have support for INT8 on GPU). We are actively working on this, though, so stay tuned as we run those comparisons! The models we're shipping will see performance gains on the A100s, as well, due to their support for semi-structured sparsity. Note, though, A100s are priced more expensive than the commonly available V100s and T4s, which will need to be considered. We generally keep our current benchmarks limited to what is available in the top cloud services to represent what is deployable for most people on servers. This usability is why we don't consider ML Perf a great source for most users. ML Perf has done a great job in…

They do. It's on the third graph: https://neuralmagic.com/wp-content/uploads/2023/11/CHART-Llama2-Sparse…

I think you are missing something, or I am. If we observe the performance comparison graph (1), 2.8->9 tok/s is achieved via quantization, but the remaining jumps from 9->16.6->24.6 tok/s are achieved from the sparse fine-tuning. (1) https://neuralmagic.com/wp-content/uploads/2023/11/CHART-Llama2-Sparse…

For anyone interested in SparseGPT, on May 25th, the author of the SparseGPT paper will show you how you can download an optimized and open-sourced LLM and run it on CPUs at GPU speeds using DeepSparse. Confirm your spot: https://neuralmagic.com/unlock-faster-and-more-efficient-language-mode…

Language models have a limit on the length of text they can process. For instance, 512 for BERT and 4096 for models such as Longformer and Big Bird. However, documents in the real world can be arbitrarily long. For instance, reviews from customers. Classifying these reviews fast can help a business in engaging with its customers quickly. Fort instance, flagging negative reviews to reduce churn and customer complaints. The problem? Language models are usually very large making them slow at inference and difficult to deploy. These models are usually overprecise and over-parameterized. You can drop some of the weight connections in these networks to achieve a small model while maintaining accuracy. This is known as sparsification. Furthermore, you can reduce the precision of the floating points in the network to reduce its size further. In my latest article, I explore how to perform text classification on long documents using sparse Hugging Face Transformers. I illustrate that it’s possible to use a 90% sparse transformer model— meaning that 90 percent of the model's weights are removed, and the rest of the parameters quantized— and still achieve accuracy similar to a dense model. Using a sparse transformer achieves a 4.8X increase in performance over the dense baseline. It also results in a smaller model which is easy to deploy on commodity CPUs, hence no expensive accelerator hardware is required. Try it yourself…

Neural Magic | Machine Learning Engineer, Full Stack Engineer, Product Manager, ML Ops Engineer | Somerville, MA | https://www.neuralmagic.com Neural Magic is a software company challenging the notion that specialized hardware like GPUs is needed for Deep Learning inference. We have created novel technologies that sparsify and quantize deep learning networks and allow them to run on commodity CPUs – at GPU speeds and better. Data scientists no longer have to compromise on model design and input size or deal with scarce and costly GPU resources. Their ground-breaking discovery became the foundation of Neural Magic. Having just closed our 30M Series A (https://neuralmagic.com/blog/neural-magic-series-a/), we are expanding our team and hiring across most disciplines. Some of our recent work has focused on compressing vision systems like YOLO and NLP systems like BERT. https://neuralmagic.com/blog/benchmark-yolov5-on-cpus-with-deepsparse/ https://neuralmagic.com/blog/sparsify-hugging-face-bert-for-better-cpu… * Senior Systems Software Engineer | https://apply.workable.com/neural-magic/j/85853A87F8/ * Machine Learning Engineer | https://apply.workable.com/neural-magic/j/4509AD76E3/ * Senior Machine Learning Engineer | https://apply.workable.com/neural-magic/j/3EA60B5DA4/ * Product Manager | https://apply.workable.com/neural-magic/j/1BB929F472/ * ML Ops | https://apply.workable.com/neural-magic/j/3E54400904/ * Full Stack Software Enginner |…