# Neural Magic

Neural Magic is a software company specializing in high-performance inference for AI models on commodity CPUs, using sparsity and quantization to achieve GPU-like speeds without specialized hardware.

Neural Magic is a software company that develops technologies for accelerating [machine-learning](https://www.wikiprompt.org/wiki/machine-learning) inference on commodity CPU hardware. Founded in 2017 and headquartered in Boston, Massachusetts, the company focuses on making [artificial-intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) deployment more accessible and cost-effective by eliminating the need for specialized accelerators such as GPUs. Its core approach combines algorithmic innovations in model sparsity and quantization with CPU-specific optimizations, enabling deep neural networks to run efficiently on standard x86 processors. The company's solutions are used across industries for tasks including computer vision, natural language processing, and large-scale [generative-ai](https://www.wikiprompt.org/wiki/generative-ai) workloads.

The company emerged from research at [mit-csail](https://www.wikiprompt.org/wiki/mit-csail) and [berkeley-ai-research](https://www.wikiprompt.org/wiki/berkeley-ai-research), where its founders explored ways to exploit the mathematical structure of neural networks to reduce computational demands. Neural Magic's technology is built on the observation that many neural networks contain significant redundancy, and by pruning or quantizing weights, inference can be performed with far fewer operations. The company's software stack, including its DeepSparse runtime and SparseML toolkit, allows developers to compress and accelerate models without requiring custom hardware. This approach has positioned Neural Magic as a notable player in the AI infrastructure space, particularly for organizations seeking to avoid the high costs and supply constraints of GPU clusters.

## History and Founding

Neural Magic was founded in 2017 by [mark-chen](https://www.wikiprompt.org/wiki/mark-chen), [jakob-uszkoreit](https://www.wikiprompt.org/wiki/jakob-uszkoreit), and [lukasz-kaiser](https://www.wikiprompt.org/wiki/lukasz-kaiser), three researchers with backgrounds in machine learning and systems. Uszkoreit and Kaiser were early contributors to the [transformer](https://www.wikiprompt.org/wiki/transformer) architecture at Google, and Chen had worked on efficient inference techniques. The trio aimed to commercialize research on sparse neural networks, which had shown that models could be pruned to a fraction of their original size while retaining accuracy. The company initially operated in stealth, publishing academic papers and developing its core technology before publicly launching its first products in 2020.

In 2020, Neural Magic released its DeepSparse inference engine, which demonstrated significant speedups for convolutional and transformer models on CPUs. The company also introduced SparseML, an open-source library for applying sparsity and quantization to models during training. These releases attracted attention from enterprises and cloud providers, leading to partnerships with companies like [intel](https://www.wikiprompt.org/wiki/intel) and [amd](https://www.wikiprompt.org/wiki/amd). By 2021, Neural Magic had raised over $50 million in funding from investors including NEA, Andreessen Horowitz, and Pillar VC. The company continued to expand its product line, adding support for [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) inference and integrating with popular frameworks such as PyTorch and Hugging Face Transformers.

## Technology and Approach

Neural Magic's core technology revolves around two main techniques: sparsity and quantization. Sparsity involves pruning a neural network's weights, setting many of them to zero, which reduces the number of multiplications required during inference. Quantization reduces the precision of weights and activations from 32-bit floating point to 8-bit integers, further cutting computational cost and memory bandwidth. The company's software automatically applies these techniques to models, often with minimal loss in accuracy, and then generates optimized code for the target CPU.

The DeepSparse runtime is the engine that executes these compressed models. It uses a technique called sparse matrix multiplication, which skips zero entries, and leverages CPU vector instructions such as AVX-512 and VNNI to accelerate the remaining operations. The runtime also includes a just-in-time compiler that generates specialized kernels for each model and hardware configuration. This approach allows Neural Magic to achieve inference speeds that rival or exceed those of GPU-based systems for certain workloads, particularly for batch inference and real-time applications.

SparseML, the companion training library, provides APIs for pruning and quantization during model training or fine-tuning. It supports a technique called gradual magnitude pruning, which progressively removes weights based on their magnitude, and quantization-aware training, which simulates low-precision arithmetic during training to preserve accuracy. These tools are designed to be easy to integrate into existing machine-learning pipelines, making the technology accessible to a broad range of developers.

## Products and Services

Neural Magic offers several products under the DeepSparse and SparseML brands. The DeepSparse runtime is available as a Python package and a Docker container, and it supports both CPU and cloud deployments. It includes a server mode for serving models via REST APIs, as well as a client library for embedding inference into applications. The company also provides pre-optimized model zoo, a collection of popular models that have been pruned and quantized, ready for immediate deployment.

SparseML is an open-source library that integrates with PyTorch and TensorFlow, offering command-line interfaces and Python APIs for applying sparsity and quantization. It also includes recipes for common models like [residual-network](https://www.wikiprompt.org/wiki/residual-network) and transformer variants, which specify the pruning schedule and hyperparameters. In addition, Neural Magic offers a cloud service called DeepSparse Cloud, which provides managed inference endpoints with autoscaling, though this service was later deprecated in favor of on-premises deployments.

For [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) inference, Neural Magic has developed specialized optimizations for models like Llama and Mistral. These optimizations include KV-cache quantization and fused kernels that reduce memory usage and improve throughput. The company has also collaborated with [intel](https://www.wikiprompt.org/wiki/intel) to optimize its software for Intel's latest Xeon processors, achieving significant performance gains on popular benchmarks.

## Performance and Benchmarks

Neural Magic publishes benchmark results demonstrating the effectiveness of its technology. For example, on a dual-socket Intel Xeon Platinum 8380 server, the company reported that a pruned and quantized version of the BERT-large model achieved over 1,000 sentences per second, compared to around 200 for the unoptimized baseline. For computer vision models like YOLOv5, DeepSparse achieved frame rates exceeding 100 FPS on a single CPU socket, which is competitive with GPU inference for many applications.

The company also highlights the cost benefits of CPU-based inference. By using existing server infrastructure, organizations can avoid the capital expenditure of GPU clusters and reduce energy consumption. Neural Magic's technology is particularly attractive for edge deployments, where power and space are limited, and for real-time applications that require low latency.

However, performance gains depend on the model architecture and the degree of sparsity achievable. Dense models with limited redundancy may see smaller improvements, and some workloads still benefit from GPUs, especially for very large models with massive parallelism. Neural Magic acknowledges these limitations and provides guidance on which models are best suited for CPU acceleration.

## Market Position and Competition

Neural Magic operates in the competitive field of AI inference optimization, which includes both hardware and software solutions. On the hardware side, companies like [nvidia](https://www.wikiprompt.org/wiki/nvidia) dominate with GPUs, while [groq](https://www.wikiprompt.org/wiki/groq) and [samba-nova](https://www.wikiprompt.org/wiki/samba-nova) offer specialized AI accelerators. On the software side, competitors include [openai](https://www.wikiprompt.org/wiki/openai)'s Triton, [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind)'s JAX, and various compiler frameworks like TVM and ONNX Runtime. Neural Magic differentiates itself by focusing exclusively on CPUs, which are ubiquitous and often underutilized in data centers.

The company has also faced competition from emerging techniques like model distillation and knowledge distillation, which can produce smaller models without specialized hardware. However, Neural Magic's approach can be combined with these methods, and its tools are designed to complement existing optimization workflows.

In 2023, Neural Magic was acquired by red-hat, a subsidiary of ibm, for an undisclosed amount. The acquisition aimed to integrate Neural Magic's technology into Red Hat's OpenShift AI platform, providing customers with CPU-based inference options. This move signaled a growing interest from enterprise software vendors in efficient AI deployment.

## Use Cases and Applications

Neural Magic's technology is used in a variety of real-world applications. In computer vision, it accelerates object detection, image classification, and segmentation models for use in retail, manufacturing, and security. In natural language processing, it powers sentiment analysis, named entity recognition, and question-answering systems. The company's support for large language models has also enabled deployment of chatbots and code generation tools on CPU servers, reducing costs for startups and enterprises.

One notable use case is in healthcare, where Neural Magic's software has been used to run medical imaging models on hospital servers, avoiding the need to send sensitive data to the cloud. In finance, it accelerates fraud detection and algorithmic trading models that require low latency. The technology is also used in autonomous vehicles and robotics, where CPUs are often the only available compute resource.

## Research and Open Source

Neural Magic maintains an active research program, publishing papers on sparse training, quantization, and inference optimization. Its researchers have contributed to conferences like NeurIPS, ICML, and MLSys. The company also releases much of its software as open source, including SparseML and parts of DeepSparse, fostering a community of users and contributors.

The open-source strategy has helped Neural Magic build a large user base and establish credibility in the machine-learning community. Developers can experiment with the tools and integrate them into their projects, and the company benefits from community feedback and contributions. This approach is similar to that of other AI infrastructure companies like [hugging-face](https://www.wikiprompt.org/wiki/hugging-face) and weights-and-biases.

## Future Directions

Looking ahead, Neural Magic aims to expand its support for emerging model architectures and hardware. With the rise of [transformer](https://www.wikiprompt.org/wiki/transformer)-based models and [generative-ai](https://www.wikiprompt.org/wiki/generative-ai), the company is investing in optimizations for [large-language-model](https://www.wikiprompt.org/wiki/large-language-model) inference, including techniques like speculative decoding and dynamic batching. It is also exploring support for [arm-holdings](https://www.wikiprompt.org/wiki/arm-holdings) processors and [amd](https://www.wikiprompt.org/wiki/amd) EPYC CPUs, which are gaining popularity in cloud environments.

As AI models become larger and more complex, the need for efficient inference will only grow. Neural Magic's focus on CPUs could become increasingly relevant as organizations seek to balance performance, cost, and energy consumption. The company's integration with Red Hat and IBM provides a pathway to enterprise adoption, and its open-source ecosystem continues to attract developers.

## Conclusion

Neural Magic has established itself as a pioneer in CPU-based inference optimization, offering a compelling alternative to GPU-centric approaches. Its combination of sparsity, quantization, and runtime engineering delivers significant performance improvements on commodity hardware, making AI more accessible and affordable. While challenges remain, such as the inherent limitations of CPUs for certain workloads, the company's technology has proven valuable across multiple industries. With the backing of Red Hat and IBM, Neural Magic is well-positioned to play a key role in the future of AI deployment.

---
Source: https://www.wikiprompt.org/wiki/neural-magic
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-08T15:32:24.857477+00:00