ZenDNN 5.2.1: Deepening Quantization and Expanding the AI Inference Frontier on AMD EPYC™ CPUs

With ZenDNN 5.2, we redefined what x86 CPUs could achieve in AI inference, delivering a major step up in performance, a seamless vLLM plugin, and a completely re-engineered runtime. That release was about laying a new foundation. ZenDNN 5.2.1 is about building on it. This incremental release sharpens the quantization story, extends the ecosystem reach, and pushes the performance envelope further, all without breaking a single line of your existing code.

The headline: quantization on AMD EPYC™ processors just got significantly more capable. With asymmetric INT4 Weight-Only Quantization (WOQ), INT8 dynamic quantization through vLLM, and fused RMSNorm kernels, ZenDNN 5.2.1 turns the experimental quantization support from 5.2 into a production-grade pipeline. And we've expanded vLLM compatibility all the way to version 0.18.0, helping ensure alignment with the rapidly evolving open-source inference ecosystem.

Why This Matters: From Experimental to Production-Ready

ZenDNN 5.2 introduced quantization as an experimental capability. With 5.2.1, we're maturing that story across multiple dimensions:

• Smaller Models, Same Intelligence: Asymmetric 4-bit WOQ allows more aggressive model compression while preserving accuracy, enabling production deployment of quantized LLMs on CPU infrastructure.
• Dynamic Quantization at Serving Time: The new INT8 dynamic quantization path integrates directly with the vLLM serving engine. This means you can now serve dynamically quantized models through vLLM with the zentorch plugin, enabling lower latency and higher throughput at inference time.
• Agentic AI at Scale: With vLLM support now spanning versions 0.15.0 through 0.18.0, developers building autonomous AI agents can stay on the latest vLLM releases while continuing to benefit from ZenDNN acceleration. The expanded compatibility window helps ensure your agentic workloads aren't locked to a stale framework version.
• Accuracy You Can Verify: The integration with the open-source LM Evaluation Harness framework provides standardized accuracy benchmarking, giving teams the confidence to deploy quantized models knowing they meet quality thresholds.

What's Under the Hood?

ZenDNN 5.2.1 builds on the modular multi-backend architecture introduced in 5.2, extending it with deeper  quantization paths and kernel-level optimizations across the ZenDNN runtime.

ZenDNN Core Runtime Enhancements

At the foundation layer, the ZenDNN runtime introduces several key capabilities:

• Asymmetric U4 Quantization: New DLP-based asymmetric unsigned 4-bit (U4) weight-only quantization API support. Multiple quantization domains are dispatched according to zero-point (ZP) type, enabling finer-grained control over quantization behavior.
• Dynamic and Static Reorder Integration: Reorder operations are now tightly integrated with MatMul, including intrinsic support for dynamic quantization in the reorder API.
• Fused Normalization Kernels: Fused add + RMS normalization with BF16, backed by dedicated AVX-512 kernels.
• AutoTuner Binning: A new binning approach for kernel selection that improves dispatch accuracy across diverse workload shapes.

Key Highlights of the Update

• Expanded vLLM Support: The zentorch plugin now supports vLLM versions 0.15.0 to 0.18.0, maintaining the zero-code-change acceleration philosophy.
• Asymmetric Weight-Only Quantization: Complete TorchAO-based asymmetric WOQ implementation (Int4WeightOnlyOpaqueTensorConfig), including bias support and Int4OpaqueTensor integration, moving beyond the symmetric-only experimental support in 5.2.
• INT8 Dynamic Quantization via vLLM: A new dynamic qlinear operator in the zentorch backend enables serving dynamically quantized models directly through the vLLM inference engine.
• Optimized RMSNorm: RMSNorm.forward is now patched to use optimized C++ kernels on CPU, complemented by fused add + RMS norm AVX-512 kernels at the ZenDNN runtime level.
• WOQ Operator Fusions: Fusion support for quantized linear operators helps reduce overhead and improve end-to-end quantized inference throughput.
• Modernized Stack: Full support for PyTorch 2.11.0, TensorFlow 2.21.0, and Python 3.10–3.13. TensorFlow backward build compatibility now spans TF 2.16 through 2.21 from a single unified codebase.
• Accuracy Benchmarking: Integrated open-source LM Evaluation Harness framework support for standardized accuracy evaluation of quantized models.

Quantization Deep Dive:

Weight-only Quantization (WoQ): Asymmetric Support

One of the most impactful changes in ZenDNN 5.2.1 is the support for asymmetric Weight-Only Quantization. To understand why this matters, consider how quantization works: model weights are mapped from their original floating-point range to a smaller integer range. In symmetric quantization, this mapping is centered around zero. In asymmetric quantization, the mapping can be offset, allowing it to better capture the actual distribution of weight values, which, in practice, are rarely perfectly centered.

With ZenDNN 5.2.1, this is implemented through a complete TorchAO-based pipeline using Int4WeightOnlyOpaqueTensorConfig, including:

• Full asymmetric 4-bit weight representation with zero-point handling
• Bias support for asymmetric quantized operations
• Operator fusion for quantized linear layers, helping reduce memory traffic and kernel launch overhead
• Validation through the open-source LM Evaluation Harness framework to help ensure quantized model quality

INT8 Dynamic Quantization: Quantize at Serve Time

Beyond weight-only quantization, ZenDNN 5.2.1 introduces INT8 dynamic quantization that integrates directly into the vLLM serving pipeline. Unlike static quantization, dynamic quantization computes scale factors on the fly from the actual activation values at inference time. This makes it ideal for serving workloads where input distributions vary and you want quantization benefits without a separate offline quantization workflow.

The new dynamic qlinear operator in the zentorch backend makes this seamless: simply serve your model through vLLM with the zentorch plugin, and dynamic quantization is applied automatically where beneficial.

Running Quantized Models with vLLM and ZenTorch

For quantizing LLMs with TorchAO and executing the quantized model with vLLM-ZenTorch, please refer the section 2.6 of ZenDNN 5.2.1 user manual.

Performance: Quantized Inference on AMD EPYC™ CPUs