AMD Zen Deep Neural Network (ZenDNN)

6.0 Release Highlights

ZenDNN 6.0.0 is a major release building on the 5.2.1 runtime architecture. It deepens the Low Overhead API (LowOHA) as the primary inference path, expands MoE group GEMM and FP16 operator coverage, and adds production-grade post-op and weight caching, with corresponding extensions to BenchDNN, gtests, and operator documentation.

Highlights at a Glance

• MoE / Group MatMul
  • Regime-aware AUTO routing (decode vs. prompt)
  • F16 group matmul support
  • Group Dynamic Quant API
  • N-tile DQ-INT8 and M-tile vertical-fusion custom kernels
  • Gated-activation enhancements
  • New runtime knob: ZENDNNL_GRP_MATMUL_ALGO
• FP16 Expansion
  • FP16 coverage now includes SDPA, normalization, quant reorder, quant embedding, and softmax, extending the F16 matmul paths already present in 5.2.1 (AOCL DLP and OneDNN)
• Caching
  • Post-op metadata cache
    (ZENDNNL_ENABLE_POSTOP_CACHE)
  • In-place weight caching
    (ZENDNNL_MATMUL_WEIGHT_CACHE=2)
  • Weight-reorder and ZP-compensation caches are now enabled by default
• Quantization
  • Vectorized per-group dynamic quant kernel
  • BenchDNN dynamic-quantization workloads
• AutoTuner
  • The ZENDNNL_MATMUL_AUTO_ALGO_CANDIDATES environment variable customizes the algorithm evaluation sequence
• BenchDNN
  • SDPA and normalization workloads
  • MoE fused inputs
  • Group-matmul hardware perf-counter infrastructure
• Activations
  • MISH activation support
  • Float conversion paths in reorder (FP32 <-> FP16/BF16)
• Build & Compatibility
  • CMake minimum raised to 3.26

ZenDNN Plugin for PyTorch (zentorch)

Overview

This is a major zentorch release that significantly expands framework support, introduces groundbreaking optimizations for Mixture of Experts (MoE) architectures, and enhances quantization capabilities for Large Language Models and Recommender Systems on AMD EPYC™ CPUs. This release transitions to PyTorch-aligned versioning, adds vLLM support up to version 0.23.0, introduces limited FP16 support for select operations, AOTI (Ahead of Time Inductor) integration, and delivers substantial performance improvements through Fused MoE operations and optimized group matmul kernels.

Improvements

1. Framework and Version Support

• PyTorch 2.12.0 Support: Full compatibility with PyTorch 2.12.0 in addition to PyTorch 2.11.0
• New plugin versioning scheme: Changed from independent ZenDNN versioning (6.0.0) to PyTorch-aligned format (v2.12.0.2, v2.11.0.2)
  • Format:
     {PYTORCH_MAJOR}.{PYTORCH_MINOR}.{PYTORCH_PATCH}.{ZENTORCH_PLUGIN_PATCH}
  • Enables clearer compatibility and smoother upgrade paths
• vLLM 0.20.0–0.23.0 Support: Extended vLLM compatibility across major releases
  • Out-of-tree plugin runtime support starts at vLLM 0.20.0
  • Dropped support for vLLM 0.15.0–0.19.1
  • Zen optimizations and features are upstreamed and available via the in-tree ZenCpuPlatform on vLLM for AMD EPYC CPUs with AVX512 ISA support
• TorchAO 0.17.0 Support: Enhanced quantization framework compatibility
  • TorchAO 0.17.0 for vLLM 0.20.0–0.23.0 and PyTorch 2.11.0

2. Mixture of Experts (MoE) Optimizations

• Group Matmul: Parallel group matrix multiplication operator (zentorch_group_matmul) for batched expert computation with gated activation post-ops (SiLU, GELU, SwigluOAI), MoE weighted-reduce fusion, INT8 quantization support, and ZenDNN LowOHA backend
• Fused MoE: End-to-end MoE FFN block optimization (zentorch_fused_moe) integrating gate+up → activation → down → weighted reduce workflow with active-set narrowing, buffer aliasing, persistent scratchpads, and vLLM integration
• Quantized MoE: TorchAO INT8 quantized MoE model support with DA8W8 (dynamic INT8 activation, INT8 weights) integration
• MoE Model Support: Phi-4 Vision, and GatedDeltaNet (GDN) architecture enablement

3. Quantization Enhancements

• LLM-Compressor Integration: Support for externally quantized models
  • W8A8 (INT8 weights and activations) and W4A16 (INT4 weights, BF16 activations) quantization schemes
  • Compatible with vLLM v0.22.0+ for LLM-Compressor quantized models
  • llm-compressor 0.10.0.2 for model quantization
  • Pre-quantized models on Hugging Face supported out-of-the-box
• MoE Model Support: LLM-Compressor quantized MoE models are not supported in this release of zentorch
• Quantization Improvements:
  • Improved WOQ fusions with better pattern matching
  • Accuracy improvements across quantized LLM models

4. FP16 (Float16) Support — Limited

• Limited FP16 Operations: Native float16 inference for select operations
  • Linear operations and all linear fusions (linear-unary, linear-binary, linear-unary-binary)
  • Matrix multiplication (mm, bmm, addmm, baddbmm) with FP16 tensors
  • Embedding and EmbeddingBag operations
  • SDPA (Scaled Dot-Product Attention) FP16 support
  • Weight prepacking for FP16
• Hardware Requirements: AVX512-FP16 ISA support required
• Known Limitations: FP16 not supported for quantized operations (WOQ, qlinear), convolution in this release

5. Performance Optimizations

• SDPA Integration: ZenDNN Scaled Dot-Product Attention kernels
  • Optimized attention computation for LLM inference
  • FP16 and BF16 SDPA support
• RMS Norm Optimization:
  • Integrated ZenDNN RMS Norm operations
  • Fused Add+RMS Norm for reduced memory bandwidth
  • vLLM RMSNorm.forward patching for CPU acceleration
• Hardware Compatibility:
  • AVX512 detection with graceful fallback to native PyTorch on unsupported hardware

6. vLLM Plugin Enhancements

• Plugin Architecture: Comprehensive restructuring for improved stability
  • Cleaner separation between platform and general plugin entry points
  • GEMM dispatch optimization and oneDNN bypass for zentorch kernels
  • Fused RMS Norm integration in vLLM plugin
  • TorchAO 0.17.0 patch support for vLLM 0.20.0–0.23.0

7. AOTI Integration

• Shim wrappers and lowerings: cpp_wrapper support for zentorch backend in torch.compile path
• QLinear, Linear and WoQ AOTI shim support along with their fused variants
• QuantEmbeddingBag AOTI shim for quantized embedding operations
• Enhanced performance for DLRMv2 model

Breaking Changes

• Dropped PyTorch 2.10.0 Support: PyTorch 2.10.0 is no longer supported; use PyTorch 2.11.0 or 2.12.0
• Dropped vLLM 0.15.0–0.19.1 Support: Minimum supported vLLM version is now 0.20.0 for out-of-tree plugin
• orchAO Version Requirements:
  • vLLM 0.20.0–0.23.0 requires TorchAO 0.17.0 for execution
• AMD Quark Deprecation: AMD Quark is no longer required for quantization; TorchAO is now the standard quantization framework for zentorch

Known Issues

• GLIBCXX version conflicts may require setting LD_PRELOAD (see README)
• Experimental Python versions (3.13T, 3.14, 3.14T) are not supported
• FP16 support limited to core operations (linear, mm variants, embedding, SDPA); not available for quantized operations, convolution
• AVX512 instruction set required for optimal performance; fallback to native PyTorch on non-AVX512 system

ZenDNN Plugin for TensorFlow (zentf) 

Improvements 

1. Framework and Version Support

• TensorFlow 2.21.0 Support: Full compatibility with TensorFlow 2.21.0 through TensorFlow 2.16.1
• New plugin versioning scheme: Changed from independent ZenDNN versioning (6.0.0) to TensorFlow-aligned format (v2.21.0.0)
  • Format:
    {TENSORFLOW_MAJOR}.{TENSORFLOW_MINOR}.{TENSORFLOW_PATCH}.{ZENTF_PLUGIN_PATCH}

2. Features Support

• Einsum Operator Support
  • Supports ij,jk->ik (2D matmul) and abc,bcd->abd (batched matmul for MoE)
  • FP32 and BF16 support
• ZenGroupEmbedding Fusion
  • Fuses GatherV2 + SafeCast + ConcatV2 into a single op
  • Multi-table embedding support with strided gather
  •  Eliminates concat overhead
• ZenSafeEmbeddingLookupSparse Fusion
  • Fuses SparseFillEmptyRows → SparseSegment → SelectV2 subgraph

3. Build Info Enhancements

• Added zentf.__version__ and zentf.__config__ attributes
• Runtime version / config introspection

4. TF-Java Support

• TF-Java interface ported to TF 2.21.0