Accelerate llama.cpp on AMD EPYC: ZenDNN Delivers Up to 4.5x the Prompt Processing

Aug 18, 2026

The story of AI inference has been told almost entirely through the GPU. And that is fair - for the heaviest training and high-throughput serving jobs, GPUs remain the gold standard. But there is a quieter shift happening in the rack. The CPU sitting in nearly every server on earth is no longer just feeding the accelerator and handling glue logic. It is becoming a credible, cost-effective inference engine in its own right.

Llama.cpp sits right at the center of that shift. It is one of the most widely used open-source LLM inference engines in the world, with over 1,700 contributors and more than 113,000 stars on GitHub. It runs everywhere - from a Raspberry Pi to a dual-socket data center server - in plain C/C++ with no external dependencies. Millions of developers reach for it first when they want to run a model locally.

The catch: on AMD EPYC™ processors, llama.cpp has historically leaned on generic CPU kernels that leave real performance on the table. The silicon underneath is capable of far more.

That is what the ZenDNN backend for llama.cpp is here to fix. ZenDNN - AMD's open-source deep learning library, purpose-built for the "Zen" cores in EPYC processors (GitHub) - now plugs directly into llama.cpp as a GGML backend. It was engineered during the ZenDNN 5.2 development cycle (see the ZenDNN 5.2 announcement) and is now upstream.

This post covers how the backend plugs into GGML and, most importantly, what speedups developers running on AMD EPYC can expect. We benchmarked twelve models across BF16 and Q8_0, and the prompt-processing numbers are not subtle.

The short version (AMD EPYC™ 9755, 128 cores, single socket):

Up to 4.5x the prompt processing on Mixtral-8x7B (Q8_0), driven by ZenDNN's MoE expert-matmul acceleration²
Up to 3.7x the prompt processing on Mixtral-8x7B (BF16)¹
Roughly 2x the prompt processing across dense models - 1.8x–2.2x in BF16, 2.3x–2.7x in Q8_0¹
Up to 2.84x the end-to-end throughput on mixed prefill + decode workloads
Token generation tracks the native CPU backend - small-batch decode currently falls back to the CPU path (a deliberate, temporary choice while ZenDNN's small-batch performance is optimized)

How ZenDNN plugs into llama.cpp

At its core, llama.cpp delegates all tensor math to GGML, its underlying tensor library. GGML uses a pluggable backend system: each backend registers itself at startup, declares which operations it supports, and a scheduler routes each operation in the compute graph to the most appropriate backend at runtime. Anything a backend does not claim falls through automatically to the CPU backend - no configuration, no manual wiring.

This is what makes adding specialized hardware support clean. The ZenDNN backend follows the same pattern as established backends like CUDA® and HIP: it plugs into the GGML backend registry ahead of the generic CPU backend, intercepts the matrix-multiplication operations that dominate transformer inference, and hands them off to the optimized AMD ZenDNN library. Everything else continues to run on the CPU backend exactly as before. The integration is deliberately surgical - small footprint, high impact.

Image Zoom

Figure 1: The llama.cpp ZenDNN backend architecture, from user-facing tools (llama-cli, llama-server, llama-bench) down to AMD EPYC CPUs

A request enters through the llama.h API. Core llama.cpp loads the GGUF model, manages the KV cache and batching, and builds a compute graph - a DAG of every tensor operation the transformer layers need: multi-head attention, feed-forward networks, and, for MoE models, expert routing and gating. That graph lands at the GGML backend scheduler, which splits it by backend, allocates buffers, and executes the splits. Because the registry lists ZenDNN ahead of CPU in priority, the two operations that matter most get routed to ZenDNN:

GGML_OP_MUL_MAT - the dense matrix multiplications that make up the bulk of transformer compute
GGML_OP_MUL_MAT_ID - the expert-gated matmul used by MoE models like Mixtral

Each matmul checks its weight precision - F32, BF16, or Q8_0 (handled via dynamic weight packing) - and calls into the ZenDNN library. Everything else - RoPE, RMS norm, softmax, SiLU/GELU and the rest - falls through to the CPU backend transparently.

Two pieces of ZenDNN do most of the work here. A Low Overhead API (LowOHA) minimizes per-call dispatch cost, which matters because transformer inference issues an enormous number of small, repeated, batched matmuls. And weight caching packs the model's constant weights once (is_weights_const = true) and reuses the packed layout on every subsequent call instead of repacking each time. Underneath, the matmul path is selectable via ZENDNNL_MATMUL_ALGO - AOCL-DLP (AMD's optimized BLAS, the recommended default), oneDNN, or LIBXSMM - with AOCL-DLP giving the best performance for most llama.cpp workloads. ZenDNN’s full layered architecture is documented in the ZenDNN repo.

The backend declares support for:

Table 1: Operations and data types accelerated by the ZenDNN backend.
Operation	F32	BF16	Q8_0	Notes
GGML_OP_MUL_MAT	✓	✓	✓	Full ZenDNN acceleration
GGML_OP_MUL_MAT_ID	✓	✓	✓	MoE; CPU fallback if experts > 32
All other ops	—	—	—	Transparent CPU backend fallback

Enabling the backend is a single CMake flag, -DGGML_ZENDNN=ON, which downloads and builds ZenDNN automatically on the first build. Full setup such as build/run commands can be found in the ZenDNN backend documentation of llama.cpp.

Performance results

All benchmarks were run on an AMD EPYC™ 9755 128-Core system, pinned to a single socket (128 cores, one NUMA node), with ZenDNN 5.2.2 against the native llama.cpp CPU backend (build b9326). We used llama-batched-bench with a 512-token prompt, 128-token generation across 128 parallel sequences - an effective input/output/batch of 512/128/128 - three runs each, average reported. "Higher is better" throughout. Full configuration is in the footnote (Footnote 1, ZD-066).

We measured two things separately, because they behave very differently

Prompt processing (prefill) - how fast the model ingests input tokens. Compute-bound.
Total / end-to-end - the combined real-world throughput.

Prompt Processing - BF16 Models

Figure 2: Prompt processing throughput (tokens/s), BF16 models - native llama.cpp CPU backend vs. ZenDNN-accelerated backend.(See Footnote 1 - ZD-066)

Prompt Processing - Q8_0 models

Figure 3: Prompt processing throughput (tokens/s), Q8_0 models - native vs. ZenDNN-accelerated backend. (See Footnote 2 - ZD-067)

Total / End-to-End Throughput

Figure 4: Total end-to-end throughput (tokens/s), BF16 and Q8_0 models - native vs. ZenDNN-accelerated backend. (See Footnote 3 - ZD-068)

Accuracy: Same Results, Faster

Faster inference only matters if the outputs hold up. The ZenDNN backend accelerates the matrix multiplications without changing the model's precision or quantization - Q8_0 stays Q8_0, BF16 stays BF16 - so results are numerically equivalent to the native CPU backend running the same GGUF.

This can be confirmed directly by measuring perplexity on WikiText-2 with the same model through both backends:

Table 2: Perplexity on WikiText-2 - native CPU backend vs. ZenDNN, same GGUF. Lower is better; the two backends should match. See Footnote 4 (ZD-069)
Model	Native CPU (PPL)	ZenDNN (PPL)	Difference
Llama-3.1-8B (BF16)	7.3214	7.3214	0.0000
Mixtral-8x7B (BF16, MoE)	4.4087	4.4083	0.0004
Mixtral-8x7B (Q8_0, MoE)	4.4134	4.4136	0.0002

Across dense and MoE models in both BF16 and Q8_0, the largest difference between the two backends is 0.0004 perplexity - well within the ±0.02–0.05 measurement error, i.e. numerical noise rather than a change in behavior. Q8_0 also stays close to BF16: Mixtral-8x7B scores 4.4136 versus 4.4083, about 0.1% apart.

Supported Hardware

Table 3: AMD EPYC processor families supported by the ZenDNN backend.
CPU Family	Codename	Generation	Architecture
AMD EPYC™ 9005 Series	"_Turin_"	5th Gen	"_Zen 5_"
AMD EPYC™ 9004 Series	"_Genoa_"	4th Gen	"_Zen 4_"
AMD EPYC™ 7003 Series	"_Milan_"	3rd Gen	"_Zen 3_"

* On Zen 3 (Milan), only FP32 is accelerated through ZenDNN. ZenDNN is a CPU-only backend. For AMD GPU acceleration, see the HIP backend with ROCm.

What's next

The ZenDNN backend is actively developed. Upcoming plans includes:

Decode-phase and low-batch acceleration - closing the small-batch performance gap so ZenDNN can take over token generation, and removing the current fallback that routes to the CPU backend
Broader quantization support beyond Q8_0 (K-quants, IQ-quants)
Further MoE optimisation to reduce the CPU fallback threshold beyond 32 experts
Fused ops and broader op coverage beyond MUL_MAT and MUL_MAT_ID, see docs/ops.md for the full GGML ops and backend support status

Contributions and feedback are welcome via the llama.cpp GitHub repository.

Acknowledgements

This work is the result of collaboration between the AMD ZenDNN team and the llama.cpp open-source community. Special thanks to the llama.cpp maintainers and reviewers who made the upstream integration possible.

Footnotes

ZD-066: Results based on AMD internal testing as of 15-Jun-2026. AMD System: 2P AMD EPYC™ 9755 128-core processors, single socket used, SMT disabled (1 thread/core), NPS1, max boost 4.06 GHz; 256 threads total across 2 NUMA nodes (node0 CPUs 0-127, node1 CPUs 128-255), workload pinned to node1; caches L1d 12 MiB, L1i 8 MiB, L2 256 MiB, L3 1 GiB; BIOS RVOT1004C; Ubuntu 22.04.5 LTS, kernel 5.15.0-174-generic. Software: llama.cpp build b9326; AMD ZenDNN 5.2.2; matmul algorithm ZENDNNL_MATMUL_ALGO=1 (AOCL-DLP). Baseline is the native llama.cpp CPU backend; the ZenDNN-accelerated configuration is the same build with -DGGML_ZENDNN=ON. Benchmark: llama-batched-bench, 512-token prompt, 128-token generation, 128 parallel sequences (512/128/128), flash attention on, KV cache matched to model precision, micro-batch (-ub) tuned per backend, 3-run average. Prompt processing throughput, tokens/s, BF16 models, higher is better: Model Native-CPU ZenDNN Speedup gemma-4-E4B 1172.32 2341.72 2.00x gpt-oss-20B 1187.31 2589.40 2.18x Llama-3.1-8B-Instruct 694.18 1233.85 1.78x Mixtral-8x7B-Instruct-v0.1 209.15 771.73 3.69x phi-4 403.06 751.24 1.86x Qwen3.5-4B 1209.53 2307.04 1.91x Results may vary due to factors including system configurations, software versions and BIOS settings.
ZD-067: Results based on AMD internal testing as of 15-Jun-2026. AMD System: 2P AMD EPYC™ 9755 128-core processors, single socket used, SMT disabled (1 thread/core), NPS1, max boost 4.06 GHz; 256 threads total across 2 NUMA nodes (node0 CPUs 0-127, node1 CPUs 128-255), workload pinned to node1; caches L1d 12 MiB, L1i 8 MiB, L2 256 MiB, L3 1 GiB; BIOS RVOT1004C; Ubuntu 22.04.5 LTS, kernel 5.15.0-174-generic. Software: llama.cpp build b9326; AMD ZenDNN 5.2.2; matmul algorithm ZENDNNL_MATMUL_ALGO=1 (AOCL-DLP). Baseline is the native llama.cpp CPU backend; the ZenDNN-accelerated configuration is the same build with -DGGML_ZENDNN=ON. Benchmark: llama-batched-bench, 512-token prompt, 128-token generation, 128 parallel sequences (512/128/128), flash attention on, KV cache matched to model precision, micro-batch (-ub) tuned per backend, 3-run average. Prompt processing throughput, tokens/s, Q8_0 models, higher is better: Model Native-CPU ZenDNN Speedup gemma-4-12B-it 422.94 986.31 2.33x gemma-4-31B-it 163.58 404.54 2.47x Mistral-Small-3.2-24B-Instruct-2506 237.18 620.85 2.62x Mixtral-8x7B-Instruct-v0.1 209.84 954.14 4.55x Qwen3.5-9B 633.40 1557.65 2.46x Qwen3.6-27B 190.73 517.55 2.71x Results may vary due to factors including system configurations, software versions and BIOS settings.
ZD-068: Results based on AMD internal testing as of 15-Jun-2026. AMD System: 2P AMD EPYC™ 9755 128-core processors, single socket used, SMT disabled (1 thread/core), NPS1, max boost 4.06 GHz; 256 threads total across 2 NUMA nodes (node0 CPUs 0-127, node1 CPUs 128-255), workload pinned to node1; caches L1d 12 MiB, L1i 8 MiB, L2 256 MiB, L3 1 GiB; BIOS RVOT1004C; Ubuntu 22.04.5 LTS, kernel 5.15.0-174-generic. Software: llama.cpp build b9326; AMD ZenDNN 5.2.2; matmul algorithm ZENDNNL_MATMUL_ALGO=1 (AOCL-DLP). Baseline is the native llama.cpp CPU backend; the ZenDNN-accelerated configuration is the same build with -DGGML_ZENDNN=ON. Benchmark: llama-batched-bench, 512-token prompt, 128-token generation, 128 parallel sequences (512/128/128), flash attention on, KV cache matched to model precision, micro-batch (-ub) tuned per backend, 3-run average. Total end-to-end throughput, tokens/s, higher is better: Model (BF16)Native-CPU ZenDNN Speedup gemma-4-E4B 1013.85 1550.29 1.53x gpt-oss-20B 1024.95 1558.54 1.52x Llama-3.1-8B-Instruct 632.17 914.85 1.45x Mixtral-8x7B-Instruct-v0.1 188.42 433.45 2.30x phi-4 374.29 568.70 1.52x Qwen3.5-4B 717.55 924.28 1.29x Model (Q8_0) Native-CPU ZenDNN Speedup gemma-4-12B-it 397.06 695.38 1.75x gemma-4-31B-it 156.06 286.03 1.83x Mistral-Small-3.2-24B-Instruct-2506 231.06 446.51 1.93x Mixtral-8x7B-Instruct-v0.1 201.90 573.59 2.84x Qwen3.5-9B 468.31 723.37 1.54x Qwen3.6-27B 153.53 260.09 1.69x Results may vary due to factors including system configurations, software versions and BIOS settings.
ZD-069: Results based on AMD internal testing as of 15-Jun-2026. AMD System: 2P AMD EPYC™ 9755 128-core processors, single socket used, SMT disabled (1 thread/core), NPS1, max boost 4.06 GHz; 256 threads total across 2 NUMA nodes (node0 CPUs 0-127, node1 CPUs 128-255), workload pinned to node1; caches L1d 12 MiB, L1i 8 MiB, L2 256 MiB, L3 1 GiB; BIOS RVOT1004C; Ubuntu 22.04.5 LTS, kernel 5.15.0-174-generic. Software: llama.cpp build b9326; AMD ZenDNN 5.2.2; matmul algorithm ZENDNNL_MATMUL_ALGO=1 (AOCL-DLP). Baseline is the native llama.cpp CPU backend; the ZenDNN-accelerated configuration is the same build with -DGGML_ZENDNN=ON. Accuracy: perplexity via llama-perplexity on WikiText-2 (wiki.test.raw), 512-token context. Identical GGUF run through both backends with the same threads, pinning, flash attention, and KV-cache type; the backend is the only variable. Perplexity, lower is betterModel Precision Native-CPU ZenDNN Difference Llama-3.1-8B BF16 7.3214 7.3214 0.0000 Mixtral-8x7B BF16 (MoE) 4.4087 4.4083 0.0004 Mixtral-8x7B Q8_0 (MoE) 4.4134 4.4136 0.0002 Results may vary due to factors including system configurations, software versions and BIOS settings.