Speculative LLM Inference on the 5th Gen AMD EPYC Processors with Parallel Draft Models (PARD) & AMD Platform Aware Compute Engine (AMD PACE)

Introduction

Auto-regressive language models process tokens sequentially and often face limitations due to memory bandwidth constraints. To address this, speculative decoding predicts multiple future tokens and verifies them in parallel, which reduces inter-token latency. Parallel draft models (PARD) technology, developed by AMD, further enhances speculative decoding by increasing the acceptance rate of predicted tokens, enabling large language models (LLMs) to achieve human-perceptible generation speeds on 5th Gen AMD EPYC™ processors. 

AMD PACE (AMD Platform Aware Compute Engine) is a high-performance inference server with speculative decoding optimized for modern CPU-based servers. When combined with PARD, AMD PACE + PARD solution achieves a throughput of approximately 380 tokens/second on the Llama 3.1 8B model. Even with a batch size of 1, this allows multiple users to experience interactive, human-readable response times. AMD PACE is also set to be open-sourced soon.

In this blog, we will explore how PARD and AMD PACE work together to optimize inference performance for Transformer models, particularly in data center environments. We will discuss the technical underpinnings of these technologies, their integration with the 5th Gen AMD EPYC processor architecture, and the results of performance evaluations based on benchmark tests. 
 

Technical Overview

Auto-Regressive Decoding

In auto-regressive decoding, each layer performs scaled dot-product attention using several parameters: batch size (B), number of attention heads (H), head dimension (D), and sequence length (n). For each new token, batched matrix multiplications (Query-Key and Attention-Value) are updated incrementally. However, this process often has low arithmetic intensity and is typically bottlenecked by memory bandwidth.

Speculative Decoding with PARD

Speculative decoding accelerates LLM inference, but speculative decoding often requires multiple draft model passes, which can introduce overhead.  PARD introduces a Parallel Mask Predict mechanism, enabling the draft model’s contribution in a single forward pass after minimal fine-tuning. This significantly boosts inference speeds for models like Llama 3 and Qwen. A single PARD-enhanced draft model can accelerate an entire family of LLMs, simplifying their deployment.

Through parallelism, PARD can efficiently speculate and verify K tokens per iteration to improve arithmetic intensity. The overall speedup is a function of both the mean number of accepted speculative tokens (m') and the value of K.  

Efficient Inference on the 5th Gen EPYC Processor Architecture

The AMD EPYC 9005 Series uses a hybrid, multi-chip design and new ‘Zen 5’ and ‘Zen 5c’ cores to address challenges in data centers. This new architecture prioritizes leadership performance, density, and efficiency in virtualized and cloud environments while supporting new AI workloads.  
Servers based on the EPYC 9005 Series leverage Vectorized Neural Network Instructions (VNNI) to accelerate neural network inference. Each socket, featuring 128 physical (256 logical) cores and operating at a clock frequency of 2.6 GHz, delivers a theoretical hardware performance of 90-100 tera operations per second (TOPS) using BFloat16 precision across two sockets.

Deep Dive: How AMD PACE and PARD Enhance Inference Performance

AMD PACE utilizes PARD to reduce inter-token latency and leverages ‘Zen 5’ processor cores in the EPYC processor for efficient multi-instance runs. Here’s how this collaboration enhances performance: 

AMD PACE improves support for Transformer models on CPUs by efficiently integrating PARD. AMD PACE leverages the optimal number of speculative tokens ('K') to minimize inter-token latency.

The optimal selection of "K" (number of speculative tokens) enhances the computational efficiency of the cores. Coupled with RAM capacities in the terabyte range, servers based on the EPYC 9005 Series can host multiple model instances, enabling concurrent real-time interactive sessions for numerous users. By carefully tuning "K" and potentially other parameters, we can effectively exploit the ‘Zen 5’ architecture for improved throughput.

Draft Model Accuracy

The draft model accuracy across different datasets is as follows: