Shift AI Inference Workloads to AMD EPYC™ Server CPUs

First, determine your performance needs. How fast do you need responses in terms of minutes, seconds, or milliseconds? How big are the models you’re running in terms of parameters? You may be able to meet performance requirements simply by upgrading to a 5th Gen AMD EPYC CPU, avoiding the cost of GPU hardware.

If you don’t need responses in real time, batch inference is cost-efficient for large-scale and long-term analysis—for example, analyzing campaign performance or predictive maintenance. Real-time inference that supports interactive use cases like financial trading and autonomous systems may need GPU accelerators. While CPUs alone are excellent for batch inference, GPUs are best for real-time inference.

CPUs alone offer enough performance for inference on models up to ~20 billion parameters and for mid-latency response times (seconds to minutes). This is sufficient for many AI assistants, chatbots, and agents. Consider adding GPU accelerators when models are larger or response times must be faster than this.

The short answer is it depends. Extracting maximum performance for a workload is very workload and expertise dependent. With that said, select 5th Gen AMD EPYC  Server CPUs outperform comparable Intel Xeon 6 in inference for many popular AI workloads, including large language models (DeepSeek-R1 671B),³ medium language models (Llama 3.1 8B⁴ and GPT-J 6B⁶), and small language models (Llama 3.2 1B).⁵

AMD EPYC server CPUs include AMD Infinity Guard which provides a silicon-based set of security features.⁷ AMD Infinity Guard includes AMD Secure Encrypted Virtualization (AMD SEV), a widely adopted confidential computing solution that uses confidential virtual machines (VMs) to help protect data, AI models, and workloads at runtime.