AMD AI NIC Technology: AMD AI NICs Now Support HPC to Accelerate AI for Science

Nov 16, 2026

Row of black server racks with glowing gold interconnects and blue simulation displays on top surfaces.

One Network for AI and Scientific Computing

AMD is expanding the capabilities of its AI NIC technology with support for high-performance computing (HPC) workloads. Built for AI and optimized for HPC, AMD AI NIC technology enables a common Ethernet network to support large-scale AI training and the MPI-based simulations central to scientific computing. 

This technology provides scale-out connectivity for Lux,  the AMD powered AI factory for science at Oak Ridge National Laboratory (ORNL), ranked No. 14 on the November 2026 TOP500 list. 

Lux brings together more than 4,000 AMD Instinct™ MI355X GPUs, AMD EPYC™ Server CPUs, and AMD Pensando™ Pollara 400 AI NICs, with each AI NIC providing each GPU with 400 Gb/s of scale-out connectivity.  Because advances in AI for Science increasingly depend on AI and HPC workloads sharing the same network, Lux runs large-scale AI training and MPI-based scientific simulation on a single Ethernet fabric.

Lux will put these capabilities to work for the U.S. Department of Energy’s (DOE) Genesis Mission. Planned projects span AI-assisted fusion power plant design, materials development and manufacturing, and Earth system prediction, bringing AI and simulation together to advance scientific discovery.

AI for Science increasingly brings AI models and scientific simulations into the same research workflow, placing different demands on the network. By contrast, AI training moves large volumes of data through collective operations, while many simulations require frequent exchanges and synchronization across processors. AMD AI NIC technology now supports both on a common, programmable Ethernet foundation that can evolve through software and firmware updates.

On the AI side, training a 123-billion-parameter language model on AMD Instinct™ MI355X GPUs with AMD Pensando™ Pollara 400 AI NICs reached 4.48 exaflops of FP8 hybrid-precision throughput on 2,048 GPUs. Growing the job 16-fold from 128 GPUs retained 98.3% of the per-GPU throughput, and at every scale tested the GPUs sustained more than 85% of their peak BF16 compute rate, with the model's linear layers running in FP8.1

Line chart showing AMD Instinct MI355X weak scaling to 2,048 GPUs, reaching 4.48 EF/s aggregate throughput at 86.97% MFU.

The rest of this post looks at the other half of the story: how the same AI NIC performs on the communication patterns that scientific simulations depend on.

How Scientific Simulations Depend on the Network

An HPC simulation divides one large scientific problem across many processors, and the network determines how well those processors work together. Processors share results with neighboring processors at every step, so a step can only move as fast as the slowest message. A delay on one link can leave thousands of processors waiting.

Another demand is coordination. Many steps require every processor to combine results before the simulation can move forward, so how long each of these small exchanges takes, and how much that time varies from step to step, can matter as much as raw bandwidth.

To manage this exchange, much of today's HPC software relies on the Message Passing Interface, or MPI, a widely adopted standard that gives scientific applications a common way to communicate across multiple networked nodes. In a weather simulation, MPI is how each node passes temperature, pressure, and wind data at the edges of its region to the nodes handling neighboring regions. MPI is also how thousands of processors across those nodes combine their results, such as total rainfall across the whole forecast area, before the next step begins.

AMD AI NIC technology supports MPI through UCX, a widely used open-source communication library that connects MPI implementations such as Open MPI to the network hardware. Support for Libfabric, another widely used open-source communication library, is in development. As a result, existing HPC applications can typically run on standards-based Ethernet without significant code changes.

Proven at Scale

Benchmarks are a useful starting point, but scientific applications are the real test. On a large cluster of AMD Instinct™ MI355X GPUs, each paired with an AMD Pensando™ Pollara 400 AI NIC, representative HPC workloads were run with Open MPI and UCX at scales of up to 3,760 GPUs. Three results show how the network holds up as jobs grow.

Collective latency: small growth at large scale. Allreduce, in which every process contributes a value and receives the combined result, sits at the heart of iterative solvers and many time-stepping codes, and for these codes small-message latency matters most. An 8-byte MPI Allreduce across 2,048 GPUs, with one MPI process per GPU and data in GPU memory, completed in 103 microseconds on average. Growing the job to 3,760 GPUs, 1.8 times as many, raised that latency by only about 17%, to 121 microseconds.2

Line chart showing MPI Allreduce average latency increasing slightly from 103 µs at 2,048 GPUs to 121 µs at 3,760 GPUs.

Molecular dynamics: near-linear strong scaling. LAMMPS is one of the most widely used molecular dynamics codes in science. Using its SNAP machine-learning interatomic potential, a fixed simulation of 17.2 billion tungsten atoms ran 40.9 times faster on 3,200 GPUs than on 64 GPUs. Strong-scaling efficiency stayed above 97% through 1,024 GPUs, held at 94.4% on 2,048 GPUs, and reached 81.9% on 3,200 GPUs, 50 times the starting GPU count.3

Line chart of LAMMPS SNAP strong scaling showing AMD Instinct MI355X speedup reaching 81.9% efficiency at 3,200 GPUs.

Turbulence: scaling a network-intensive spectral code. Spectral turbulence simulations are among the most communication-intensive workloads in HPC: every time step transposes the full three-dimensional grid across all GPUs using MPI Alltoall. In weak-scaling runs of M-PSDNS, an OLCF benchmark derived from a production turbulence solver, the grid grew with the system to 12,288 × 6,144 × 6,144 points on 1,024 GPUs. At that scale the code sustained 72.7% FP32 weak-scaling efficiency relative to a single node, processing about 188 billion grid points per second.4 Because exchanges like these are bandwidth-bound, workloads of this kind stand to gain directly from higher per-port bandwidth.

Line chart of M-PSDNS weak scaling efficiency dropping to 72.7% for FP32 and 59.6% for FP64 at 128 nodes (1,024 GPUs).

Together with the AI training results presented earlier, these HPC results demonstrate AMD AI NIC technology supporting both workload types at a scale of thousands of GPUs. Research organizations can use a common Ethernet foundation for AI training and established scientific applications as they develop AI for Science workflows.

Bringing Open, Programmable Networking to HPC

Research organizations typically keep a supercomputer in service for four to six years, according to LUMI, while the standards running on those systems continue to change. The MPI standard that many scientific applications rely on has been updated three times since 2021, and the libraries that connect MPI to the network continue to advance alongside it. Keeping pace over that lifetime calls for two things: a network built on open, standards-based Ethernet, so organizations can choose from a range of compatible hardware as the system grows, and a programmable NIC that can add new communication capabilities through software and firmware updates over the life of the system.

AMD AI NIC technology is built on a fully programmable P4 data path. Rather than fixing transport behavior when the chip is designed, it allows protocol logic to be developed, tested, and refined as networking requirements change. As MPI and communication libraries evolve, new capabilities can be introduced through software and firmware updates rather than by waiting for an entirely new hardware platform, extending the value of hardware already deployed.

That programmability is also how AMD optimized an AI NIC for HPC. HPC enablement work from AMD added capabilities that MPI applications rely on, including a native UCX provider over reliable-connected RDMA, shared receive queues that reduce per-connection memory, GPU-to-NIC affinity and connection setup that scale to thousands of GPUs, and a NIC configuration with the queue-pair capacity that large MPI jobs need. All of it was delivered through software and firmware on the same AI NIC hardware that carries AI training, extending its capabilities to HPC without requiring a new AI NIC.

Across generations, AMD AI NIC technology shares a common programmable foundation, so the transport software, validation effort, and operational experience built on one generation carry forward to the next. That foundation spans the AMD Pensando™ Pollara 400 AI NIC at 400G and the AMD Pensando™ Vulcano 800 AI NIC at 800G. Moving to 800G brings new NIC hardware, but not a restart: the HPC software validated on Pollara 400 carries forward, and for bandwidth-heavy simulations, doubling per-port bandwidth can reduce the time GPUs spend waiting on the network.

For scientific communities, AMD AI NIC technology offers a network that can adapt as research needs change and carry validated work from one generation to the next. Together, these benefits help more of each investment go toward discovery rather than rebuilding infrastructure.

Advancing Scientific Computing

Scientific computing will continue to place greater demands on infrastructure as AI for Science matures. Higher-resolution climate models, more detailed engineering simulations, larger molecular systems, and AI models trained on scientific data all require efficient communication across growing numbers of processors. As researchers combine AI and HPC in their workflows, they need networking that supports both and adapts as applications evolve over the life of a system.

On AMD Instinct™ MI355X GPU systems with AMD Pensando™ Pollara 400 AI NICs, AMD AI NIC technology already does both: it sustains multi-exaflop FP8 AI training and scales MPI applications to thousands of GPUs over one standards-based Ethernet network. Built for AI and optimized for HPC, it connects existing MPI applications to that network through UCX, with Libfabric support on the way.

Deployed in Lux, AMD AI NIC technology provides scale-out connectivity for AI training and HPC simulation, helping researchers use both approaches on a common Ethernet network to advance AI for Science.

A fully programmable architecture allows the network to adapt through software as standards change, while a common foundation across generations carries validated work forward to higher bandwidth. Through ecosystem choice, AMD also gives research organizations greater flexibility in how they build and expand the infrastructure supporting their science.

Learn more about AMD AI NIC technology.
Footnotes

1. AI training: 123B-parameter dense transformer (Mistral-Large-Instruct-2411 architecture, 88 layers) trained with Primus/Megatron-LM and RCCL in an AMD-optimized configuration (custom FP8 GEMMs, fused kernels, cached FP8 weights, pipeline scheduling, and communication overlap); FP8 hybrid precision (linear layers in FP8); tensor parallel 1, four pipeline stages with three interleaved virtual stages each, data parallel 2 per node, micro-batch of one 4,096-token sequence, global batch of 128 sequences per node; throughput computed at 759.8 GFLOP per token. 16 to 256 nodes (128 to 2,048 GPUs): 2,227 TFLOP/s per GPU (285 PFLOP/s aggregate) on 128 GPUs and 2,189 TFLOP/s per GPU (4.48 EFLOP/s aggregate) on 2,048 GPUs. Weak-scaling efficiency = per-GPU throughput on N GPUs / per-GPU throughput on 128 GPUs. Utilization is relative to the MI355X dense BF16 peak of 2,516.6 TFLOP/s (88.5% on 128 GPUs, 87.0% on 2,048 GPUs); relative to the dense FP8 peak of 5,033.2 TFLOP/s it is approximately 44%. Short throughput runs from random initialization, 12 iterations per scale, average of nine after three warm-up iterations; these runs measure throughput, not convergence. Software: ROCm 7.15.0, RCCL 2.30.4, PyTorch 2.12.0, Primus 26.7 System: AMD Instinct™ MI355X GPU cluster; each node with 2x AMD EPYC™ 9575F (64-core) CPUs, 8x AMD Instinct™ MI355X GPUs (288 GB HBM3E each), and 8x AMD Pensando™ Pollara 400 AI NICs (one per GPU, 3.2 Tb/s per node) on a fat-tree RoCEv2 Ethernet fabric. Testing as of October 2026. Results are from one run per scale and may vary based on configuration, software versions, and system conditions. [Claim ID: MI350-091]

2. MPI Allreduce: OSU Micro-Benchmarks 7.5 (osu_allreduce) built for ROCm, 8-byte messages, GPU device buffers, one rank per GPU bound to its paired NIC and a local CPU core; Open MPI 5.0.10, UCX 1.22.0 (site-installed software modules); AI NIC firmware 1.117.5-a-196 with 8K queue-pair profile; average of 1,000 iterations, results from one run per scale. 256 to 470 nodes (2,048 to 3,760 GPUs). System: AMD Instinct™ MI355X GPU cluster; each node with 2x AMD EPYC™ 9575F (64-core) CPUs, 8x AMD Instinct™ MI355X GPUs (288 GB HBM3E each), and 8x AMD Pensando™ Pollara 400 AI NICs (one per GPU, 3.2 Tb/s per node) on a fat-tree RoCEv2 Ethernet fabric. Testing as of October 2026. Results are from one run per scale and may vary based on configuration, software versions, and system conditions. [Claim ID: MI350-092]

3. LAMMPS (22 Jul 2025, Update 3) with Kokkos 4.6.2, SNAP potential, Thompson-Wood tungsten benchmark replicated 2,048 times in each dimension (17,179,869,184 atoms), 100 steps; one MPI rank per GPU; SPX/NPS1 mode; Open MPI 5.0.10, UCX 1.22, UCC 1.8, ROCm 7.2.4. 8 to 400 nodes (64 to 3,200 GPUs); loop time 4,481.6 s on 64 GPUs and 109.5 s on 3,200 GPUs. Strong-scaling efficiency = (T64 x 64) / (TN x N). System: AMD Instinct™ MI355X GPU cluster; each node with 2x AMD EPYC™ 9575F (64-core) CPUs, 8x AMD Instinct™ MI355X GPUs (288 GB HBM3E each), and 8x AMD Pensando™ Pollara 400 AI NICs (one per GPU, 3.2 Tb/s per node) on a fat-tree RoCEv2 Ethernet fabric. Testing as of October 2026. Results are from one run per scale and may vary based on configuration, software versions, and system conditions. [Claim ID: MI350-093]

4. M-PSDNS (OLCF-6 benchmark), FP32, ten steps per run; hipFFT with OpenMP target offload; SPX/NPS1 mode; Open MPI 5.0.10, UCX 1.22, UCC 1.8, ROCm AFAR 22.2; mesh grows from 1,536³ on 1 node to 12,288 x 6,144² on 128 nodes (1,024 GPUs), 4.53 x 10⁸ points per GPU. Weak-scaling efficiency = 1-node step time / N-node step time; throughput = grid points per RK2 step / step time. Chart also shows FP64 (2-node baseline; 59.6% at 128 nodes). System: AMD Instinct™ MI355X GPU cluster; each node with 2x AMD EPYC™ 9575F (64-core) CPUs, 8x AMD Instinct™ MI355X GPUs (288 GB HBM3E each), and 8x AMD Pensando™ Pollara 400 AI NICs (one per GPU, 3.2 Tb/s per node) on a fat-tree RoCEv2 Ethernet fabric. Testing as of October 2026. Results are from one run per scale and may vary based on configuration, software versions, and system conditions. [Claim ID: MI350-094]

Share:

Article By


Director of Product Management, Networking Technology and Solutions Group

Related Blogs