Chile’s NLHPC supercharged research performance with AMD

Chile’s NLHPC supercharged research performance with AMD

The National Laboratory for HPC in Chile doubled its scientific research performance per watt with AMD EPYC™ CPUs and AMD Instinct™ GPUs

High-performance computing is supercharging academic research around the world. One country in South America playing a prominent role is Chile via its National Laboratory for HPC (NLHPC, hosted at the Mathematical Modeling Center (CMM) of the University of Chile.). Like every institution relying on public funding, NLHPC needs the best performance it can get while keeping within national goals for sustainability. AMD EPYC™ CPUs and AMD Instinct™ GPUs delivered everything NLHPC required in its most recent HPC deployment.

“We have been providing computing resources to the scientific community in Chile for 15 years,” says Ginés Guerrero, Executive Head, NLHPC. “The University of Chile is the main partner, but 44 other universities are currently associated with us, accommodating virtually all the scientific researchers in Chile. We are funded by the country’s National Agency for Research and Development (ANID).” The broad range of its workloads makes NLHPC’s computing requirements particularly complex. “The number and diversity of users is challenging. Our 500 users come from 40 different areas of research, such as quantum chemistry, bioinformatics, astronomy, nanotechnology, and physics.”

Doubled performance per watt

“One of the most intense workloads is related to addressing climate change and the discovery of new materials,” says Guerrero. “For example, researchers are investigating photovoltaic material that captures solar energy more efficiently. Astronomy is a big thing in Chile, too. We have one of the best skies to study the stars. We have many, very significant telescopes financed internationally. In astrocomputing, we look at galaxy formation. There have been many papers written on supermassive black holes and on the images obtained from telescopes. This includes ALeRCE (Automatic Learning for the Rapid Classification of Events), which is an astronomic broker that handles all the images from telescopes. These need to be analyzed in real time, so that ALeRCE can send alerts of any changes, such as supernovas.”

“As the National Lab of Computing in Chile, we must keep up to date with what's happening in HPC hardware,” says Guerrero. “The AMD team has been very proactive in letting us know about the latest advances in its technology. When we began looking at 4^th Gen AMD EPYC CPUs, the team brought us up to speed on the CPU’s AVX-512 capabilities. We were also concerned about changing GPUs since we had been using our previous vendor for a long time. So, the AMD team put us in touch with US engineers to help with a smooth transition once we began integrating AMD GPUs into our system.”

NLHPC tested 4^th Gen AMD EPYC processors and AMD Instinct MI210 GPUs. “When we were installing the servers and running the testing, we were surprised by the results,” says Guerrero. During the bidding process, performance was the key factor but energy efficiency played a major role. “Our testing was very diverse. We had compiled at least 100 workloads. All the users said that the performance was much better with AMD EPYC CPUs, and we also employed the LINPACK benchmark. When we ran this on the previous architecture, we saw that the performance was up to 60 percent lower than the maximum theoretical one. With the current cluster powered by 4^th Gen AMD EPYC CPUs, we obtained more than 100 percent of the theoretical maximum. The energy that the previous cluster needed was also double the new one for the same results. With AMD EPYC CPUs, we obtained four times more performance with LINPACK at just double the energy.”

Boosting Chile’s scientific progress

“With AMD EPYC CPUs, many of our workloads could run right away,” says Guerrero. “But once we recompiled applications with GNU Complier Collection or AMD Optimizing C/C++ Compiler, we obtained much better performance than the code from our previous compiler. With the AMD Instinct GPUs, it was necessary to use new software. So we deployed the containers that were provided by the AMD Infinity Hub. The platforms available with AMD ROCm™ software were the best choice for this conversion.”

NLHPC’s new cluster runs 27 Lenovo ThinkSystem SR645 V3 servers with dual 128-core 4^th Gen AMD EPYC 9754 CPUs and 768GB of memory each, for a total of 6,912 cores for compute and access nodes. Its two Lenovo ThinkSystem SR675 V3 GPU servers run dual 24-core 4^th Gen AMD EPYC 9224 CPUs with six AMD Instinct MI210 GPUs each, comprising 12 accelerators. Lenovo’s Neptune water cooling system was used to maintain CPU and GPU operating temperatures during intensive AI workloads.

“The most important thing for us is to be able to contribute to Chile making scientific progress,” says Guerrero. “The more resources we can offer, the greater impact it has on the entire community.” The feedback from Chile’s scientists has been overwhelmingly positive. “A user running the Weather Research & Forecasting (WRF) Model told us that their workload was executing much faster than previously. The Nanoscale Molecular Dynamics (NAMD) model was another software application showing great performance.”

Towards a bright research future with AMD

The new cluster is improving how researchers deploy their workloads. “If we have many cores in one server, we can run shared memory tasks,” says Guerrero. “With AMD EPYC CPUs, we can now use 256 cores in a single task. Most of our users don't use MPI. They launch shared memory tasks. If they have 200-plus cores in one server, they can use 200 compute units. This was impossible when we only had servers with 40 cores like our previous CPUs. That's a great benefit.” NLHPC’s AMD Instinct accelerators are also delivering greater performance per watt. “With the AMD Instinct MI210 GPUs, we can run more floating-point operations per second with more energy efficiency. Most of our users employ that technology for molecular dynamics.”

"I often use a phrase that translates from Spanish to 'if you don’t compute you don’t compete'," says Guerrero. "If we don't invest in computing and move forward, we will be less competitive on the world stage." In Chile, there is a competition that provides $7 million for purchasing infrastructure using more GPUs for emerging workloads such as AI, and of course, we will participate in this competition. "With AI today, you're seeing countries investing enormous amounts of money to increase computing capacity. This has an impact for industry, for the state, for research, and for all citizens. The US, Europe, and China are the leaders, but in Latin America, there's still a great deal to be done, and we are working tirelessly to ensure progress is made.

Guerrero sees a bright future for the relationship between NLHPC and AMD to deliver this next phase. “On the Top 500 list of all the supercomputers in the world, it's clear that AMD processors are gaining ground,” says Guerrero. “They're there more and more. If you look at what is happening globally, you should consider integrating AMD technology because the results we've obtained are very positive. AMD has done a sensational job. The architecture is impeccable. This means a general advance for society to have better computing. The collaboration between the NLHPC and the AMD engineering team was the key to offering the best solution with more performance and the best energy efficiency. It has really been incredible.”