Introduction

CloverLeaf® is a mini-app that solves the compressible Euler equations on a Cartesian grid, using an explicit, second-order accurate method. This page describes the CloverLeaf-ref version which contains a hybrid OpenMP/MPI implementation.

Official website for CloverLeaf:  https://uk-mac.github.io/CloverLeaf

Official website for CloverLeaf-ref:  https://github.com/UK-MAC/CloverLeaf_ref

Build CloverLeaf-ref using Spack

Please refer to this link for getting started with Spack using AMD Zen Software Studio. 

		# Example for building CloverLeaf-ref using AOCC 
$ spack install cloverleaf-ref@1.3 %aocc ^openmpi fabrics=cma,ucx

Explanation of the command options:

Symbol

Meaning

%aocc

Build CloverLeaf using the AOCC compiler.

@1.3

Using version 1.3 of Cloverleaf-ref.

Cloverleaf-ref has two different versions (i.e., 1.1 and 1.3) and an up-to-date master commit (#0fdb917) which is included in the recipe package.py. 

^openmpi fabrics=cma,ucx

Use OpenMPI as the MPI provider and use the CMA network for efficient intra-node communication, falling back to the UCX network fabric, if required. 

Note: It is advised to specifically set the appropriate fabric for the host system if possible. Refer to Open MPI with AMD Zen Software Studio for more guidance.

 

Running CloverLeaf-ref

For running CloverLeaf-ref, it expects a file called clover.in in the working directory. Sample input files are provided with source distribution (available at /InputDecks). To run CloverLeaf-ref rename one of the files in /InputDecks as clover.in and copy it into the run directory where CloverLeaf-ref executable is invoked.


Note: Different AMD EPYC CPUs have fewer/more CPUs per L3 cache and should use a different MPI/OpenMP layout. The given example has been tested on a dual socket AMD 5th Gen EPYC™ 9755 Processor with 256 (128x2) cores.

Run Script for AMD EPYC™ Processors

		#!/bin/bash

# Load CloverLeaf-ref built with AOCC
spack load cloverleaf-ref@1.3

# Download data from GitHub
wget https://raw.githubusercontent.com/UK-MAC/CloverLeaf_CAF_buffers/master/clover_bm1024_short.in
mv clover_bm1024_short.in clover.in

# OpenMP Settings
export CORES_PER_CCX=8
NUM_CORES=$(nproc)
# OpenMP Settings
export OMP_NUM_THREADS=${CORES_PER_CCX}  # 8 threads per MPI rank. Recommended OMP_NUM_THREADS= #cores per L3 cache
export OMP_PROC_BIND=TRUE     # bind threads to specific resources
export OMP_PLACES="cores"     # bind threads to cores
NRANKS=$(( NUM_CORES / OMP_NUM_THREADS ))

# MPI settings suggested for a dual socket AMD 5th Gen EPYC™ 9755 Processor with 256 (128x2) cores.
MPI_OPTS="--map-by ppr:1:l3cache:pe=$OMP_NUM_THREADS -np $NRANKS"

# Run CloverLeaf-ref

mpirun "$MPI_OPTS clover_leaf
mpirun $MPI_OPTS clover_leaf

Note: The above build and run steps apply to CloverLeaf-ref-1.3, AOCC-5.1.0,AOCL-5.2.0 and OpenMPI-5.0.9 on Rocky Linux 9.5 (Blue Onyx) using Spack v1.1.0.dev0 and the builtin repo from spack-packages. (commit id : 952f6f5)

For technical support on the tools, benchmarks and applications that AMD offers on this page and related inquiries, reach out to us at toolchainsupport@amd.com.