Introduction

CloverLeaf® is a mini-app that solves the compressible Euler equations on a Cartesian grid, using an explicit, second-order accurate method. Each cell stores three values: energy, density, and pressure. A velocity vector is stored at each cell corner. This arrangement of data, with some quantities at cell centers, and others at cell corners is known as a staggered grid.

Official website for CloverLeaf:  https://uk-mac.github.io/CloverLeaf

This page describes the CloverLeaf-ref version. This version contains a hybrid OpenMP/MPI implementation.

Official website for CloverLeaf-ref:  https://github.com/UK-MAC/CloverLeaf_ref

Build CloverLeaf-ref using Spack

Please refer to this link for getting started with spack using AMD Zen Software Studio. 

    # Example for building CloverLeaf-ref using AOCC 
$ spack install cloverleaf-ref@1.3 %aocc ^openmpi fabrics=cma,ucx

Explanation of the command options:

Symbol

Meaning

%aocc

Build CloverLeaf using the AOCC compiler.

@1.3

Using v1.3 version of Cloverleaf-ref.

Cloverleaf-ref has two different versions (i.e., 1.1 and 1.3) and an up-to-date master commit (#0fdb917) which is included in the recipe package.py. 

^openmpi fabrics=cma,ucx

Use OpenMPI as the MPI provider and use the CMA network for efficient intra-node communication, falling back to the UCX network fabric, if required. 

Note: It is advised to specifically set the appropriate fabric for the host system if possible. Refer to Open MPI with AMD Zen Software Studio for more guidance.

 

Running CloverLeaf-ref

For running CloverLeaf-ref, It expects a file called clover.in in the working directory. Sample input files are provided with source distribution ( available at /InputDecks). To run CloverLeaf-ref rename one of the file in /InputDecks as clover.in and copy it into the run directory where CloverLeaf-ref executable is invoked.


Note: Different AMD EPYC CPUs have fewer/more CPUs per l3cache and should use a different MPI/OpenMP layout. Given example has been tested on a dual socket AMD 5th Gen EPYC™ 9755 Processor with 256 (128x2) cores.

Run Script for AMD EPYC™ Processors

    #!/bin/bash

# Load CloverLeaf-ref built with AOCC
spack load cloverleaf-ref@1.3

# Download data from GitHub
wget https://raw.githubusercontent.com/UK-MAC/CloverLeaf_CAF_buffers/master/clover_bm1024_short.in
mv clover_bm1024_short.in clover.in

# OpenMP Settings
export CORES_PER_CCX=8
NUM_CORES=$(nproc)
# OpenMP Settings
export OMP_NUM_THREADS=${CORES_PER_CCX}  # 8 threads per MPI rank. Recommended OMP_NUM_THREADS= #cores per L3 cache
export OMP_PROC_BIND=TRUE     # bind threads to specific resources
export OMP_PLACES="cores"     # bind threads to cores
NRANKS=$(( $NUM_CORES / $OMP_NUM_THREADS ))

# MPI settings suggested for a dual socket AMD 5th Gen EPYC™ 9755 Processor with 256 (128x2) cores.
MPI_OPTS="--map-by ppr:1:l3cache:pe=$OMP_NUM_THREADS -np $NRANKS"

# Run CloverLeaf-ref
mpirun $MPI_OPTS clover_leaf

Note: The above build and run steps apply to CloverLeaf-ref-1.3, AOCC-5.0.0, and OpenMPI-5.0.8 on Rocky Linux 9.5 (Blue Onyx) using Spack v1.1.0.dev0 and the builtin repo from spack-packages (commit id: 7824c23443).

For technical support on the tools, benchmarks and applications that AMD offers on this page and related inquiries, reach out to us at toolchainsupport@amd.com.