8. AOCL-ScaLAPACK#

AOCL-ScaLAPACK is a library of high-performance linear algebra routines for parallel distributed memory machines. It can be used to solve linear systems, least squares problems, eigenvalue problems, and singular value problems. AOCL-ScaLAPACK is optimized for AMD “Zen”-based processors. It depends on the external libraries BLAS and LAPACK; thus, the use of AOCL-BLAS and AOCL-LAPACK is recommended.

Following table lists the CMake flags supported by AOCL-ScaLAPACK:

Table 8.1 AOCL-ScaLAPACK CMake Parameter List#

Build Feature

CMake Command

Select debug or Release mode build

CMAKE_BUILD_TYPE=Debug/Release

Shared library

  • BUILD_SHARED_LIBS=ON

  • BUILD_STATIC_LIBS=OFF

Static library

  • BUILD_STATIC_LIBS=ON

  • BUILD_SHARED_LIBS=OFF

Provide external BLAS/AOCL-BLAS library

BLAS_LIBRARIES=<Path to BLAS/AOCL-BLAS lib>

Provide external LAPACK/AOCL- LAPACK library

LAPACK_LIBRARIES=<Path to lapack/AOCL-LAPACK lib>

Note

If linked with AOCL-LAPACK, include the AOCL-Utils library path too.

Integer bit length

  • ENABLE_ILP64=ON => 64-bit integer length

  • ENABLE_ILP64=OFF => 32-bit integer length

Build only the library

SCALAPACK_BUILD_SOURCE=ON

Build only the test project

SCALAPACK_BUILD_TESTS=ON

(If this option is set and SCALAPACK_BUILD_SOURCE is OFF, then it requires path to AOCL-ScaLAPACK library to be set using SCALAPACK_LIBRARY_PATH)

Flags disabled by default

USE_OPTIMIZED_LAPACK_BLAS

Set OpenMP library path

OpenMP_libomp_LIBRARY=<path to OpenMP library>

Enable Trace and Log feature

ENABLE_DTL=OFF

Configurable flag for Windows builds

  • CDEFS = NoChange (Fortran to C interface API’s called with small case)

  • CDEFS = UpCase (Fortran to C interface API’s called with upper case)

  • CDEFS = Add_ (Fortran to C interface API’s called with small case and underscore)

  • DCDEFS=NoChange

8.1. Installation#

AOCL-ScaLAPACK can be installed from source or pre-built binaries.

8.1.1. Building AOCL-ScaLAPACK from Source on Linux#

GitHub URL: amd/aocl-scalapack

Prerequisites

Building AOCL-ScaLAPACK library requires linking to the following libraries installed using pre-built binaries or built from source:

  • GCC, g++, and Gfortran

  • AOCC

  • AOCL-BLAS

  • AOCL-LAPACK

  • AOCL-Utils

  • An MPI library (AMD has validated with OpenMPI library)

For more information on validated versions of the compilers and CMake, refer to Validation Matrix.

Complete the following steps to build AOCL-ScaLAPACK from source:

  1. Clone the GitHub repository (amd/aocl-scalapack.git).

  2. Execute the command:

    $ cd aocl-scalapack
    
  3. CMake as follows:

    1. Create a new directory. For example, build:

      $ mkdir build
      $ cd build
      
    2. Export PATH and LD_LIBRARY_PATH to the lib and bin folders of the MPI installation respectively:

      $ export PATH=<MPI installation folder>/bin:$PATH
      $ export LD_LIBRARY_PATH=<MPI installation folder>/lib:$LD_LIBRARY_PATH
      
    3. Run cmake command based on the compiler and the type of library generation required.

Note

AOCL-LAPACK is dependent on the AOCL-Utils library, which in turn depends on libstdc++. Hence, you must link with AOCL-Utils and libstdc++(-lstdc++) along with the AOCL-LAPACK library while specifying the path for LAPACK_LIBRARIES in the CMake flags.

Table 8.2 GCC Compiler and Type of Library#

Library Type

Threading

Command

Static

Single-thread AOCL-BLAS

$ cmake .. -DBUILD_SHARED_LIBS=OFF -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLAS library>/libblis.a" -DLAPACK_LIBRARIES="<path to AOCL-LAPACK library>/libflame.a;<path to AOCL-Utils library>/libaoclutils.a;-lstdc++" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-DENABLE_ILP64=ON]

Multi-thread AOCL-BLAS

$ cmake .. -DBUILD_SHARED_LIBS=OFF -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLAS library>/libblis-mt.a" -DLAPACK_LIBRARIES="-lstdc++ <path to AOCL-LAPACK library>/libflame.a;<path to AOCL-Utils library>/libaoclutils.a;-lstdc++" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-DENABLE_ILP64=ON]

Shared

Single-thread AOCL-BLAS

$ cmake .. -DBUILD_SHARED_LIBS=ON -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLAS library>/libblis.so" -DLAPACK_LIBRARIES="<path to AOCL-LAPACK library>/libflame.so;<path to AOCL-Utils library>/libaoclutils.so;-lstdc++" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-DENABLE_ILP64=ON]

Multi-thread AOCL-BLAS

$ cmake .. -DBUILD_SHARED_LIBS=ON -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLAS library>/libblis-mt.so" -DLAPACK_LIBRARIES="-lstdc++ <path to AOCL-LAPACK library>/libflame.so;<path to AOCL-Utils library>/libaoclutils.so;-lstdc++" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-DENABLE_ILP64=ON]

Table 8.3 AOCC Compiler and Type of Library#

Library Type

Threading

Command

Static

Single-thread AOCL-BLAS

$ cmake .. -DBUILD_SHARED_LIBS=OFF -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLAS library>/libblis.a" -DLAPACK_LIBRARIES="<path to AOCL-LAPACK library>/libflame.a;<path to AOCL-Utils library>/libaoclutils.a;-lstdc++" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-DENABLE_ILP64=ON]

Multi-thread AOCL-BLAS

$ cmake .. -DBUILD_SHARED_LIBS=OFF -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLAS library>/libblis-mt.a" -DLAPACK_LIBRARIES="-lstdc++ <path to AOCL-LAPACK library>/libflame.a;<path to AOCL-Utils library>/libaoclutils.a;-lstdc++" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-DENABLE_ILP64=ON]

Shared

Single-thread AOCL-BLAS

$ cmake .. -DBUILD_SHARED_LIBS=ON -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLAS library>/libblis.so" -DLAPACK_LIBRARIES="<path to AOCL-LAPACK library>/libflame.so;<path to AOCL-Utils library>/libaoclutils.so;-lstdc++" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-DENABLE_ILP64=ON]

Multi-thread AOCL-BLAS

$ cmake .. -DBUILD_SHARED_LIBS=ON -DBLAS_LIBRARIES="-fopenmp <path to AOCL-BLAS library>/libblis-mt.so" -DLAPACK_LIBRARIES="-lstdc++ <path to AOCL-LAPACK library>/libflame.so;<path to AOCL-Utils library>/libaoclutils.so;-lstdc++" -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF [-DENABLE_ILP64=ON]

On Linux, the inbuilt communications sub-module of AOCL-ScaLAPACK, called Basic Linear Algebra Communication Subprograms (BLACS), exposes the API symbols in lower case with underscore format.

You can build AOCL-ScaLAPACK with an external BLACS library on Linux using the following configure option:

Example: To build static library with external BLACS library:

$ cmake .. -DBUILD_SHARED_LIBS=OFF -DBLAS_LIBRARIES="-fopenmp <path
to AOCL-BLAS library>/ libblis-mt.a" -DLAPACK_LIBRARIES="<path to
AOCL-LAPACK library>/libflame.a;<path to AOCL-Utils
library>/libaoclutils.a;-lstdc++" -DBLACS_LIBRARIES=<path to BLACS
library>/libBLACS.a - DCMAKE_C_COMPILER=mpicc
-DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF

You can build AOCL-ScaLAPACK with Intel MPI and ICC compiler tool chain using the following configure option.

Example: To build a static library with Intel MPI and ICC compiler:

$ cmake .. -DBUILD_SHARED_LIBS=OFF -DBLAS_LIBRARIES="-fopenmp <path to
AOCL-BLAS library>/ libblis-mt.a" -DLAPACK_LIBRARIES="<path to
AOCL-LAPACK library>/libflame.a;<path to AOCL-Utils
library>/libaoclutils.a;-lstdc++" -DCMAKE_C_COMPILER=mpiicc -
DCMAKE_Fortran_COMPILER=mpiifort -DUSE_OPTIMIZED_LAPACK_BLAS=OFF;
  1. Ensure CMake locates AOCL-LAPACK and AOCL-BLAS libraries. On completion, a message, “LAPACK routine dgesv is found: 1” similar to the following in CMake output is displayed:

    ...
    -- CHECKING BLAS AND LAPACK LIBRARIES
    -- --> LAPACK supplied by user is <path>/libflame.a.
    -- --> LAPACK routine dgesv is found: 1.
    -- --> LAPACK supplied by user is WORKING, will use
    <path>/libflame.a.
    -- BLAS library: <path>/libblis.a
    -- LAPACK library: <path>/libflame.a
    ...
    
  2. Compile the code:

    $ make -j
    

When the building process is complete, the AOCL-ScaLAPACK library is created in the lib directory. The test application binaries are generated in the <aocl-scalapack>/build/TESTING folder.

8.1.2. Using Pre-Built Libraries#

AOCL-ScaLAPACK library binaries for Linux are available at the following URL:

https://www.amd.com/en/developer/aocl/scalapack.html

Also, AOCL-ScaLAPACK binary can be installed from the AOCL master installer tar file available at the following URL:

https://www.amd.com/en/developer/aocl.html

The tar file includes pre-built binaries of other AMD Libraries as explained in Using Master Package.

8.2. Usage#

You can find the applications demonstrating the usage of ScaLAPACK APIs in the TESTING directory of ScaLAPACK source package:

$ cd aocl-scalapack/TESTING

8.3. Building AOCL-ScaLAPACK from Source on Windows#

GitHub URL: amd/aocl-scalapack

AOCL-ScaLAPACK uses CMake along with Microsoft Visual Studio for building the binaries from the sources on Windows. The following sections explain the GUI and command-line schemes of building the binaries and test suite.

Prerequisites

The following prerequisites must be met:

  • AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils libraries

  • Windows10/11 or Windows Server 2019/2022

  • LLVM 16/17

  • LLVM plug-in for Microsoft Visual Studio (if latest version of LLVM is installed separately, this plug-in enables linking Microsoft Visual Studio with the installed LLVM tool-chain)

  • For more information on CMake versions validated, refer to Build Utilities

  • Intel MPI compiler

  • Microsoft Visual Studio 2019 (build 16.8.7) through 2022 (build 17.3.2)

  • Microsoft Visual Studio tools

    • Python development

    • Desktop development with C++: C++ Clang-Cl for v142 build tool (x64 or x86)

8.3.1. Building AOCL-ScaLAPACK Using GUI#

8.3.1.1. Preparing Project with CMake GUI#

Complete the following steps to prepare the project with CMake GUI:

  1. Set the source (folder containing aocl-scalapack source code) and build (folder in which the project files will be generated, for example, out) folder paths. It is not recommended to use the folder named build as a folder with that name exists at the top of AOCL-LAPACK source tree.

  2. Click on the Configure button to prepare the project options.

  3. Set the generator to Visual Studio 17 2022 and the compiler to ClangCl or LLVM.

  4. Select the available and recommended options as follows:

    _images/image15_aocl.jpeg

    Figure 8.1 AOCL-ScaLAPACK CMake Options#

  5. Click the Generate button and then Open Project.

8.3.1.2. Building the Project in Visual Studio GUI#

Complete the following steps in Microsoft Visual Studio GUI:

  1. Open the project generated by CMake (build folder) in Preparing Project with CMake GUI.

  2. To generate the AOCL-ScaLAPACK binaries, build the ScaLAPACK project. The library files would be generated in the folder out based on the project settings.

For example:

aocl-scalapack/out/lib/Release/scalapack.lib

aocl-scalapack/out/Testing/Release/scalapack.dll

8.3.2. Building AOCL-ScaLAPACK Using Command-Line Arguments#

The project configuration and build procedures can be triggered from the command prompt as follows:

8.3.2.1. Configuring the Project in Command Prompt#

Complete the following steps to configure the project using the command prompt:

  1. In the ScaLAPACK project folder, create a folder out.

  2. Open the command prompt in that directory and run the following command:

    $ cmake -S .. -B . -G "Visual Studio 17 2022"
    -DCMAKE_BUILD_TYPE=Release
    -DBUILD_SHARED_LIBS=ON -DCDEFS=UpCase /NoChange/Add\_
    -DBUILD_STATIC_LIBS=OFF -DBLAS_LIBRARIES="<path to AOCL-BLAS
    library>/AOCL-LibBlis-Win-MT-dll.lib"
    -DLAPACK_LIBRARIES="<path to AOCL-LAPACK
    library>/AOCL-LibFLAME-Win-MT-dll.lib;<path to AOCL-Utils
    library>/libaoclutils.lib"
    

8.3.2.2. Building the Project in Command Prompt#

Complete the following steps to build the project using the command prompt:

  1. Open command prompt in the aocl-scalapack/out directory.

  2. Invoke CMake with the build command and release or debug option. For example:

    $ cmake --build . --config Release
    

The library files would be generated inside the folder Release or Debug based on the project settings.

On Windows, the inbuilt communications submodule of ScaLAPACK, called Basic Linear Algebra Communication Subprograms(BLACS), exposes the API symbols in upper case without underscore format.

8.3.3. Building and Running the Individual Tests#

Microsoft Visual Studio projects for the individual tests are generated as part of the CMake generate step. Refer the previous sections to build the test projects from Microsoft Visual Studio GUI or command prompt.

The test application binaries are generated in the folder <aocl-scalapack>/out/Testing/Release or

<aocl-scalapack>/out/Testing/Debug based on the project settings. Run the tests from the command prompt as follows:

Release> mpiexec xcbrd.exe

8.4. Checking AOCL-ScaLAPACK Operation Progress#

AOCL libraries perform tasks that can be computationally expensive. The AOCL Progress feature provides a mechanism, for a selected set of APIs, for the application to check how far a computation has progressed through a callback function.

Usage

The application must define a callback function in a specific format and register this callback function with the AOCL-ScaLAPACK library.

The callback function prototype must be defined as follows:

int aocl_scalapack_progress(
   const char* api, const integer *lenapi,
   const integer *progress, const integer *mpi_rank,
   const integer *total_mpi_processes)

The following table explains AOCL-ScaLAPACK Progress feature callback function parameters:

Table 8.4 AOCL-ScaLAPACK Progress Feature Callback Function Parameters#

Parameter

Purpose

api

Name of the API running currently

lenapi

Length of the API name character buffer

progress

Linear progress made in the current thread so far

mpi_rank

Current process rank

total_mpi_processes

Total number of processes used to perform the operation

Callback Registration

The callback function must be registered with the library to report the progress:

aocl_scalapack_set_progress(aocl_scalapack_progress);

Example:

int AOCL_progress(const char* const api, const int *lenapi, const
  int *progress, const int *mpi_rank, const int *total_mpi_processes)
{
  printf( "In AOCL Progress MPI Rank: %i API: %s progress: %i MPI processes: %i\n",
    *mpi_rank, api, *progress, *total_mpi_processes);
  return 0;
}

Limitation

Currently, AOCL-ScaLAPACK progress feature is supported only on Linux.

8.5. Additional Features#

The additional features supported at runtime through the environment variable setting are as follows:

Table 8.5 Additional Features#

Feature

Description

Environment Variable

OS Support

Trace

Enable function call trace

AOCL_SL_TRACE

Linux, Windows

Log

Enable logging of input argument values

AOCL_SL_LOG

Linux, Windows

AOCL Progress

Check how far a computation has progressed through a callback function for 3 major factorization APIs (LU, QR, Cholesky) for all data type variants

AOCL_SL_PROGRESS

Linux, Windows

Note

To use the Trace and Log feature, ensure that AOCL-ScaLAPACK is built with the flag ENABLE_DTL=ON.

Example:

  • export AOCL_SL_LOG=1 in Linux enables the log file at run time.

  • set AOCL_SL_PROGRESS=1 in Windows enables the AOCL Progress feature at run time.