8. AOCL-ScaLAPACK#
AOCL-ScaLAPACK is a library of high-performance linear algebra routines for parallel distributed memory machines. It can be used to solve linear systems, least squares problems, eigenvalue problems, and singular value problems. AOCL-ScaLAPACK is optimized for AMD “Zen”-based processors. It depends on the external libraries BLAS and LAPACK; thus, the use of AOCL-BLAS and AOCL-LAPACK is recommended.
Following table lists the CMake flags supported by AOCL-ScaLAPACK:
Build Feature |
CMake Command |
|---|---|
Select debug or Release mode build |
|
Shared library |
|
Static library |
|
Provide external BLAS/AOCL-BLAS library |
|
Provide external LAPACK/AOCL- LAPACK library |
Note If linked with AOCL-LAPACK, include the AOCL-Utils library path too. |
Integer bit length |
|
Build only the library |
|
Build only the test project |
(If this option is set and |
Flags disabled by default |
|
Set OpenMP library path |
|
Enable Trace and Log feature |
|
Configurable flag for Windows builds |
|
8.1. Installation#
AOCL-ScaLAPACK can be installed from source or pre-built binaries.
8.1.1. Building AOCL-ScaLAPACK from Source on Linux#
GitHub URL: amd/aocl-scalapack
Prerequisites
Building AOCL-ScaLAPACK library requires linking to the following libraries installed using pre-built binaries or built from source:
GCC, g++, and Gfortran
AOCC
AOCL-BLAS
AOCL-LAPACK
AOCL-Utils
An MPI library (AMD has validated with OpenMPI library)
For more information on validated versions of the compilers and CMake, refer to Validation Matrix.
Complete the following steps to build AOCL-ScaLAPACK from source:
Clone the GitHub repository (amd/aocl-scalapack.git).
Execute the command:
$ cd aocl-scalapack
CMake as follows:
Create a new directory. For example, build:
$ mkdir build $ cd build
Export
PATHandLD_LIBRARY_PATHto the lib and bin folders of the MPI installation respectively:$ export PATH=<MPI installation folder>/bin:$PATH $ export LD_LIBRARY_PATH=<MPI installation folder>/lib:$LD_LIBRARY_PATH
Run cmake command based on the compiler and the type of library generation required.
Note
AOCL-LAPACK is dependent on the AOCL-Utils library, which in turn depends on libstdc++. Hence, you must link with AOCL-Utils and libstdc++(-lstdc++) along with the AOCL-LAPACK library while specifying the path for LAPACK_LIBRARIES in the CMake flags.
Library Type |
Threading |
Command |
|---|---|---|
Static |
Single-thread AOCL-BLAS |
|
Multi-thread AOCL-BLAS |
|
|
Shared |
Single-thread AOCL-BLAS |
|
Multi-thread AOCL-BLAS |
|
Library Type |
Threading |
Command |
|---|---|---|
Static |
Single-thread AOCL-BLAS |
|
Multi-thread AOCL-BLAS |
|
|
Shared |
Single-thread AOCL-BLAS |
|
Multi-thread AOCL-BLAS |
|
On Linux, the inbuilt communications sub-module of AOCL-ScaLAPACK, called Basic Linear Algebra Communication Subprograms (BLACS), exposes the API symbols in lower case with underscore format.
You can build AOCL-ScaLAPACK with an external BLACS library on Linux using the following configure option:
Example: To build static library with external BLACS library:
$ cmake .. -DBUILD_SHARED_LIBS=OFF -DBLAS_LIBRARIES="-fopenmp <path
to AOCL-BLAS library>/ libblis-mt.a" -DLAPACK_LIBRARIES="<path to
AOCL-LAPACK library>/libflame.a;<path to AOCL-Utils
library>/libaoclutils.a;-lstdc++" -DBLACS_LIBRARIES=<path to BLACS
library>/libBLACS.a - DCMAKE_C_COMPILER=mpicc
-DCMAKE_Fortran_COMPILER=mpif90 -DUSE_OPTIMIZED_LAPACK_BLAS=OFF
You can build AOCL-ScaLAPACK with Intel MPI and ICC compiler tool chain using the following configure option.
Example: To build a static library with Intel MPI and ICC compiler:
$ cmake .. -DBUILD_SHARED_LIBS=OFF -DBLAS_LIBRARIES="-fopenmp <path to
AOCL-BLAS library>/ libblis-mt.a" -DLAPACK_LIBRARIES="<path to
AOCL-LAPACK library>/libflame.a;<path to AOCL-Utils
library>/libaoclutils.a;-lstdc++" -DCMAKE_C_COMPILER=mpiicc -
DCMAKE_Fortran_COMPILER=mpiifort -DUSE_OPTIMIZED_LAPACK_BLAS=OFF;
Ensure CMake locates AOCL-LAPACK and AOCL-BLAS libraries. On completion, a message, “LAPACK routine dgesv is found: 1” similar to the following in CMake output is displayed:
... -- CHECKING BLAS AND LAPACK LIBRARIES -- --> LAPACK supplied by user is <path>/libflame.a. -- --> LAPACK routine dgesv is found: 1. -- --> LAPACK supplied by user is WORKING, will use <path>/libflame.a. -- BLAS library: <path>/libblis.a -- LAPACK library: <path>/libflame.a ...
Compile the code:
$ make -j
When the building process is complete, the AOCL-ScaLAPACK library is created in the lib directory. The test application binaries are generated in the <aocl-scalapack>/build/TESTING folder.
8.1.2. Using Pre-Built Libraries#
AOCL-ScaLAPACK library binaries for Linux are available at the following URL:
https://www.amd.com/en/developer/aocl/scalapack.html
Also, AOCL-ScaLAPACK binary can be installed from the AOCL master installer tar file available at the following URL:
https://www.amd.com/en/developer/aocl.html
The tar file includes pre-built binaries of other AMD Libraries as explained in Using Master Package.
8.2. Usage#
You can find the applications demonstrating the usage of ScaLAPACK APIs in the TESTING directory of ScaLAPACK source package:
$ cd aocl-scalapack/TESTING
8.3. Building AOCL-ScaLAPACK from Source on Windows#
GitHub URL: amd/aocl-scalapack
AOCL-ScaLAPACK uses CMake along with Microsoft Visual Studio for building the binaries from the sources on Windows. The following sections explain the GUI and command-line schemes of building the binaries and test suite.
Prerequisites
The following prerequisites must be met:
AOCL-BLAS, AOCL-LAPACK, and AOCL-Utils libraries
Windows10/11 or Windows Server 2019/2022
LLVM 16/17
LLVM plug-in for Microsoft Visual Studio (if latest version of LLVM is installed separately, this plug-in enables linking Microsoft Visual Studio with the installed LLVM tool-chain)
For more information on CMake versions validated, refer to Build Utilities
Intel MPI compiler
Microsoft Visual Studio 2019 (build 16.8.7) through 2022 (build 17.3.2)
Microsoft Visual Studio tools
Python development
Desktop development with C++: C++ Clang-Cl for v142 build tool (x64 or x86)
8.3.1. Building AOCL-ScaLAPACK Using GUI#
8.3.1.1. Preparing Project with CMake GUI#
Complete the following steps to prepare the project with CMake GUI:
Set the source (folder containing aocl-scalapack source code) and build (folder in which the project files will be generated, for example, out) folder paths. It is not recommended to use the folder named build as a folder with that name exists at the top of AOCL-LAPACK source tree.
Click on the Configure button to prepare the project options.
Set the generator to Visual Studio 17 2022 and the compiler to ClangCl or LLVM.
Select the available and recommended options as follows:
Figure 8.1 AOCL-ScaLAPACK CMake Options#
Click the Generate button and then Open Project.
8.3.1.2. Building the Project in Visual Studio GUI#
Complete the following steps in Microsoft Visual Studio GUI:
Open the project generated by CMake (build folder) in Preparing Project with CMake GUI.
To generate the AOCL-ScaLAPACK binaries, build the ScaLAPACK project. The library files would be generated in the folder out based on the project settings.
For example:
aocl-scalapack/out/lib/Release/scalapack.lib
aocl-scalapack/out/Testing/Release/scalapack.dll
8.3.2. Building AOCL-ScaLAPACK Using Command-Line Arguments#
The project configuration and build procedures can be triggered from the command prompt as follows:
8.3.2.1. Configuring the Project in Command Prompt#
Complete the following steps to configure the project using the command prompt:
In the ScaLAPACK project folder, create a folder out.
Open the command prompt in that directory and run the following command:
$ cmake -S .. -B . -G "Visual Studio 17 2022" -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DCDEFS=UpCase /NoChange/Add\_ -DBUILD_STATIC_LIBS=OFF -DBLAS_LIBRARIES="<path to AOCL-BLAS library>/AOCL-LibBlis-Win-MT-dll.lib" -DLAPACK_LIBRARIES="<path to AOCL-LAPACK library>/AOCL-LibFLAME-Win-MT-dll.lib;<path to AOCL-Utils library>/libaoclutils.lib"
8.3.2.2. Building the Project in Command Prompt#
Complete the following steps to build the project using the command prompt:
Open command prompt in the aocl-scalapack/out directory.
Invoke CMake with the build command and release or debug option. For example:
$ cmake --build . --config Release
The library files would be generated inside the folder Release or Debug based on the project settings.
On Windows, the inbuilt communications submodule of ScaLAPACK, called Basic Linear Algebra Communication Subprograms(BLACS), exposes the API symbols in upper case without underscore format.
8.3.3. Building and Running the Individual Tests#
Microsoft Visual Studio projects for the individual tests are generated as part of the CMake generate step. Refer the previous sections to build the test projects from Microsoft Visual Studio GUI or command prompt.
The test application binaries are generated in the folder <aocl-scalapack>/out/Testing/Release or
<aocl-scalapack>/out/Testing/Debug based on the project settings. Run the tests from the command prompt as follows:
Release> mpiexec xcbrd.exe
8.4. Checking AOCL-ScaLAPACK Operation Progress#
AOCL libraries perform tasks that can be computationally expensive. The AOCL Progress feature provides a mechanism, for a selected set of APIs, for the application to check how far a computation has progressed through a callback function.
Usage
The application must define a callback function in a specific format and register this callback function with the AOCL-ScaLAPACK library.
The callback function prototype must be defined as follows:
int aocl_scalapack_progress(
const char* api, const integer *lenapi,
const integer *progress, const integer *mpi_rank,
const integer *total_mpi_processes)
The following table explains AOCL-ScaLAPACK Progress feature callback function parameters:
Parameter |
Purpose |
|---|---|
api |
Name of the API running currently |
lenapi |
Length of the API name character buffer |
progress |
Linear progress made in the current thread so far |
mpi_rank |
Current process rank |
total_mpi_processes |
Total number of processes used to perform the operation |
Callback Registration
The callback function must be registered with the library to report the progress:
aocl_scalapack_set_progress(aocl_scalapack_progress);
Example:
int AOCL_progress(const char* const api, const int *lenapi, const
int *progress, const int *mpi_rank, const int *total_mpi_processes)
{
printf( "In AOCL Progress MPI Rank: %i API: %s progress: %i MPI processes: %i\n",
*mpi_rank, api, *progress, *total_mpi_processes);
return 0;
}
Limitation
Currently, AOCL-ScaLAPACK progress feature is supported only on Linux.
8.5. Additional Features#
The additional features supported at runtime through the environment variable setting are as follows:
Feature |
Description |
Environment Variable |
OS Support |
|---|---|---|---|
Trace |
Enable function call trace |
AOCL_SL_TRACE |
Linux, Windows |
Log |
Enable logging of input argument values |
AOCL_SL_LOG |
Linux, Windows |
AOCL Progress |
Check how far a computation has progressed through a callback function for 3 major factorization APIs (LU, QR, Cholesky) for all data type variants |
AOCL_SL_PROGRESS |
Linux, Windows |
Note
To use the Trace and Log feature, ensure that
AOCL-ScaLAPACK is built with the flag ENABLE_DTL=ON.
Example:
export AOCL_SL_LOG=1in Linux enables the log file at run time.set AOCL_SL_PROGRESS=1in Windows enables the AOCL Progress feature at run time.