NWChem

The NWChem application container assumes that the server contains the required x86-64 CPU(s) and at least one of the listed AMD GPUs. Also, the server must have one of the required operating systems and the listed ROCm driver version installed to run the Docker container. The server must also have a Docker Engine installed to run the container.

Single-Node Server Requirements

Please visit the Docker Engine install web site at https://docs.docker.com/engine/install/ to install the latest Docker Engine for the operating system installed on the server. If Singularity use is planned, please visit https://sylabs.io/docs/ for the latest Singularity install documentation.

Please visit https://rocm.docs.amd.com/en/latest/ for ROCm installation procedures and validation checks.

Running Containers

Using Docker

Interactive

To run the container interactively run the following command, substituting the image name from the Pull Command section above if newer:

docker run --rm -it --ipc=host --device /dev/dri --device /dev/kfd --security-opt seccomp=unconfined -w /tmp amdih/nwchem:7.0.2.amd1 /bin/bash

(Note: running docker via sudo may be required on your system. For example: sudo docker run ....)

There are 3 "water" benchmarks of NWChem i.e. w3, w5 and w7. You can use the run-benchmark script bundled in the /opt/nwchem/ folder, to run the water benchmark, with a varying number of GPUs.

For example, if you have 8 GPUs available, the w3 benchmark may be run as:

run-benchmark w3 --ngpus 8

Miscellaneous benchmark input files can be found in the /opt/nwchem/misc/ subfolder. In order to run a particular individual simulation, customize the following command:

OMP_NUM_THREADS=8 mpirun --report-bindings -np 8 nwchem /opt/nwchem/gpu/water/w5/w5_1gpu.nw

In the above examples, the job is executed with 8 MPI ranks, each rank binding to a particular GPU and 8 OpenMP threads on a single-socket server with 64 physical cores. You can tune OMP_NUM_THREADS and the number of ranks as to better match your available hardware.

The number of available GPUs is controlled through NWChem's input files (.nw files), so if you want to run your simulation with more than one GPU, it is necessary to modify the cuda option inside each file and replace the given number by the number of desired GPUs to run on. Benchmarks files with different numbers of GPUs are available in /opt/nwchem/misc/ subfolders.

For example, to run an 8 GPU simulation of the w5 benchmark:

OMP_NUM_THREADS=8 mpirun --report-bindings -np 8 nwchem /opt/nwchem/gpu/water/w5/w5_8gpu.nw

That would invoke an 8-rank job, each rank utilizing 8 OpenMP threads, and it would utilize 8 GPUs on a node. Output files are printed to stdout, so output should be redirected to a file when running simulations and the output needs to be saved.

Non-Interactive

To run the container non-interactively, use a complete command line to execute the benchmark. For example, to run the default w3 benchmark on one GPU, substituting the image name from the Pull Command section above if newer:

docker run --rm -it --ipc=host --device /dev/dri --device /dev/kfd --security-opt seccomp=unconfined -w /tmp amdih/nwchem:7.0.2.amd1 run-benchmark w3 --ngpus 1

Environment variables are passed to Docker using the "-e" option, so the w5 example (from above) could also be run non-interactively, substituting the image name from the Pull Command section above if newer:

docker run --rm -it --ipc=host --device /dev/dri --device /dev/kfd --security-opt seccomp=unconfined -w /tmp -e OMP_NUM_THREADS=8 amdih/nwchem:7.0.2.amd1 mpirun --report-bindings -np 8 nwchem /opt/nwchem/gpu/water/w5/w5_1gpu.nw

Using Singularity

This section assumes that an up-to-date version of Singularity is installed on your system. Please consult with your system administrator or view official Singularity documentation.

Pull the Docker container and create the Singularity container, substituting the image name from the Pull Command section above if newer:

singularity pull nwchem.sif docker://amdih/nwchem:7.0.2.amd1

Run the Singularity based code, as follows, using the example from above:

export OMP_NUM_THREADS=8
singularity run ./nwchem.sif mpirun --report-bindings -np 8 nwchem /opt/nwchem/gpu/water/w5/w5_1gpu.nw

Licensing Information

Your use of this application is subject to the terms of the applicable component-level license identified below. To the extent any subcomponent in this container requires an offer for corresponding source code, AMD hereby makes such an offer for corresponding source code form, which will be made available upon request. By accessing and using this application, you are agreeing to fully comply with the terms of this license. If you do not agree to the terms of this license, do not access or use this application.

The application is provided in a container image format that includes the following separate and independent components: Ubuntu (License: Creative Commons CC-BY-SA version 3.0 UK licence), GlobalArrays (License: BSD 3), OpenMPI (License: BSD 3-Clause), OpenUCX (License: BSD 3-Clause), OpenBLAS (License: BSD), ScaLAPACK (License: BSD), ROCm (License: Custom/MIT/Apache V2.0/UIUC OSL), NWChem (License: Educational Community License (ECL) 2.0). Additional third-party content in this container may be subject to additional licenses and restrictions. The components are licensed to you directly by the party that owns the content pursuant to the license terms included with such content and is not licensed to you by AMD. ALL THIRD-PARTY CONTENT IS MADE AVAILABLE BY AMD “AS IS” WITHOUT A WARRANTY OF ANY KIND. USE OF THE CONTAINER IS DONE AT YOUR SOLE DISCRETION AND UNDER NO CIRCUMSTANCES WILL AMD BE LIABLE TO YOU FOR ANY THIRD-PARTY CONTENT. YOU ASSUME ALL RISK AND ARE SOLELY RESPONSIBLE FOR ANY DAMAGES THAT MAY ARISE FROM YOUR USE OF THE CONTAINER.

See the NWChem public repo for more information on licensing.

Disclaimer

The information contained herein is for informational purposes only, and is subject to change without notice. In addition, any stated support is planned and is also subject to change. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale.  

Notices and Attribution 

© 2023 Advanced Micro Devices, Inc. All rights reserved. AMD, the AMD Arrow logo, Instinct, Radeon Instinct, ROCm, and combinations thereof are trademarks of Advanced Micro Devices, Inc. 

Docker and the Docker logo are trademarks or registered trademarks of Docker, Inc. in the United States and/or other countries. Docker, Inc. and other parties may also have trademark rights in other terms used herein.  Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.   

All other trademarks and copyrights are property of their respective owners and are only mentioned for informative purposes.