LUMI (CSC)
The LUMI cluster is located at CSC (Finland). Each node contains 4 AMD MI250X GPUs, each with 2 Graphics Compute Dies (GCDs) for a total of 8 GCDs per node. You can think of the 8 GCDs as 8 separate GPUs, each having 64 GB of high-bandwidth memory (HBM2E).
Introduction
If you are new to this system, please see the following resources:
Batch system: Slurm
-
$HOME: single user, intended to store user configuration files and personal data (20GB default quota)/project/$proj: shared with all members of a project, purged at the end of a project (50 GB default quota)/scratch/$proj: temporary storage, main storage to be used for disk I/O needs when running simulations on LUMI, purged every 90 days (50TB default quota)
Installation
Use the following commands to download the WarpX source code and switch to the correct branch:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
We use the following modules and environments on the system ($HOME/lumi_warpx.profile).
# please set your project account
#export proj=<yourProject>
# required dependencies
module load LUMI/23.03 partition/G
module load rocm/5.2.3 # waiting for 5.5 for next bump
module load buildtools/23.03
# optional: for QED lookup table generation support
module load Boost/1.81.0-cpeCray-23.03
# optional: for openPMD support
module load cray-hdf5/1.12.2.3
# optional: for Python bindings or libEnsemble
module load cray-python/3.9.13.1
# an alias to request an interactive batch node for one hour
# for paralle execution, start on the batch node: srun <command>
alias getNode="salloc -A $proj -J warpx -t 01:00:00 -p dev-g -N 1 --ntasks-per-node=8 --gpus-per-task=1 --gpu-bind=closest"
# an alias to run a command on a batch node for up to 30min
# usage: runNode <command>
alias runNode="srun -A $proj -J warpx -t 00:30:00 -p dev-g -N 1 --ntasks-per-node=8 --gpus-per-task=1 --gpu-bind=closest"
# GPU-aware MPI
export MPICH_GPU_SUPPORT_ENABLED=1
# optimize ROCm/HIP compilation for MI250X
export AMREX_AMD_ARCH=gfx90a
# compiler environment hints
export CC=$(which cc)
export CXX=$(which CC)
export FC=$(which ftn)
export CFLAGS="-I${ROCM_PATH}/include"
export CXXFLAGS="-I${ROCM_PATH}/include -Wno-pass-failed"
export LDFLAGS="-L${ROCM_PATH}/lib -lamdhip64"
We recommend to store the above lines in a file, such as $HOME/lumi_warpx.profile, and load it into your shell after a login:
source $HOME/lumi_warpx.profile
And since LUMI does not yet provide a module for them, install c-blosc and ADIOS2:
export CMAKE_PREFIX_PATH=${HOME}/sw/lumi/gpu/c-blosc-1.21.1:$CMAKE_PREFIX_PATH
export CMAKE_PREFIX_PATH=${HOME}/sw/lumi/gpu/adios2-2.8.3:$CMAKE_PREFIX_PATH
# c-blosc (I/O compression)
git clone -b v1.21.1 https://github.com/Blosc/c-blosc.git src/c-blosc
rm -rf src/c-blosc-build
cmake -S src/c-blosc -B src/c-blosc-build -DBUILD_TESTS=OFF -DBUILD_BENCHMARKS=OFF -DDEACTIVATE_AVX2=OFF -DCMAKE_INSTALL_PREFIX=${HOME}/sw/lumi/gpu/c-blosc-1.21.1
cmake --build src/c-blosc-build --target install --parallel 16
# ADIOS2
git clone -b v2.8.3 https://github.com/ornladios/ADIOS2.git src/adios2
rm -rf src/adios2-build
cmake -S src/adios2 -B src/adios2-build -DADIOS2_USE_Blosc=ON -DADIOS2_USE_Fortran=OFF -DADIOS2_USE_Python=OFF -DADIOS2_USE_ZeroMQ=OFF -DCMAKE_INSTALL_PREFIX=${HOME}/sw/lumi/gpu//adios2-2.8.3
cmake --build src/adios2-build --target install -j 16
Then, cd into the directory $HOME/src/warpx and use the following commands to compile:
cd $HOME/src/warpx
rm -rf build
cmake -S . -B build -DWarpX_COMPUTE=HIP
cmake --build build -j 16
The general cmake compile-time options apply as usual.
That’s it!
A 3D WarpX executable is now in build/bin/ and can be run with a 3D example inputs file.
Most people execute the binary directly or copy it out to a location in /scratch/<project>.
Running
MI250X GPUs (2x64 GB)
In non-interactive runs:
#!/bin/bash -l
#SBATCH -A <project id>
#SBATCH --job-name=warpx
#SBATCH --output=%x-%j.out
#SBATCH --error=%x-%j.err
#SBATCH --partition=standard-g
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=8
#SBATCH --gpus-per-node=8
#SBATCH --time=00:10:00
export MPICH_GPU_SUPPORT_ENABLED=1
# note (12-12-22)
# this environment setting is currently needed on LUMI to work-around a
# known issue with Libfabric
#export FI_MR_CACHE_MAX_COUNT=0 # libfabric disable caching
# or, less invasive:
export FI_MR_CACHE_MONITOR=memhooks # alternative cache monitor
# note (9-2-22, OLCFDEV-1079)
# this environment setting is needed to avoid that rocFFT writes a cache in
# the home directory, which does not scale.
export ROCFFT_RTC_CACHE_PATH=/dev/null
export OMP_NUM_THREADS=1
# LUMI documentation suggests using the following wrapper script
# to set the ROCR_VISIBLE_DEVICES to the value of SLURM_LOCALID
# see https://docs.lumi-supercomputer.eu/runjobs/scheduled-jobs/lumig-job/
cat << EOF > select_gpu
#!/bin/bash
export ROCR_VISIBLE_DEVICES=\$SLURM_LOCALID
exec \$*
EOF
chmod +x ./select_gpu
sleep 1
# LUMI documentation suggests using the following CPU bind
# so that the node local rank and GPU ID match
# see https://docs.lumi-supercomputer.eu/runjobs/scheduled-jobs/lumig-job/
CPU_BIND="map_cpu:48,56,16,24,1,8,32,40"
srun --cpu-bind=${CPU_BIND} ./select_gpu ./warpx inputs | tee outputs.txt
rm -rf ./select_gpu
Post-Processing
Note
TODO: Document any Jupyter or data services.
Known System Issues
Warning
December 12th, 2022: There is a caching bug in libFabric that causes WarpX simulations to occasionally hang on LUMI on more than 1 node.
As a work-around, please export the following environment variable in your job scripts until the issue is fixed:
#export FI_MR_CACHE_MAX_COUNT=0 # libfabric disable caching
# or, less invasive:
export FI_MR_CACHE_MONITOR=memhooks # alternative cache monitor
Warning
January, 2023: We discovered a regression in AMD ROCm, leading to 2x slower current deposition (and other slowdowns) in ROCm 5.3 and 5.4. Reported to AMD and fixed for the 5.5 release of ROCm.
Upgrade ROCm or stay with the ROCm 5.2 module to avoid.
Warning
May 2023: rocFFT in ROCm 5.1+ tries to write to a cache in the home area by default. This does not scale, disable it via:
export ROCFFT_RTC_CACHE_PATH=/dev/null