Tuolumne (LLNL)
The Tuolumne AMD GPU cluster (short name tuo) is located at LLNL. Tuolumne is an unclassified sibling system of El Capitan, sharing the same architecture.
El Capitan & Tuolumne provide four AMD MI300A APUs per compute node.
Introduction
If you are new to this system, please see the following resources:
LLNL user account (login required)
Batch system: Flux with Slurm Wrappers
Jupyter service (documentation, login required)
Production directories (login required):
/p/lustre5/${USER}: personal directory on the parallel filesystem (also:lustre2)Note that the
$HOMEdirectory and the/usr/workspace/${USER}space are NFS mounted and not suitable for production quality data generation.
Login
ssh tuolumne.llnl.gov
Preparation
Use the following commands to download the WarpX source code:
git clone https://github.com/BLAST-WarpX/warpx.git /p/lustre5/${USER}/tuolumne/src/warpx
We use system software modules, add environment hints and further dependencies via the file $HOME/tuolumne_mi300a_warpx.profile.
Create it now:
cp /p/lustre5/${USER}/tuolumne/src/warpx/Tools/machines/tuolumne-llnl/tuolumne_mi300a_warpx.profile.example $HOME/tuolumne_mi300a_warpx.profile
Edit the 2nd line of this script, which sets the export proj="" variable.
Currently, this is unused and can be kept empty.
Once project allocation becomes required, e.g., if you are member of the project abcde, then run vi $HOME/tuolumne_mi300a_warpx.profile.
Enter the edit mode by typing i and edit line 2 to read:
export proj="abcde"
Exit the vi editor with Esc and then type :wq (write & quit).
Important
Now, and as the first step on future logins to Tuolumne, activate these environment settings:
source $HOME/tuolumne_mi300a_warpx.profile
Finally, since Tuolumne does not yet provide software modules for some of our dependencies, install them once:
bash /p/lustre5/${USER}/tuolumne/src/warpx/Tools/machines/tuolumne-llnl/install_mi300a_dependencies.sh source /p/lustre5/${USER}/tuolumne/warpx/mi300a/venvs/warpx-tuolumne-mi300a/bin/activate
Compilation
Use the following cmake commands to compile the application executable:
cd /p/lustre5/${USER}/tuolumne/src/warpx
cmake --fresh -S . -B build_tuolumne -DWarpX_COMPUTE=HIP -DWarpX_FFT=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_tuolumne -j 24
The WarpX application executables are now in /p/lustre5/${USER}/tuolumne/src/warpx/build_tuolumne/bin/.
Additionally, the following commands will install WarpX as a Python module:
cmake --fresh -S . -B build_tuolumne_py -DWarpX_COMPUTE=HIP -DWarpX_FFT=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_tuolumne_py -j 24 --target pip_install
Now, you can submit tuolumne compute jobs for WarpX Python (PICMI) scripts (example scripts).
Or, you can use the WarpX executables to submit tuolumne jobs (example inputs).
For executables, you can reference their location in your job script or copy them to a location in $PROJWORK/$proj/.
Update WarpX & Dependencies
If you already installed WarpX in the past and want to update it, start by getting the latest source code:
cd /p/lustre5/${USER}/tuolumne/src/warpx
# read the output of this command - does it look ok?
git status
# get the latest WarpX source code
git fetch
git pull
# read the output of these commands - do they look ok?
git status
git log # press q to exit
And, if needed,
log out and into the system, activate the now updated environment profile as usual,
As a last step rebuild WarpX.
Running
MI300A APUs (128GB)
Each compute node is divided into 4 sockets, each with:
1 MI300A GPU,
21 available user CPU cores, with 3 cores reserved for the OS (2 hardware threads per core)
128GB HBM3 memory (a single NUMA domain)
The batch script below can be used to run a WarpX simulation on 1 node with 4 APUs on the supercomputer Tuolumne at LLNL.
Replace descriptions between chevrons <> by relevant values, for instance <input file> could be plasma_mirror_inputs.
WarpX runs with one MPI rank per GPU.
Note that we append these non-default runtime options:
amrex.use_gpu_aware_mpi=1: make use of fast APU to APU MPI communications
#!/bin/bash -l
# Copyright 2024 The WarpX Community
#
# This file is part of WarpX.
#
# Authors: Axel Huebl, Joshua David Ludwig
# License: BSD-3-Clause-LBNL
#SBATCH -t 00:30:00
#SBATCH -N 1
#SBATCH -J WarpX
#S BATCH -A <proj> # project name not needed yet
#SBATCH -o WarpX.o%j
#SBATCH -e WarpX.e%j
# Transparent huge pages on CPU
# https://hpc.llnl.gov/documentation/user-guides/using-el-capitan-systems/introduction-and-quickstart/pro-tips
#S BATCH --thp=always
if [[ -z "${MY_PROFILE}" ]]; then
echo "WARNING: FORGOT TO"
echo " source $HOME/tuolumne_mi300a_warpx.profile"
echo "before submission. Doing that now."
source $HOME/tuolumne_mi300a_warpx.profile
fi
# executable & inputs file or python interpreter & PICMI script here
EXE=./warpx
INPUTS=inputs
# pin to closest NIC to GPU
export MPICH_OFI_NIC_POLICY=GPU
# Transparent huge pages on CPU
# https://hpc.llnl.gov/documentation/user-guides/using-el-capitan-systems/introduction-and-quickstart/pro-tips
#export HSA_XNACK=1
#export HUGETLB_MORECORE=yes
# threads for OpenMP and threaded compressors per MPI rank
# note: 16 avoids hyperthreading (32 virtual cores, 16 physical)
export OMP_NUM_THREADS=16
# GPU-aware MPI optimizations
GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1"
# MPI parallel processes
NNODES=$(flux resource list -s up -no {nnodes})
srun -N ${NNODES} \
--ntasks-per-node=4 \
${EXE} ${INPUTS} \
${GPU_AWARE_MPI} \
> output.txt
To run a simulation, copy the lines above to a file tuolumne_mi300a.sbatch and run
sbatch tuolumne_mi300a.sbatch
to submit the job.