Great Lakes (UMich)
The Great Lakes cluster is located at University of Michigan. The cluster has various partitions, including GPU nodes and CPU nodes.
Introduction
If you are new to this system, please see the following resources:
Batch system: Slurm
-
$HOME: per-user directory, use only for inputs, source and scripts; backed up (80GB)/scratch: per-project production directory; very fast for parallel jobs; purged every 60 days (10TB default)
Preparation
Use the following commands to download the WarpX source code:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
On Great Lakes, you can run either on GPU nodes with fast V100 GPUs (recommended), the even faster A100 GPUs (only a few available) or CPU nodes.
We use system software modules, add environment hints and further dependencies via the file $HOME/greatlakes_v100_warpx.profile.
Create it now:
cp $HOME/src/warpx/Tools/machines/greatlakes-umich/greatlakes_v100_warpx.profile.example $HOME/greatlakes_v100_warpx.profile
Edit the 2nd line of this script, which sets the export proj="" variable.
For example, if you are member of the project iloveplasma, then run nano $HOME/greatlakes_v100_warpx.profile and edit line 2 to read:
export proj="iloveplasma"
Exit the nano editor with Ctrl + O (save) and then Ctrl + X (exit).
Important
Now, and as the first step on future logins to Great Lakes, activate these environment settings:
source $HOME/greatlakes_v100_warpx.profile
Finally, since Great Lakes does not yet provide software modules for some of our dependencies, install them once:
bash $HOME/src/warpx/Tools/machines/greatlakes-umich/install_v100_dependencies.sh
source ${HOME}/sw/greatlakes/v100/venvs/warpx-v100/bin/activate
Note
This section is TODO.
Note
This section is TODO.
Compilation
Use the following cmake commands to compile the application executable:
cd $HOME/src/warpx
rm -rf build_v100
cmake -S . -B build_v100 -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_v100 -j 8
The WarpX application executables are now in $HOME/src/warpx/build_v100/bin/.
Additionally, the following commands will install WarpX as a Python module:
cd $HOME/src/warpx
rm -rf build_v100_py
cmake -S . -B build_v100_py -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_v100_py -j 8 --target pip_install
Note
This section is TODO.
Note
This section is TODO.
Now, you can submit Great Lakes compute jobs for WarpX Python (PICMI) scripts (example scripts).
Or, you can use the WarpX executables to submit greatlakes jobs (example inputs).
For executables, you can reference their location in your job script or copy them to a location in /scratch.
Update WarpX & Dependencies
If you already installed WarpX in the past and want to update it, start by getting the latest source code:
cd $HOME/src/warpx
# read the output of this command - does it look ok?
git status
# get the latest WarpX source code
git fetch
git pull
# read the output of these commands - do they look ok?
git status
git log # press q to exit
And, if needed,
log out and into the system, activate the now updated environment profile as usual,
As a last step, clean the build directory rm -rf $HOME/src/warpx/build_* and rebuild WarpX.
Running
The batch script below can be used to run a WarpX simulation on multiple nodes (change -N accordingly) on the supercomputer Great Lakes at University of Michigan.
This partition has 20 nodes, each with two V100 GPUs.
Replace descriptions between chevrons <> by relevant values, for instance <input file> could be plasma_mirror_inputs.
Note that we run one MPI rank per GPU.
$HOME/src/warpx/Tools/machines/greatlakes-umich/greatlakes_v100.sbatch.#!/bin/bash -l
# Copyright 2024 The WarpX Community
#
# Author: Axel Huebl
# License: BSD-3-Clause-LBNL
#SBATCH -t 00:10:00
#SBATCH -N 1
#SBATCH -J WarpX
#SBATCH -A <proj>
#SBATCH --partition=gpu
#SBATCH --exclusive
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=20
#SBATCH --gpus-per-task=v100:1
#SBATCH --gpu-bind=single:1
#SBATCH -o WarpX.o%j
#SBATCH -e WarpX.e%j
# executable & inputs file or python interpreter & PICMI script here
EXE=./warpx
INPUTS=inputs
# threads for OpenMP and threaded compressors per MPI rank
# per node are 2x 2.4 GHz Intel Xeon Gold 6148
# note: the system seems to only expose cores (20 per socket),
# not hyperthreads (40 per socket)
export SRUN_CPUS_PER_TASK=20
export OMP_NUM_THREADS=${SRUN_CPUS_PER_TASK}
# GPU-aware MPI optimizations
GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1"
# run WarpX
srun --cpu-bind=cores \
${EXE} ${INPUTS} ${GPU_AWARE_MPI} \
> output.txt
To run a simulation, copy the lines above to a file greatlakes_v100.sbatch and run
sbatch greatlakes_v100.sbatch
to submit the job.
This partition has 2 nodes, each with four A100 GPUs that provide 80 GB HBM per A100 GPU. To the user, each node will appear as if it has 8 A100 GPUs with 40 GB memory each.
Note
This section is TODO.
The Great Lakes CPU partition as up to 455 nodes, each with 2x Intel Xeon Gold 6154 CPUs and 180 GB RAM.
Note
This section is TODO.
Post-Processing
For post-processing, many users prefer to use the online Jupyter service (documentation) that is directly connected to the cluster’s fast filesystem.
Note
This section is a stub and contributions are welcome. We can document further details, e.g., which recommended post-processing Python software to install or how to customize Jupyter kernels here.