Polaris (ALCF)
The Polaris cluster is located at ALCF.
Introduction
If you are new to this system, please see the following resources:
Batch system: PBS
Preparation
Use the following commands to download the WarpX source code:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
On Polaris, you can run either on GPU nodes with fast A100 GPUs (recommended) or CPU nodes.
We use system software modules, add environment hints and further dependencies via the file $HOME/polaris_gpu_warpx.profile.
Create it now:
cp $HOME/src/warpx/Tools/machines/polaris-alcf/polaris_gpu_warpx.profile.example $HOME/polaris_gpu_warpx.profile
Edit the 2nd line of this script, which sets the export proj="" variable.
For example, if you are member of the project proj_name, then run nano $HOME/polaris_gpu_warpx.profile and edit line 2 to read:
export proj="proj_name"
Exit the nano editor with Ctrl + O (save) and then Ctrl + X (exit).
Important
Now, and as the first step on future logins to Polaris, activate these environment settings:
source $HOME/polaris_gpu_warpx.profile
Finally, since Polaris does not yet provide software modules for some of our dependencies, install them once:
bash $HOME/src/warpx/Tools/machines/polaris-alcf/install_gpu_dependencies.sh
source ${CFS}/${proj%_g}/${USER}/sw/polaris/gpu/venvs/warpx/bin/activate
Under construction
Compilation
Use the following cmake commands to compile the application executable:
cd $HOME/src/warpx
rm -rf build_pm_gpu
cmake -S . -B build_pm_gpu -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_pm_gpu -j 16
The WarpX application executables are now in $HOME/src/warpx/build_pm_gpu/bin/.
Additionally, the following commands will install WarpX as a Python module:
cd $HOME/src/warpx
rm -rf build_pm_gpu_py
cmake -S . -B build_pm_gpu_py -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_pm_gpu_py -j 16 --target pip_install
Under construction
Now, you can submit Polaris compute jobs for WarpX Python (PICMI) scripts (example scripts).
Or, you can use the WarpX executables to submit Polaris jobs (example inputs).
For executables, you can reference their location in your job script or copy them to a location in $PSCRATCH.
Update WarpX & Dependencies
If you already installed WarpX in the past and want to update it, start by getting the latest source code:
cd $HOME/src/warpx
# read the output of this command - does it look ok?
git status
# get the latest WarpX source code
git fetch
git pull
# read the output of these commands - do they look ok?
git status
git log # press q to exit
And, if needed,
update the polaris_gpu_warpx.profile or polaris_cpu_warpx files,
log out and into the system, activate the now updated environment profile as usual,
As a last step, clean the build directory rm -rf $HOME/src/warpx/build_pm_* and rebuild WarpX.
Running
The batch script below can be used to run a WarpX simulation on multiple nodes (change <NODES> accordingly) on the supercomputer Polaris at ALCF.
Replace descriptions between chevrons <> by relevant values, for instance <input file> could be plasma_mirror_inputs.
Note that we run one MPI rank per GPU.
$HOME/src/warpx/Tools/machines/polaris-alcf/polaris_gpu.pbs.#!/bin/bash -l
#PBS -A <proj>
#PBS -l select=<NODES>:system=polaris
#PBS -l place=scatter
#PBS -l walltime=0:10:00
#PBS -l filesystems=home:eagle
#PBS -q debug
#PBS -N test_warpx
# Set required environment variables
# support gpu-aware-mpi
# export MPICH_GPU_SUPPORT_ENABLED=1
# Change to working directory
echo Working directory is $PBS_O_WORKDIR
cd ${PBS_O_WORKDIR}
echo Jobid: $PBS_JOBID
echo Running on host `hostname`
echo Running on nodes `cat $PBS_NODEFILE`
# executable & inputs file or python interpreter & PICMI script here
EXE=./warpx
INPUTS=input1d
# MPI and OpenMP settings
NNODES=`wc -l < $PBS_NODEFILE`
NRANKS_PER_NODE=4
NDEPTH=1
NTHREADS=1
NTOTRANKS=$(( NNODES * NRANKS_PER_NODE ))
echo "NUM_OF_NODES= ${NNODES} TOTAL_NUM_RANKS= ${NTOTRANKS} RANKS_PER_NODE= ${NRANKS_PER_NODE} THREADS_PER_RANK= ${NTHREADS}"
mpiexec -np ${NTOTRANKS} ${EXE} ${INPUTS} > output.txt
To run a simulation, copy the lines above to a file polaris_gpu.pbs and run
qsub polaris_gpu.pbs
to submit the job.
Under construction