Karolina (IT4I)
The Karolina cluster is located at IT4I, Technical University of Ostrava.
Introduction
If you are new to this system, please see the following resources:
Batch system: PBS
Jupyter service: not provided/documented (yet)
-
$HOME: per-user directory, use only for inputs, source and scripts; backed up (25GB default quota)/scatch/: production directory; very fast for parallel jobs (20TB default)
Preparation
Use the following commands to download the WarpX source code:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
On Karolina, you can run either on GPU nodes with fast A100 GPUs (recommended) or CPU nodes.
We use system software modules, add environment hints and further dependencies via the file $HOME/karolina_gpu_warpx.profile.
Create it now:
cp $HOME/src/warpx/Tools/machines/karolina-it4i/karolina_gpu_warpx.profile.example $HOME/karolina_gpu_warpx.profile
Edit the 2nd line of this script, which sets the export proj="" variable.
For example, if you are member of the project DD-23-83, then run vi $HOME/karolina_gpu_warpx.profile.
Enter the edit mode by typing i and edit line 2 to read:
export proj="DD-23-83"
Exit the vi editor with Esc and then type :wq (write & quit).
Important
Now, and as the first step on future logins to Karolina, activate these environment settings:
source $HOME/karolina_gpu_warpx.profile
Finally, since Karolina does not yet provide software modules for some of our dependencies, install them once:
bash $HOME/src/warpx/Tools/machines/karolina-it4i/install_gpu_dependencies.sh
source $HOME/sw/karolina/gpu/venvs/warpx-gpu/bin/activate
CPU usage is documentation is TODO.
Compilation
Use the following cmake commands to compile the application executable:
cd $HOME/src/warpx
rm -rf build_gpu
cmake -S . -B build_gpu -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_gpu -j 12
The WarpX application executables are now in $HOME/src/warpx/build_gpu/bin/.
Additionally, the following commands will install WarpX as a Python module:
rm -rf build_gpu_py
cmake -S . -B build_gpu_py -DWarpX_COMPUTE=CUDA -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_gpu_py -j 12 --target pip_install
cd $HOME/src/warpx
rm -rf build_cpu
cmake -S . -B build_cpu -DWarpX_COMPUTE=OMP -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_cpu -j 12
The WarpX application executables are now in $HOME/src/warpx/build_cpu/bin/.
Additionally, the following commands will install WarpX as a Python module:
cd $HOME/src/warpx
rm -rf build_cpu_py
cmake -S . -B build_cpu_py -DWarpX_COMPUTE=OMP -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_cpu_py -j 12 --target pip_install
Now, you can submit Karolina compute jobs for WarpX Python (PICMI) scripts (example scripts).
Or, you can use the WarpX executables to submit Karolina jobs (example inputs).
For executables, you can reference their location in your job script or copy them to a location in /scatch/.
Update WarpX & Dependencies
If you already installed WarpX in the past and want to update it, start by getting the latest source code:
cd $HOME/src/warpx
# read the output of this command - does it look ok?
git status
# get the latest WarpX source code
git fetch
git pull
# read the output of these commands - do they look ok?
git status
git log # press q to exit
And, if needed,
update the karolina_gpu_warpx.profile or karolina_cpu_warpx.profile files,
log out and into the system, activate the now updated environment profile as usual,
As a last step, clean the build directory rm -rf $HOME/src/warpx/build_* and rebuild WarpX.
Running
The batch script below can be used to run a WarpX simulation on multiple GPU nodes (change #PBS -l select= accordingly) on the supercomputer Karolina at IT4I.
This partition as up to 72 nodes.
Every node has 8x A100 (40GB) GPUs and 2x AMD EPYC 7763, 64-core, 2.45 GHz processors.
Replace descriptions between chevrons <> by relevant values, for instance <proj> could be DD-23-83.
Note that we run one MPI rank per GPU.
$HOME/src/warpx/Tools/machines/karolina-it4i/karolina_gpu.qsub.#!/bin/bash -l
# Copyright 2023 The WarpX Community
#
# This file is part of WarpX.
#
# Authors: Axel Huebl, Andrei Berceanu
# License: BSD-3-Clause-LBNL
#PBS -q qgpu
#PBS -N WarpX
# Use two full nodes, 8 GPUs per node, 16 GPUs total
#PBS -l select=2:ncpus=128:ngpus=8:mpiprocs=8:ompthreads=16,walltime=00:10:00
#PBS -A <proj>
cd ${PBS_O_WORKDIR}
# executable & inputs file or python interpreter & PICMI script here
EXE=./warpx.rz
INPUTS=inputs_rz
# OpenMP threads per MPI rank
export OMP_NUM_THREADS=16
# run
mpirun -np ${PBS_NP} bash -c "
export CUDA_VISIBLE_DEVICES=\${OMPI_COMM_WORLD_LOCAL_RANK};
${EXE} ${INPUTS}" \
> output.txt
To run a simulation, copy the lines above to a file karolina_gpu.qsub and run
qsub karolina_gpu.qsub
to submit the job.
CPU usage is documentation is TODO.
Post-Processing
Note
This section was not yet written. Usually, we document here how to use a Jupyter service.