Quartz (LLNL)
The Quartz Intel CPU cluster is located at LLNL.
Introduction
If you are new to this system, please see the following resources:
LLNL user account (login required)
Batch system: Slurm
Jupyter service (documentation, login required)
-
/p/lustre1/$(whoami)
and/p/lustre2/$(whoami)
: personal directory on the parallel filesystemNote that the
$HOME
directory and the/usr/workspace/$(whoami)
space are NFS mounted and not suitable for production quality data generation.
Preparation
Use the following commands to download the WarpX source code:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
We use system software modules, add environment hints and further dependencies via the file $HOME/quartz_warpx.profile
.
Create it now:
cp $HOME/src/warpx/Tools/machines/quartz-llnl/quartz/quartz_warpx.profile.example $HOME/quartz_warpx.profile
Edit the 2nd line of this script, which sets the export proj=""
variable.
For example, if you are member of the project tps
, then run vi $HOME/quartz_warpx.profile
.
Enter the edit mode by typing i
and edit line 2 to read:
export proj="tps"
Exit the vi
editor with Esc
and then type :wq
(write & quit).
Important
Now, and as the first step on future logins to Quartz, activate these environment settings:
source $HOME/quartz_warpx.profile
Finally, since Quartz does not yet provide software modules for some of our dependencies, install them once:
bash $HOME/src/warpx/Tools/machines/quartz-llnl/install_dependencies.sh
source /usr/workspace/${USER}/quartz/venvs/warpx-quartz/bin/activate
Compilation
Use the following cmake commands to compile the application executable:
cd $HOME/src/warpx
rm -rf build_quartz
cmake -S . -B build_quartz -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_quartz -j 6
The WarpX application executables are now in $HOME/src/warpx/build_quartz/bin/
.
Additionally, the following commands will install WarpX as a Python module:
rm -rf build_quartz_py
cmake -S . -B build_quartz_py -DWarpX_PSATD=ON -DWarpX_QED_TABLE_GEN=ON -DWarpX_APP=OFF -DWarpX_PYTHON=ON -DWarpX_DIMS="1;2;RZ;3"
cmake --build build_quartz_py -j 6 --target pip_install
Now, you can submit Quartz compute jobs for WarpX Python (PICMI) scripts (example scripts).
Or, you can use the WarpX executables to submit Quartz jobs (example inputs).
For executables, you can reference their location in your job script or copy them to a location in $PROJWORK/$proj/
.
Update WarpX & Dependencies
If you already installed WarpX in the past and want to update it, start by getting the latest source code:
cd $HOME/src/warpx
# read the output of this command - does it look ok?
git status
# get the latest WarpX source code
git fetch
git pull
# read the output of these commands - do they look ok?
git status
git log # press q to exit
And, if needed,
log out and into the system, activate the now updated environment profile as usual,
As a last step, clean the build directory rm -rf $HOME/src/warpx/build_quartz
and rebuild WarpX.
Running
Intel Xeon E5-2695 v4 CPUs
The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Quartz at LLNL.
Replace descriptions between chevrons <>
by relevant values, for instance <input file>
could be plasma_mirror_inputs
.
#!/bin/bash -l
# Just increase this number of you need more nodes.
#SBATCH -N 2
#SBATCH -t 24:00:00
#SBATCH -A <allocation ID>
#SBATCH -J WarpX
#SBATCH -q pbatch
#SBATCH --qos=normal
#SBATCH --license=lustre1,lustre2
#SBATCH --export=ALL
#SBATCH -e error.txt
#SBATCH -o output.txt
# one MPI rank per half-socket (see below)
#SBATCH --tasks-per-node=2
# request all logical (virtual) cores per half-socket
#SBATCH --cpus-per-task=18
# each Quartz node has 1 socket of Intel Xeon E5-2695 v4
# each Xeon CPU is divided into 2 bus rings that each have direct L3 access
export WARPX_NMPI_PER_NODE=2
# each MPI rank per half-socket has 9 physical cores
# or 18 logical (virtual) cores
# over-subscribing each physical core with 2x
# hyperthreading led to a slight (3.5%) speedup on Cori's Intel Xeon E5-2698 v3,
# so we do the same here
# the settings below make sure threads are close to the
# controlling MPI rank (process) per half socket and
# distribute equally over close-by physical cores and,
# for N>9, also equally over close-by logical cores
export OMP_PROC_BIND=spread
export OMP_PLACES=threads
export OMP_NUM_THREADS=18
EXE="<path/to/executable>" # e.g. ./warpx
srun --cpu_bind=cores -n $(( ${SLURM_JOB_NUM_NODES} * ${WARPX_NMPI_PER_NODE} )) ${EXE} <input file>
To run a simulation, copy the lines above to a file quartz.sbatch
and run
sbatch quartz.sbatch
to submit the job.