Quartz (LLNL)¶
The Quartz Intel CPU cluster is located at LLNL.
If you are new to this system, please see the following resources:
Batch system: Slurm
-
/p/lustre1/$(whoami)
and/p/lustre2/$(whoami)
: personal directory on the parallel filesystemNote that the
$HOME
directory and the/usr/workspace/$(whoami)
space are NFS mounted and not suitable for production quality data generation.
Installation¶
Use the following commands to download the WarpX source code and switch to the correct branch:
git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx
We use the following modules and environments on the system ($HOME/warpx.profile
).
# please set your project account
export proj=<yourProject>
# required dependencies
module load cmake/3.20.2
module load intel/19.1.2
module load mvapich2/2.3
# optional: for PSATD support
module load fftw/3.3.8
# optional: for QED lookup table generation support
module load boost/1.73.0
# optional: for openPMD support
# TODO ADIOS2
module load hdf5-parallel/1.10.2
# optional: for PSATD in RZ geometry support
# TODO: blaspp lapackpp
# optional: for Python bindings
module load python/3.8.2
# optional: an alias to request an interactive node for two hours
alias getNode="srun --time=0:30:00 --nodes=1 --ntasks-per-node=2 --cpus-per-task=18 -p pdebug --pty bash"
# fix system defaults: do not escape $ with a \ on tab completion
shopt -s direxpand
# compiler environment hints
export CC=$(which icc)
export CXX=$(which icpc)
export FC=$(which ifort)
# we need a newer libstdc++:
export CFLAGS="-gcc-name=/usr/tce/packages/gcc/gcc-8.3.1/bin/gcc ${CFLAGS}"
export CXXFLAGS="-gxx-name=/usr/tce/packages/gcc/gcc-8.3.1/bin/g++ ${CXXFLAGS}"
We recommend to store the above lines in a file, such as $HOME/quartz_warpx.profile
, and load it into your shell after a login:
source $HOME/quartz_warpx.profile
Then, cd
into the directory $HOME/src/warpx
and use the following commands to compile:
cd $HOME/src/warpx
rm -rf build
cmake -S . -B build -DWarpX_OPENPMD=ON
cmake --build build -j 6
This will build an executable in build/bin/
.
The other general compile-time options apply as usual.
Running¶
Intel Xeon E5-2695 v4 CPUs¶
The batch script below can be used to run a WarpX simulation on 2 nodes on the supercomputer Quartz at LLNL.
Replace descriptions between chevrons <>
by relevant values, for instance <input file>
could be plasma_mirror_inputs
.
#!/bin/bash -l
# Just increase this number of you need more nodes.
#SBATCH -N 2
#SBATCH -t 24:00:00
#SBATCH -A <allocation ID>
#SBATCH -J WarpX
#SBATCH -q pbatch
#SBATCH --qos=normal
#SBATCH --license=lustre1,lustre2
#SBATCH --export=ALL
#SBATCH -e error.txt
#SBATCH -o output.txt
# one MPI rank per half-socket (see below)
#SBATCH --tasks-per-node=2
# request all logical (virtual) cores per half-socket
#SBATCH --cpus-per-task=18
# each Quartz node has 1 socket of Intel Xeon E5-2695 v4
# each Xeon CPU is divided into 2 bus rings that each have direct L3 access
export WARPX_NMPI_PER_NODE=2
# each MPI rank per half-socket has 9 physical cores
# or 18 logical (virtual) cores
# over-subscribing each physical core with 2x
# hyperthreading led to a slight (3.5%) speedup on Cori's Intel Xeon E5-2698 v3,
# so we do the same here
# the settings below make sure threads are close to the
# controlling MPI rank (process) per half socket and
# distribute equally over close-by physical cores and,
# for N>9, also equally over close-by logical cores
export OMP_PROC_BIND=spread
export OMP_PLACES=threads
export OMP_NUM_THREADS=18
EXE="<path/to/executable>" # e.g. ./warpx
srun --cpu_bind=cores -n $(( ${SLURM_JOB_NUM_NODES} * ${WARPX_NMPI_PER_NODE} )) ${EXE} <input file>
To run a simulation, copy the lines above to a file batch_quartz.sh
and run
sbatch batch_quartz.sh
to submit the job.