Lassen (LLNL)¶

The Lassen V100 GPU cluster is located at LLNL.

If you are new to this system, please see the following resources:

LLNL user account
Lassen user guide
Batch system: LSF
Production directories:
- /p/gpfs1/$(whoami): personal directory on the parallel filesystem
- Note that the $HOME directory and the /usr/workspace/$(whoami) space are NFS mounted and not suitable for production quality data generation.

Installation¶

Use the following commands to download the WarpX source code and switch to the correct branch:

git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx

We use the following modules and environments on the system.

# please set your project account
export proj=<yourProject>

# required dependencies
module load cmake/3.16.8
module load gcc/8.3.1
module load cuda/11.0.2

# optional: for PSATD support
module load fftw/3.3.8

# optional: for QED support
module load boost/1.70.0

# optional: for openPMD support
# TODO ADIOS2 & HDF5

# optional: for PSATD in RZ geometry support
# TODO: blaspp lapackpp

# optional: for Python bindings
module load python/3.8.2

# optional: an alias to request an interactive node for two hours
alias getNode="bsub -G $proj -W 2:00 -nnodes 1 -Is /bin/bash"

# fix system defaults: do not escape $ with a \ on tab completion
shopt -s direxpand

# compiler environment hints
export CC=$(which gcc)
export CXX=$(which g++)
export FC=$(which gfortran)
export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=$(which g++)

We recommend to store the above lines in a file, such as $HOME/lassen_warpx.profile, and load it into your shell after a login:

source $HOME/lassen_warpx.profile

Then, cd into the directory $HOME/src/warpx and use the following commands to compile:

rm -rf build/
cmake -B build -DWarpX_COMPUTE=CUDA -DWarpX_OPENPMD=ON
cmake --build build -j 10

This will build an executable in build/bin/. The other general compile-time options apply as usual.

Running¶

Please see our example job scripts on how to run WarpX on Lassen.

See Visualization with yt (for plotfiles) for more information on how to visualize the simulation results.