Gkeyll
The Gkeyll simulation framework is a modular, multi-scale, multi-physics code base developed for simulations in plasma physics, space physics, general relativity, and high-energy astrophysics. The core is written in C with a lightweight design and minimal external dependencies, complemented by a Lua scripting layer (via the Lua–C API) for flexible specification of simulation inputs. The framework is organized hierarchically, with shared infrastructure for grids, parallelism, I/O, and common numerical operations, and a suite of physics solvers built on top. These include finite-volume solvers for hyperbolic PDE systems (e.g. relativistic multi-fluid dynamics, Maxwell and Einstein equations), modal discontinuous Galerkin solvers for continuum kinetic models such as Vlasov–Maxwell, Vlasov–Poisson, gyrokinetics, and parallel-kinetic–perpendicular-moment systems, and supporting tools for ODE integration, Poisson solvers, and curved-space geometries relevant to astrophysical and fusion applications.
Gkeyll Quick-Start on Discovery¶
Spack in the EPaCO lab environment
We first configure and activate the Spack environment, a very helpful tool for using Gkeyll in the EPaCO lab environment. To ensure Spack uses the correct configuration and cache directories, add the following lines to your ~/.bashrc file:
export SPACK_USER_CONFIG_PATH=/dartfs/rc/lab/E/EPaCO/libs/spack-user-config
export SPACK_USER_CACHE_PATH=/dartfs/rc/lab/E/EPaCO/libs/spack-user-cache
~/.bashrc, apply the changes:
The above steps are for first-time users. After setting up this environment, you can initialize Spack by sourcing the system Spack setup script:
Activate the preconfigured EPaCO Spack environment:
After activating the environment, the cluster is configured for building and running Gkeyll applications. You will need to run
If you wish to build your own version of Gkeyll on Discovery, follow the steps below. For additional details and advanced options, refer to the Gkeyll install documentation.
Pre-built Gkeyll Installation
A pre-built version of the Gkeyll Vlasov solver is available in the EPaCO lab space:
The pre-built Gkeyll executable provided in the EPaCO lab space is compiled with MPI support using the Spack environment described above. In particular, it relies on the MPICH. Before running the pre-builtgkeyll executable in parallel, load the MPI environment via:
From the gkeyll/install-deps/ directory, build the required dependencies. The prefix shown below is an example; you may choose a different installation prefix if desired.
./mkdeps.sh \
--build-openblas=yes \
--build-superlu=yes \
--build-luajit=yes \
--prefix=/dartfs/rc/lab/E/EPaCO/libs/gkylsoft
gkeyll/ directory, configure the build. In this example, only the Vlasov solver is enabled, with Lua and MPI support. You can use a different MPI implementation.
./configure \
--app=vlasov \
CC=/dartfs/rc/lab/E/EPaCO/libs/spack/opt/spack/linux-zen2/mpich-4.3.2-r3peu7d2xpco4n3aqaagzcdru33qe67u/bin/mpicc \
--use-mpi=yes \
--mpi-inc=/dartfs/rc/lab/E/EPaCO/libs/spack/opt/spack/linux-zen2/mpich-4.3.2-r3peu7d2xpco4n3aqaagzcdru33qe67u/include \
--mpi-lib=/dartfs/rc/lab/E/EPaCO/libs/spack/opt/spack/linux-zen2/mpich-4.3.2-r3peu7d2xpco4n3aqaagzcdru33qe67u/lib \
--prefix=/dartfs-hpc/rc/home/7/f007gj7/epaco/libs/gkylsoft \
--use-lua=yes
After this step completed, you can compile and install Gkeyll using:
Discovery Cluster
Compilation of entity on Discovery's login node can be slow at times. If you encounter long compilation times, consider building the code in an interactive session, requested for example like this:
After installation, the Gkeyll executable will be available under the specified prefix. You can find a simple example simulation with the Vlasov module in the Gkeyll documentation. In an interactive session, running the Gkeyll example would require, for instance:
Tip
Post-processing of Gkeyll output is provided via PostGkeyll, the respective python packages are available if you use the EPaCO python kernel entity (see also Jupyter Notebook introduction).