HyPar Documentation

Author

Debojyoti Ghosh [Email: (first name) (dot) (last name) (at) gmail (dot) com, Website: http://debog.github.io/]

John Loffeld

Youngdae Kim

HyPar: Hyperbolic-Parabolic (with Source) Partial Differential Equations Solver

HyPar is a finite-difference algorithm to solve hyperbolic-parabolic partial differential equations (with source terms) on Cartesian grids. It is a unified framework that can handle systems of PDEs with arbitrary number of spatial dimensions and solution components. It provides the spatial discretization and time integration functions, functions to read and write solutions from/to files, as well as functions required to solve the system on parallel (MPI) platforms. The physical models define the physics-specific functions such as the exact forms of the hyperbolic flux, parabolic flux, source terms, upwinding functions, etc.

Features

• Solves hyperbolic-parabolic PDEs with source terms.
• Allows arbitrary number of spatial dimensions and vector components per grid point.
• Solves the PDEs over Cartesian grids.
• Can use sparse grids for faster computations on high-dimensional problems
• Written entirely in C/C++ and uses the MPICH library. It can use OpenMP threads and CUDA on NVidia GPUs, but these are works-in-progress.
• Can be compiled with PETSc (https://petsc.org/release/), if available, where it can use PETSc's time integration module TS (https://petsc.org/release/src/ts/).
• For 3-dimensional simulations, the immersed boundaries can be used to solve over non-Cartesian geometries.
• Can be compiled with libROM (https://www.librom.net/), if available, where it can be used with the reduced order modeling tools implemented in libROM.
• Can be compiled with Python (https://www.python.org/), if available, to enable features like in-situ plotting of solution.

HyPar has been developed to be scalable, and apart from the usual functionalities to solve a system of PDEs on distributed architectures, it provides scalable file I/O functions. It has been tested on several platforms, including DOE Leadership-class supercomputers, with up to ~0.5 million MPI ranks.

Download

The code is available at:

• https://github.com/debog/hypar
• https://gitlab.com/debojyoti.ghosh/hypar

Follow the instructions on these pages to clone using git. For example,

• git clone git@g.nosp@m.itla.nosp@m.b.com:debojyoti.ghosh/hypar.git
• git clone https://github.com/debog/hypar.git

Both these platforms allow downloading the package as a zip/tarball, for example

• https://github.com/debog/hypar/archive/refs/heads/master.zip
• https://gitlab.com/debojyoti.ghosh/hypar/-/archive/master/hypar-master.zip

Documentation

To generate a local copy of this documentation, run "doxygen Doxyfile" in . The folder /doc should contain the generated documentation in HTML format.

Compiling

To compile HyPar, follow these steps in the root directory:

  autoreconf -i
  [CFLAGS="..."] [CXXFLAGS="..."] ./configure [options]
  make
  make install

CFLAGS and CXXFLAGS should include all the compiler flags.

Note: Default installation target is its own directory, and thus "make install" should not require administrative privileges. The binary will be placed in bin/ subdirectory.

The configure options can include options such as BLAS/LAPACK location, MPI directory, etc. Type "./configure --help" to see a full list. The options specific to HyPar are:

• --enable-serial: Compile a serial version without MPI.
• --with-mpi-dir: Specify path where mpicc is installed, if not in standard path.
• --enable-omp: Enable OpenMP threads.
• --enable-cuda: Enable CUDA if NVidia GPU present.
• --enable-python: Enable Python interface.
• --with-cuda-dir: Specify path where CUDA is installed, if not in standard path.
• --enable-scalapack: Enable ScaLAPACK (this will make available a tridiagonal solver using ScaLAPACK).
• --enable-fftw: Enable FFTW (this will make available features that use the FFTW library; needs MPI).
• --with-blas-dir: Specify path where BLAS is installed (relevant only if --enable-scalapack is specified).
• --with-lapack-dir: Specify path where LAPACK is installed (relevant only if --enable-scalapack is specified).
• --with-scalapack-dir: Specify path where ScaLAPACK is installed (relevant only if --enable-scalapack is specified).
• --with-fftw-dir: Specify path where FFTW is installed (relevant only if --enable-fftw is specified). Note that the FFTW library should be compiled with MPI.
• --with-fortran-lib: Specify path where FORTRAN libraries are installed (for ScaLAPACK) (relevant only if --enable-scalapack is specified).

Notes:

• Limited parts of the code have been implemented on CUDA. See Examples for currently available CUDA-enabled simulations.
• Interfacing with Python and the features it enables are a work-in-progress. It is not very robust; for example it may result in inexplicable segmentation faults depending on the Python version and the MPI library. It should work okay when compiled with OpenMPI and when compiled without any MPI (serial). It is based on the approach in PythonFOAM (https://github.com/argonne-lcf/PythonFOAM). The following environment variables must be consistently defined:
  • PYTHON_LIB_PATH - location of Python libraries
  • PYTHON_BIN_PATH - location of Python binary
  • PYTHON_INCLUDE_PATH - location of Python headers
  • NUMPY_INCLUDE_PATH - location of numpy headers (for eg., /path/to/python3.8/dist-packages/numpy/core/include)
  • PYTHON_LIB_NAME - name of the Python library (for eg., lpython3.8)

Compiling with other scientific computing libraries

HyPar can be compiled with and use the functionalities of the following libraries. It is assumed that the user is familiar with them and has access to their documentation.

Compiling with PETSc: Install PETSc and make sure the environment variables PETSC_DIR and PETSC_ARCH are defined. Please see PETSc's installation instructions for this. Once these environment variables are present, HyPar will use them to compile itself with PETSc functionalities.

Compiling with libROM: Download and compile libROM and make sure the environment variable LIBROM_DIR is defined as the location of libROM source directory. Once this environment variable is present, HyPar will use it to compile itself with libROM functionalities.

Notes

• This package has been tested using the GNU C and C++ compilers. The configuration script is designed to look for these compilers only.
• Feel free to contact me about anything regarding this (doubts/difficulties/suggestions).
• Feel free to use and modify the code in any way.

Running

• It's best to start with some examples. See the section on examples.
• To run more cases, see the section in input files for a complete description of input files required.

Testing and Baselines

• A set of baselines are available at: https://gitlab.com/debojyoti.ghosh/hypar_baselines These were obtained using the master branch of the main HyPar repo.
• The script Extras/generate_baselines.sh can be used to clone this baselines repo and regenerate the solutions.
• The scripts Extras/test_local.sh and Extras/test_repo.sh can be used to test a local or remote copy (git repo and branch) of HyPar against these baselines.