Starflood is an open-source astrophysical simulation code written in C.

A Starflood N-body simulation (N = 65536) of an expanding self-gravitating sphere, visualized using Blender Cycles

Features

• Parallelization via OpenMP
  • Device/GPU acceleration (when using a toolchain with offloading support)
• N-body gravitational simulation
  • Gravitational solver using O(N²) brute-force method a.k.a. particle-particle method
• Smoothed-particle hydrodynamics (SPH)
  • Hydrodynamics solver O(N²) brute-force method for neighbor search
• Leapfrog integration
  • is a second-order symplectic integrator
• Visualization
  • Accumulation rasterization using atomic operations
  • Spatial anti-aliasing (Gaussian window function stochastic sampling)
• Generation of Initial Conditions (ICs)
• File I/O
  • Simulation snapshots can be saved as files (in a raw binary format), body positions can be saved in the Standford Polygon File Format (.ply)
  • Visualization can be done during a run, or afterwards by loading snapshot files
    • The frame sequence is saved as a series of individual images, which can be encoded using tools such as ffmpeg
    • Floating-point images can be saved in the PFM graphic image file format (.pfm)

Planned (To-do List)

• Add support for distributed computing
  • Message Passing Interface (MPI)
• Add support for volume rendering to visualization
• Create physically accurate initial conditions
  • Andromeda-Milky Way collision
  • Molecular cloud/Stellar nursery
• Improve physical solvers
  • Barnes-Hut tree method
  • Fast Multipole Method (FMM)

References

• Create Your Own Smoothed-Particle Hydrodynamics Simulation (With Python) by Philip Mocz
  • Used as a reference implementation of SPH, which is nice since the Wikipedia article only contains mathematical formulae/equations!
• Dynamics and Astrophysics of Galaxies by Jo Bovy
  • Web-based book
• GADGET-4 (GAlaxies with Dark matter and Gas intEracT) by Volker Springel et al.
  • Used as a reference for what a well-organized astrophysical simulation codebase should look like. Also an awesome collaboration/project that was an inspiration to me.
• The Barnes-Hut Approximation: Efficient computation of N-body forces by Jeffrey Heer
  • An epic article with interactive visualizations outlining the Barnes-Hut algorithm.
• STARFORGE: Star Formation in Gaseous Environments by Mike Grudić et al.
  • Another cool collaboration/project that inspires me.
• PCG, A Family of Better Random Number Generators by Melissa E. O'Neill
  • Fast/high-quality/simple hash functions used for generating the simulation intial conditions and graphical visualizations.
  • Hash Functions for GPU Rendering by Mark Jarzynski and Marc Olano
    • Source of the pcg4d() variant of PCG.
• stb: single-file public domain (or MIT licensed) libraries for C/C++
  • Source of the stb_image.h and stb_image_write.h C headers for image file I/O. I highly recommend anybody working on a small C/C++ project like this take a look at the available stb headers.

Documentation

Prerequisites

Starflood was developed on a couple of Linux machines (x86 and ARM) and is intended to be ran on UNIX-like operating systems. Since November 2017, every system on the TOP500 list is running Linux. The compatibility/stability with other types of systems may vary, but feel free to open an issue or pull request if you encounter any issues.

The following sections assume you are using a *NIX system with Git, GNU Make, and a C compiler (such as Clang or GCC) installed. In addition, ffmpeg is recommended for encoding image frame sequences as playable video files.

Obtaining the Source Code

First, clone the Starflood repository (--recurse-submodules will automatically clone and initialize any of Starflood's git submodules):

git clone --recurse-submodules https://github.com/Zi7ar21/starflood.git

Then, change to the directory of the repository root:

cd starflood

Configuration

Before compiling, there are a few parameters at the top of src/config.h you may want to change, such as NUM_BODIES:

// number of bodies in the simulation (N)
#define NUM_BODIES 65536

Building the Code

First, create a directory for the build. The recommended default is build (already in the .gitignore). You can create it now (if it doesn't already exist):

mkdir -p build

GNU Make

First, make any desired changes to the Makefile. Some lines are commented/uncommented near the top of the file that you might want to tweak.

For parallel compilation, use -j to specify the number of jobs:

make -j$(nproc) all

($(nproc) just returns the number of available processors, it can be substituted for a number or decreased to meet memory/resource limitations).

To clean up after the build (required if any modifications have been made to the source code):

make clean

Flag Description

• -ffast-math: Allows replacement of standard math library functions with native instructions (i.e. sqrt() becomes the native x86 SSE instruction sqrtss). Note: This flag may also change associativity (order of floating-point operations), causing runs to be non-deterministic across compilers and vendors!
• -march=native: Tells the compiler to tune generated code for the local processor on the host machine (i.e. cache size-aware optimizations, allows the use of instruction sets such as x86 AVX or ARM NEON).

OpenMP Offloading

Starflood supports offloading to devices (i.e. coprocessors, GPUs, etc.) using OpenMP target directives.

Intel oneAPI DPC++/C++ Compiler

source /opt/intel/oneapi/setvars.sh

For OpenMP offloading using the Intel OneAPI DPC++/C++ Compiler:

icx -fiopenmp -fopenmp-targets=spir64 -march=native -O3 -pedantic -std=c99 -Wall -Wconversion -Wextra -Wshadow -o build/starflood src/*.c -lm

NVIDIA HPC SDK

Tip: On Arch Linux, the NVIDIA HPC SDK has a package (extra/nvhpc).

For OpenMP offloading using the NVIDIA HPC Compilers:

nvc -gpu=ccnative -mp=gpu -fopenmp -g -march=native -O3 -pedantic -std=c99 -Wall -Wconversion -Wextra -Wshadow -o build/starflood src/*.c -lm

• -mp=gpu: Enables compilation of OpenMP target constructs for GPU execution.
• -gpu=ccnative: Only generate code for visible GPUs on the compiler host machine.

Mounting a tmpfs (Optional)

Disk I/O can be annoying during development if you aren't planning on keeping run data or large frame sqeuences around for a while.

Thankfully, Linux has an easy way to create a virtual memory filesystem (sometimes called a "ramdisk" by Windows users) using the mount comand:

mkdir -p out

sudo mount -o size=4G -t tmpfs tmpfs out

Where size=4G indicates a filesystem size of 4 GiB.

Remember to unmount the tmpfs when you are finished!

sudo umount out

Please see Tmpfs in The Linux Kernel documentation or tmpfs man page for more details.

man tmpfs

Running Starflood

If file I/O is enabled, please ensure the appropriate output directories exist. The default directories are out for statistics/timings, out/sim for simulation snapshots, and out/vis for visualizations.

mkdir -p out/sim out/vis

To safely stop a run prematurely, create a file named stop in the output directory. If OUTPUT_DIR is "./out" (defined in src/config.h):

touch out/stop

Running with Nice

nice is a *NIX command that can be used to run a program with a higher/lower niceness (userspace priority).

The following command will run ./build/starflood with a niceness value 1 higher than the shell nice was called from:

nice -n 1 ./build/starflood

A higher niceness value tells the scheduler to select a lower priority, which ensures that other processes running on your system (terminal emulators, window managers, etc.) don't get starved of resources.

There shouldn't be any problems just running starflood by itself, but if parallelization is enabled (using all available processors) it helps ensure you can still control the system (or do other tasks) during runs.

For the most accurate profiling however, you shouldn't use too high of a niceness value (or else you run the risk of overly frequent context switching causing high variability in execution timing).

Viewing the Visualization

Encoding an Image Frame Sequence with ffmpeg

Look at encode_ffmpeg.sh for examples.

Playing an Image Frame Sequence with ffplay

Look at ffplay.sh for examples.

Simulation Snapshots

Simulation snapshots are currently saved in a raw binary format, that depends on the options Starflood was configured with.

Profiling Starflood

GNU Time

A simple way to time execution is using GNU Time:

/usr/bin/time -v ./build/starflood

Linux perf

Performance Counters

You can collect and measurements from Hardware performance counters during execution (elevated privileges required for perf stat):

sudo perf stat -ddd sudo -u $USER ./build/starflood

NVIDIA Nsight Compute

[to-do]

Profile-guided optimization (PGO)

Compilers such as Clang/GCC support profile-guided optimization (PGO), a technique where profiling data collected during execution can be used to further optimize generated code for the target machine.

Useful article: https://ddmler.github.io/compiler/2018/06/29/profile-guided-optimization.html

Clang

See "Profile Guided Optimization" in the Clang Compiler User's Manual.

GCC

See "3.12 Options That Control Optimization" in the GNU Manual.