Skip to content

Optimise particle I/O with chunking and add verbosity control#12

Merged
WillJRoper merged 34 commits intomainfrom
only-load-what-is-needed
Nov 19, 2025
Merged

Optimise particle I/O with chunking and add verbosity control#12
WillJRoper merged 34 commits intomainfrom
only-load-what-is-needed

Conversation

@WillJRoper
Copy link
Collaborator

Implements chunked particle reading to minimize I/O operations and adds configurable output verbosity. More specifically:

  1. Chunked Particle Reading

Optimised particle I/O by reading contiguous chunks instead of individual cells:

  • Added ParticleChunk struct to track contiguous cell ranges
  • Implemented gap-filling strategy: merge nearby chunks if gap < gap_fill_fraction * total_particles
  • New gap_fill_fraction parameter controls tradeoff between unused particle reads and I/O call count
  • Reduces HDF5 read operations by orders of magnitude for sparse grids

Performance impact: Fewer I/O calls at the cost of reading some unused particles (controlled by gap fraction).

  1. Verbosity Control

Added optional verbosity argument for cleaner output:

  • 0 = errors only
  • 1 = rank 0 only (default)
  • 2 = all ranks

Usage: ./parent_gridder params.yml 8 42 1

  1. Code Cleanup
  • Removed unnecessary grid point percolation code (grid points never leave top-level cells)
  • Simplified output functions by ~50 lines
  • Added debug checks for grid point distribution invariants

WillJRoper and others added 3 commits November 18, 2025 19:04
Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
@WillJRoper WillJRoper merged commit 3e454f8 into main Nov 19, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant