RAPID v2.0.0 Release Planning Discussion

[c-h-david]

This discussion is intended to outline the current thinking around
when and how to release RAPID v2.0.0, and to invite early
feedback.

RAPID v2 (https://github.com/c-h-david/rapid2) represents a substantial
redesign relative to RAPID v1 (https://github.com/c-h-david/rapid).
Although the scientific purpose is similar, the underlying architecture,
implementation language, and development philosophy have changed enough
to warrant a new repository and a deliberate release process.

Background

• RAPID v1 was primarily implemented in Fortran with MPI-based
  parallelism, targeting HPC environments.
• RAPID v2 is a Python-based reimplementation emphasizing:
  • Simpler execution and installation
  • Easier contribution and maintenance
  • A lower barrier to entry for new users

Because of this shift, RAPID v2 should not be viewed as a drop-in
replacement for v1, but rather as a next-generation implementation with
overlapping goals.

At present:

• The repository has no tagged releases.
• The codebase is still evolving.
• Several features are in flux.

This post aims to define a path toward a first stable public release.

Release Philosophy

• The first official release will be tagged v2.0.0.
• Prior to that, pre-release versions may be used to gather feedback
  and stabilize workflows:
  • v2.0.0-beta
  • v2.0.0-rc1, v2.0.0-rc2, ...

The intent is to avoid prematurely freezing the software while still giving
users something concrete to test and comment on.

Community Feedback and Beta Testing

Before tagging v2.0.0, we would like to solicit feedback from a small
group of experienced RAPID users, focusing on real scientific use
cases rather than synthetic benchmarks.

Potential beta testers include collaborators from prior RAPID studies

Feedback from this group will help guide: - CLI stabilization - Workflow
clarity - Documentation priorities

Proposed Criteria for v2.0.0-rc1

The following items represent minimum readiness for a first release
candidate.

1. Minimal Scientific Capability

At minimum, RAPID v2 should be able to run end-to-end and produce scientifically meaningful output given:

• River connectivity
• Model parameters
• External inflows
• Initial state

Advanced capabilities (e.g., uncertainty quantification or data
assimilation) may exist in the codebase but are not required for
v2.0.0.

2. Codebase Stability

• Core architecture is reasonably stable (no major refactors
  expected).
• Python 3 + SciPy + sparse-matrix stack is established.
• Existing quality controls are in place:
  • Linting
  • Doctests
  • Continuous Integration (CI)
  • Continuous Linting (CL)
  • Continuous Docker image deployment (CD)

Potential CI improvements under consideration:

• Automated comparison of NetCDF files.
• Automated comparison of CSV files
• Inclusion of a real-world test case (in addition to the sandbox) for CI. Ideally, this test case matches the tutorial example.
• Clear conventions for CSV inputs (e.g., whether headers are required).

3. Sandbox

• A clean, minimal sandbox configuration that runs successfully with
  the released code.
• Intended for quick testing and developer onboarding.

4. Command-Line Interface (CLI)

• Clear and consistent CLI behavior:
  • Informative --help output
  • --version flag
  • Usage examples
  • Progress indicators for long runs (where feasible). tqdm is candidate for this.
• Preference for minimal additional dependencies.

5. Tutorial (Real Science Use Case)

• A complete tutorial based on a real scientific application.
• Goes beyond simply running the model to include:
  • Inspecting inputs
  • Examining outputs
  • Basic interpretation and analysis

This tutorial will serve as the primary reference example for new users.

6. Communication Channels

• A Slack channel or workspace for early users and developers created on Slack.
• Message retention on the order of six months is sufficient.
• Added to current dev team workspace.
• Automated workflow bot, pointing to Discussions, User Guide.

7. Website

• A public-facing website connected to rapid-hub.org.
• Current draft sites:
  • https://c-h-david.github.io/rapid-hub/
  • https://github.com/c-h-david/rapid-hub

The guiding principle is: > Incomplete but public is better than
complete but private. Minimum deliverable for this activity:

• Home (clarify free to use, modify, include in your codebase)
• User guide
• Citing
• Contact (https://science.jpl.nasa.gov/people/chdavid/, Slack, GitHub Discussions)
• Slack channel bottom icon

8. Versioning and Distribution

• Adopt and document a clear versioning scheme (semantic versioning
  assumed here).
• Consider PyPI releases via GitHub Actions once v2.0.0-rc1 criteria
  are met.

Next Steps

• Convert the above criteria into GitHub issues.
• Define tentative milestone dates.
• Invite comments on:
  • Release criteria
  • Missing items
  • Expectations for v2.0.0

Feedback is very welcome, especially from users with prior RAPID
experience.

[c-h-david]

@riverma I cleaned up our brainstorm from yesterday. What do you think?

[c-h-david]

Adding @taddyb, @geosjim and @rileyhales to the brainstorm. Particularly interested in your thoughts on the data formats currently used in the SANDBOX.md. Apache Parquet was mentioned by @taddyb as a potential alternate to .csv and .nc4 to be considered here.

[taddyb]

Apache Parquet was mentioned by @taddyb as a potential alternate to .csv and .nc4 to be considered here.

Yup, I've found specifically for larger river networks, CSV files lose their advantage of readability, and become cumbersome for IO. I've saved matrices in the a sparse format with Zarr arrays. The sparse matrix form is very efficient for saving data files. I've been able to save a sparse network matrix with 850,000 river segments in 8MB of storage using the binsparse spec.

sandbox_adjacency.zarr.zip

I attached the sandbox adjacency file in the above zip as an example of the zarr binsparse format. Once unzipped, here is the code to read it from disk. Zarr version is >=3

"""A file to handle all reading from data sources"""
from pathlib import Path

import zarr
from scipy import sparse

def read_coo(path: Path, key: str) -> tuple[sparse.coo_matrix, zarr.Group]:
    """Reading a Binsparse specified coo matrix from zarr.

    Parameters
    ----------
    path : Path
        Path to zarr store.
    key : str
    Gage ID to read from the zarr store.
    """
    if path.exists():
        store = zarr.storage.LocalStore(root=path, read_only=True)
        root = zarr.open_group(store, mode="r")
        try:
            gauge_root = root[key]
        except KeyError as e:
            raise KeyError(f"Cannot find key: {key}") from e

        attrs = dict(gauge_root.attrs)
        shape = tuple(attrs["shape"])

        coo = sparse.coo_matrix(
            (
                gauge_root["values"][:],
                (
                    gauge_root["indices_0"][:],
                    gauge_root["indices_1"][:],
                ),
            ),
            shape=shape,
        )
        return coo, gauge_root
    else:
        raise FileNotFoundError(f"Cannot find file: {path}")

Parquet files would be efficient for saving anything tabular as well. I've used parquets as an alternative to geopackages since geometries can be compressed to well-known-binaries, saving space

[c-h-david]

Noted! Thanks @taddyb. Do you have a sense for how big the rapid_connect.csv corresponding to that same 850k size network matrix is? And how big a parquet file would be? I have grown to like the ability to create the network matrix on the fly dynamically for a given sort (usually in a riv_bas_id.csv that also allows sub-basin simulations). I wonder if parquet might be the happy place for storage and flexibility between csv and zarr here for network connectivity information.