State of Dev 2025-04-22

[MikeNeilson]

Overview

There is a lot of work going on at the OpenDCS project and below describes some of the current state of things.
As always we seek new people to contribute to the project. That includes just trying RCs and reporting issues. Additional help doing triage, prioritization, and organization (links, assigning to GitHub projects and the like) would also be useful.

However, as many are seeing, bugs are being reported and we are struggling to get to them all. Some of this is due to the limit number of people actively contributing to the project, and some of it is that fixes those bugs or adding those features requires initial work before they can be properly addressed.

BLUF: We have way more work than the current contributors can meaningfully get to in a reasonable amount of time. If something interests you, please jump in, even if it's just paying a contractor to do it. We understand that not every OpenDCS user is going to be a developer themselves.

OpenDCS/REST API/WebGUI

Work to Modernize OpenDCS and improve the maintainability of the software is in full swing.
To support some required major database overhaul work and not duplicate any effort in the rest_api, we have created a 7.5 branch. This will be kept in line with the main branch and the database schema of 7.5 and 7.0 will be the same.

The changes in 7.5 are in how the database is accessed to allow:

1. Easier maintenance - will currently have two "database" mechanisms in the code and this lets us merge them
2. Improved performance and consistency - we can use proper transactions

Please note that with main and 7.5 not everything, especially the GUIs will behave correctly. Please report errors as you find them, but don't get hung up on things.

Work will continue on 7.0; however, not every fix will come to this branch as some of the fixes really do require the changes that drove the creation of 7.5.

Other changes coming to 7.5:

1. Renaming the OpenTSDB database to OpenHydroDb as OpenTSDB was taken well before OpenDCS used that name.
2. Increased JVM requirements. We will likely jump to Java 25 when releases, but will at least jump to Java 21
   a. We have too many critical dependencies that require at least Java 17, it is not realistic to stick with Java 8 or even Java 11.

7.0 will continue to target Java 8, but will likely receive fewer and fewer feature improvements as it will depend on contributor time.

LRGS

Any help in both of the following tasks would be greatly appreciated.
See LRGS Modernization for details of current LRGS work to be done.

Password Handling

In the creation of the LRGS Kubernetes Operator we have come to the realization that the username and password handling in the LRGS is, to be blunt, incredibly bad. While I don't want to break the 7.0 branch for existing users, main and 7.5 are going to get an overhaul to actually store password with appropriate hashing, and the client will just send the password given. As such, in 7.5 TLS will eventually be a requirement for sane usage of an LRGS. This effort will also expand into allowing other forms of authentication, like OpenID Connect using OAuth.

Archive

The LRGS currently expects to write to a filesystem. @MikeNeilson is using NFS in his HomeLab setup without issue, and the current USACE cloud setup appears to be using EFS without much trouble. However, even in the homelab case an actual filesystem isn't ideal. One of the major projects is to support additional mechanisms of storing the LRGS archive. Such as S3 style storage or a database. This work is currently stalled due to the volume of other work OpenDCS contributors are working on.

REST API and WebGUI

RMA has done a lot of work on the REST API and Web GUI, including some database interaction fixes to the main project, over the last few months.

Anyone interested in the use of this system should go check out the project: https://github.com/opendcs/rest_api.
The readme has instructions for running it in a development environment. Users of the OpenDCS Postgres and Oracle database should be able to use it without too much difficulty.

After a certain level of stability, we plan to merge the RestAPI code into the main project so it's easier to keep the database access up-to-date.

The GUI vs The WebGUI

With the addition of the WebGUI, the planned move to Java 11+, and the database interaction changes forcing overhauls to the existing GUIs we are at a crossroads of effort. Very shortly we need to make a decission about the following:

1. Only have the WebGUI - NOTE: we would still maintain a "desktop" app in a similar to how PgAdmin4 handles things
2. Maintain the WebGUI and Java GUI
   a. Migrate the Java GUI to OpenJFX (only in Main and 7.5, unless we can bump 7.0 to java 11.)

Other OpenDCS sub-projects

Gherkin test case processing

We have created a Gradle and Ant plugin that can process Gherkin .feature files of manual test cases into and publish into the KiwiTCMS system.
While not implemented yet, that plugin is designed to be somewhat generic. At this time we plan on also outputting to CSV, and possibly PDF or a "web page" to allow local creation and a run through any manual tests.

Maven Central Upload

https://github.com/opendcs/maven-central-upload/

Maven Central (https://central.sonatype.com) is sunsetting the current method of publishing our project artifacts to Maven Central and there is currently no first party (Gradle or Maven Central) plugin that handles the new publishing API. There are third party plugins, but they required too many changes to the project.

As such we have created a plugin that using the existing gradle maven-publish plugin information. This project is suitable for users outside of OpenDCS if they desire, though the goal is for it to be replaced by either Gradle or Sonatype. Unfortunately, given the current timelines, we did not see that happening before the existing method vanished.

OpenDCS Shef

https://github.com/opendcs/opendcs-shef

The current SHEF parsing in OpenDCS is limited to .A and .E formats, and even then doesn't always handle valid SHEF. This project is establishing an Antlr4 grammar to parser SHEF file.

It will include a generic Java library to output the data that one would otherwise get from shefit, and eventually include the creation of SHEF data in the desired format. We are not intending to recreate shefit, especially the -2 format that often looses information.

The Antlr4 grammar can also be used by other languages as that project can generate parsers for several languages.

Help here would be appreciated, currently only one contributor works on this in his spare time.

To-Date:

1. Can parse .A format statements
2. Can parse .E format statements
3. .B is in progress

The current conundrum is how to get the parser to ignore any text before valid format statements.

OpenDCS Java Algebra System

The current JEP project has not been updated in 8 years. the next replacement that seems suitable was https://github.com/JiachenRen/java-algebra-system; however, it appears that developer has either lost interest or doesn't have the time. The good news is he released the code under an MIT license and had a large suite of tests. So we have forked the project and are releasing it branded as OpenDCS. USACE is currently using it to process unit conversions in their database setup and it will eventually replace JEP in the expression algorithm of OpenDCS.

Kubernetes/Docker

We have started the creation of helm charts, the initial use of which almost immediately required the use of the Kubernetes operator model to really function. As such we have an OpenDCS operator: https://github.com/opendcs/opendcs-operator and a helm chart to deploy it: https://github.com/opendcs/charts

At this time the operator only handles the LRGS and as you will notice if you go to those sites, the documentation is rather lacking.

These are not production ready and we welcome anyone who is interested and wishes to contribute. Even just pointing out things I'm missing would be helpful.

For the Docker images in general, we will be starting to create them with an SBOM included as we've discovered that the AWS vulnerability scanner goes a bit... crazy, without it. And since the work required to do that is the same, we are going to start creating some multi-arch images. Namely x86_64 (intel/amd) and ARM64 (RaspberryPi and the ARM offerings that AWS and other cloud vendor provide.)

Dcp Monitor

There is some working going on over at https://github.com/opendcs/dcpmon to modernize that interface.
It will be some time before even a release candidate is ready as we are targeting the new HTTP-over-DDS interface to increase what we are able to accomplish with that interface.

Once this project is ready for an initial release OpenDCS may create a public LRGS. How that happens, when, and how it's funded are still to be determined. There will be rate-limiting to avoid excessive bandwidth costs; however, the actual HRIT stream is rather low in data usage so this is likely not an issue.

End of Report

Hopefully these will be more regular in the future. But as pointed out we have a lot of work to do and so, unfortunately, we tend to forget about these niceties like keeping interested parties informed.

[ktarbet]

Thanks for this report!