Optimization of the Environmental Data Integration Pipeline for Spatiotemporal Consistency and Unit Safety #134
Description
Problem:
The current implementation of weather data fetching and merging in WeatherCondEnvAutomatic and WeatherCondFromFile lacks a robust, automated validation layer. Specifically, the code relies on hard-coded time offsets and issues "warnings" for unit mismatches rather than performing active normalization. This leads to potential "silent failures" where the ship's fuel consumption is calculated using mismatched atmospheric variables (e.g., mixing GFS and CMEMS data with slight temporal misalignments).
Key issues identified in the source code:
Time-Step Mismatches: In check_data_consistency, critical assertions regarding time-sync between wave and physics datasets are commented out because the datasets frequently contain different step counts.
Passive Unit Validation: The check_units method identifies discrepancies (e.g., Pressure in Pa vs. hPa) but only logs a warning instead of correcting the data.
Inconsistent Metadata: Variable naming (like height_above_ground vs. height_above_ground2) differs between the automated downloader and file-based loaders, causing potential KeyError exceptions during routing.
Proposed Approach:
To solve this, we should move from a "Checking" approach to a "Normalization" approach. The goal is to ensure that by the time a WeatherCond object reaches the routing algorithm, the data is guaranteed to be consistent.
Step A: Implement a Dynamic Time-Alignment Layer
Instead of using hard-coded offsets (like time - 30*60), we should implement a nearest-neighbor interpolation strategy using xarray.
Action: Modify WeatherCondEnvAutomatic.read_dataset to define a "Master Timeline" based on the user's requested departure and arrival.
Action: Use .interp(time=master_time) for all sub-datasets (Wave, Current, Wind) to ensure every coordinate point has a matching weather state.
Step B: Active Unit Normalization Engine
Convert check_units from a passive reporter to an active converter.
Action: Create a mapping of known units to SI standards.
Action: If a dataset is detected with Celsius, automatically convert it to Kelvin using the UNITS_DICT metadata before it is merged into the main self.ds.
Step C: Standardized Accessor Interface
Standardize the internal representation of atmospheric layers.
Action: Implement a mapping layer that renames provider-specific strings (like height_above_ground2) to a standard internal key (e.g., wind_height_10m) regardless of the data source.
Success Measures (The "Criteria"):
A successful resolution of this issue would be measured by:
Zero Commented Assertions: The check_data_consistency method should be able to run fully without throwing errors on standard CMEMS/GFS fetches.
Unit Invariance: The routing results should remain identical regardless of whether the input file uses Pa or hPa, as the normalization layer handles the conversion.
Reduced Algorithm Failures: Elimination of KeyError or ValueError exceptions currently caused by mismatched coordinates in IsoBased.estimate_fuel_consumption.
Impact of this :
This improvement directly enhances the Reliability and Scientific Accuracy of the tool. By ensuring that the "Ship Module" always receives perfectly aligned environmental data, we reduce the margin of error in fuel consumption predictions, which is the primary value proposition of the Weather Routing Tool.
@kdemmich Assign this to me