Skip to content

Renaming Nulls to NANs #287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ram-from-tvl wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Renaming Nulls to NANs #287

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a296872
finished the renaming
ram-from-tvl Sep 2, 2025
f58d29c
Merge branch 'main' into renaming-null-to-nans
ram-from-tvl Sep 2, 2025
60ee2e2
Merge branch 'main' into renaming-null-to-nans
ram-from-tvl Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 10 additions & 10 deletions src/nwp_consumer/internal/entities/tensorstore.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ class ParameterScanResult:
This is determined according to the parameter's limits and threshold.
See `entities.parameters.Parameter`.
"""
has_nulls: bool
"""Whether the parameter's data contains null values."""
has_nans: bool
"""Whether the parameter's data contains nan values."""


@dataclasses.dataclass(slots=True)
Expand Down Expand Up @@ -336,15 +336,15 @@ def _has_nans(store_da: xr.DataArray) -> ResultE[bool]:
os.getenv("ALLOWED_VALIDATION_FAILURE_PERCENTAGE", "0.02"),
)

def _calc_null_percentage(data: np.typing.NDArray[np.float32]) -> float:
nulls = np.isnan(data)
def _calc_nan_percentage(data: np.typing.NDArray[np.float32]) -> float:
nans = np.isnan(data)
if 0 in data.shape:
log.warning(
"Validation region has 0 area, check input slices correspond"
"to coordinate values in the dataset",
)
return 1.0
return float(nulls.sum() / np.prod(nulls.shape))
return float(nans.sum() / np.prod(nans.shape))

if "latitude" in store_da.dims:
spatial_dims: list[str] = ["latitude", "longitude"]
Expand All @@ -362,7 +362,7 @@ def _calc_null_percentage(data: np.typing.NDArray[np.float32]) -> float:
)

result = xr.apply_ufunc(
_calc_null_percentage,
_calc_nan_percentage,
store_da,
input_core_dims=[spatial_dims],
vectorize=True,
Expand All @@ -376,14 +376,14 @@ def _calc_null_percentage(data: np.typing.NDArray[np.float32]) -> float:
log.warning(
f"Dataset failed validation. "
f"{failed_image_percentage:.2%} of images have greater than "
f"{int(nans_in_image_threshold * 100)}% null values"
f"{int(nans_in_image_threshold * 100)}% nan values"
f"({failed_image_count}/{total_image_count})",
)
return Success(True)
log.info(
f"{failed_image_count}/{total_image_count} "
f"({failed_image_percentage:.2%}) of images have greater than "
f"{int(nans_in_image_threshold * 100)}% null values",
f"{int(nans_in_image_threshold * 100)}% nan values",
)
return Success(False)

Expand Down Expand Up @@ -434,7 +434,7 @@ def validate_store(self) -> ResultE[None]:
return Failure(e)
case Success(scan):
log.debug(f"Scanned parameter {param.name}: {scan.__repr__()}")
if not scan.is_valid or scan.has_nulls:
if not scan.is_valid or scan.has_nans:
return Failure(ValueError("Parameter validation failed."))
"""

Expand Down Expand Up @@ -510,7 +510,7 @@ def scan_parameter_values(self, p: Parameter) -> ResultE[ParameterScanResult]:
ParameterScanResult(
Copy link
Contributor

@peterdudfield peterdudfield Sep 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have you changed the object ParameterScanResult anywhere? I suspect you will need to if you want this change.

Aslo have you run pytest to check these changes?

Copy link
Author

@ram-from-tvl ram-from-tvl Sep 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes @peterdudfield , I updated the ParameterScanResult object completely.

  • Field renamed: has_nulls: bool → has_nans: bool with updated documentation
  • scan.has_nulls → scan.has_nans and has_nulls=False → has_nans=False
  • Tests pass: All 4 tests in test_tensorstore.py run successfully.

Please let me know if any changes are required..?
Thank you!

Copy link
Contributor

@peterdudfield peterdudfield Sep 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for that

mean=mean,
is_valid=True,
has_nulls=False,
has_nans=False,
),
)

Expand Down