Skip to content

Conversation

@samland1116
Copy link
Contributor

  • adds brier score and brier skill score to probabilistic metrics

    • requires the user define a quantile threshold to evaluate BS (fractional, between 0-1. default=0.75)
  • updates probabilistic metrics routine to:

    • to avoid failure when calculating skill score if encountering divide by 0 in _calculate_metric_skill_score(), updated to use try_divide(), returns null. Provides warning for which configs fail.
    • Updates routine in _calculate_metric_skill_score() to calculate skill score to allow for metrics.query() calls with multiple probabilistic metrics. Was encountering an error when trying to calculate CRPSS/BSS in one call. Corrected to work as intended by updating _post_process_metric_results.
      • since we were assigning the results of _calculate_metric_skill_score() directly to the self.sdf, the method was removing all columns aside from the columns specified in group_by, the column corresponding to the metric in the input arg, and the calculated skill score column. Once the method gets called again for the next metric in include_metrics -- the metric column it is targeting was removed in the first iteration and fails.

@samland1116 samland1116 self-assigned this Nov 20, 2025
@samland1116 samland1116 added enhancement New feature or request v0.6-dev Denotes that the issue is meant to be merged into the v0.6-beta development branch. main This is an issue that is considered v0.5.0 maintenance and should be merged into main priority: High This is blocking work, let's get it done as soon as possible. labels Nov 20, 2025
@samland1116 samland1116 linked an issue Nov 20, 2025 that may be closed by this pull request
@samland1116 samland1116 added this to the v0.6 Release milestone Nov 20, 2025
Copy link
Collaborator

@samlamont samlamont left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In test_ensemble_metrics() I was getting slightly different results that were causing the CRPS assertions to fail.

I was also not able to run both crsp and brier score in the same call (adding both to include_metrics) with the reference forecasts specified, which may have been a memory error on my system.

Do you see these as well? Seems like my local WSL might have some memory issues

@samland1116 samland1116 requested a review from samlamont December 8, 2025 19:09
@samland1116 samland1116 merged commit a5afc24 into main Dec 8, 2025
@samland1116
Copy link
Contributor Author

Merging -- might revise the size of the new test data soon to avoid unnecessary memory issues in a later PR

@samland1116 samland1116 deleted the 570-implement-brier-score-and-brier-skill-score branch December 8, 2025 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request main This is an issue that is considered v0.5.0 maintenance and should be merged into main priority: High This is blocking work, let's get it done as soon as possible. v0.6-dev Denotes that the issue is meant to be merged into the v0.6-beta development branch.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Brier Score and Brier Skill Score

3 participants