Skip to content

Conversation

@NateTomasetti
Copy link

@NateTomasetti NateTomasetti commented Nov 18, 2025

Continuous Ranked Probability Score (CRPS) is a proper scoring rule to evaluate the prediction of a probability distribution.
It rewards distributions that:

  • Have a large amount of density close to the actual value
  • Are not spread too wide
  • Hits the X% percentile X% of the time

This implements a scaled version of CRPS in the mmm-eval alongside the existing accuracy metrics (MAPE, R2, etc.).
The scaling factor makes this a generalisation of MAPE, so the output value can be consistently compared against a threshold regardless of what kinds of values the data has.

  • Regular CRPS is a probabilitstic version of MAE, so the value depends on the magnitude of sales.

Note

Introduces scaled CRPS accuracy metric and plumbs posterior predictive distributions from adapters into validation to compute it, with docs, thresholds, tests, and version updated.

  • Metrics:
    • Add scaled CRPS computation (crps_one_date, calculate_crps) and include in AccuracyMetricNames/Results and CrossValidationMetricNames/Results (mean_crps).
    • Update thresholds to include CRPS/MEAN_CRPS.
  • Adapters:
    • Change BaseAdapter predict/fit_and_predict/fit_and_predict_in_sample to return (posterior_mean, posterior_distribution).
    • Meridian: return posterior mean and flattened distribution; apply holdout mask to both.
    • PyMC: use predict_posterior(combined=True); return mean and distribution (transposed) tuples.
  • Validation:
    • Accuracy and CV tests now pass predictive distributions to compute CRPS.
    • Add distribution_to_dataframe utility to reshape distributions for metric calc.
  • Docs:
    • Add CRPS to metrics/tests guides, definitions, interpretation, and benchmarks.
  • Tests:
    • Update unit/integration/E2E tests for new adapter return types and CRPS calculations.
  • Release:
    • Bump version to 0.13.0; update CHANGELOG.md.

Written by Cursor Bugbot for commit 9058b5a. This will update automatically on new commits. Configure here.

Copy link
Contributor

@sjmccorm1993 sjmccorm1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, could you just

  • ensure the unit tests still run and add tests for CRPS specifically
  • update metrics.md and tests.md to document the new metric you've added?

@sjmccorm1993 sjmccorm1993 self-requested a review November 25, 2025 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants