large-scale hot/cold cases too large to run metrics

[amymcgovern]

I setup a script to run the large cases in large_events.yaml and they are too large to run because we can't densify them.

joblib.externals.loky.process_executor._RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/amy/ExtremeWeatherBench/.venv/lib/python3.13/site-packages/joblib/externals/loky/process_executor.py", line 490, in _process
_worker
r = call_item()
File "/home/amy/ExtremeWeatherBench/.venv/lib/python3.13/site-packages/joblib/externals/loky/process_executor.py", line 291, in call
return self.fn(*self.args, **self.kwargs)
~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/amy/ExtremeWeatherBench/.venv/lib/python3.13/site-packages/joblib/parallel.py", line 607, in call
return [func(*args, **kwargs) for func, args, kwargs in self.items]
~~~~^^^^^^^^^^^^^^^^^
File "/home/amy/ExtremeWeatherBench/src/extremeweatherbench/evaluate.py", line 403, in compute_case_operator
_evaluate_metric_and_return_df(
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
forecast_ds=aligned_forecast_ds,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...<5 lines>...
**metric_kwargs,
^^^^^^^^^^^^^^^^
)
^
File "/home/amy/ExtremeWeatherBench/src/extremeweatherbench/evaluate.py", line 557, in _evaluate_metric_and_return_df
metric_result = metric.compute_metric(
forecast_data,
target_data,
**kwargs,
)
File "/home/amy/ExtremeWeatherBench/src/extremeweatherbench/metrics.py", line 44, in _compute_metric_with_docstring
return _original_compute_metric(self, *args, **kwargs)
File "/home/amy/ExtremeWeatherBench/src/extremeweatherbench/metrics.py", line 44, in _compute_metric_with_docstring
return _original_compute_metric(self, *args, **kwargs)
File "/home/amy/ExtremeWeatherBench/src/extremeweatherbench/metrics.py", line 44, in _compute_metric_with_docstring
return _original_compute_metric(self, *args, **kwargs)
File "/home/amy/ExtremeWeatherBench/src/extremeweatherbench/metrics.py", line 137, in compute_metric
target = utils.maybe_densify_dataarray(target)
File "/home/amy/ExtremeWeatherBench/src/extremeweatherbench/utils.py", line 649, in maybe_densify_dataarray
da.data = da.data.maybe_densify(max_size=max_size)
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
File "/home/amy/ExtremeWeatherBench/.venv/lib/python3.13/site-packages/sparse/numba_backend/_coo/core.py", line 1441, in maybe_densify
raise ValueError("Operation would require converting large sparse array to dense")
ValueError: Operation would require converting large sparse array to dense
"""

[linear]

EWB-73

[amymcgovern]

I tested both the full 5 year and a single year and both are too large

[darothen]

@amymcgovern can you share a code snippet that runs one of the cases that leads to this error?

[amymcgovern]

if you go to my branch (amy/new-event-types-paper) and run docs/notebooks/run_global_heat.py it will do it. You need my branch as I have made edits to the long_events.yaml file to try both the 5 year and the one year heat events

[aaTman]

maybe_densify() has a max_size argument that needs to be set to be much larger (I have a helper function utils.maybe_densify_dataarray() which includes max_size as a passthrough arg), or there needs to be a slightly different way to manage the point obs data, e.g. using utils.stack_dataarray_from_dims() to prevent densifying an absurdly large dataset.

[darothen]

@amymcgovern and I discussed this on Monday; if a configuration hack can get this done, then it's fine. But the fundamental analysis task here explicitly violates the problem scope we set for EWB - the data and task is way too large. If the config hack doesn't work then we will break out the core "Forecaster's Dilemma" work into a stand-alone analysis.