Fix broken cases in USFS Catchpole

[rmcdermo]

At this point we have several cases that we effective take offline by using a "0" instead of a "Rate of Spread" quantity for USFS Catchpole. There are two issues:

1. Cases that are killed by the CTRL function do not have _devc data within the metric window.
2. Some case have a period where the slope is relatively flat and the "slope" logic returns a nan for something that should just be 0.

I tested Jason's new CTRL logic and fixed a couple of minor things:

&DEVC ID='xT', QUANTITY='TEMPERATURE', SPATIAL_STATISTIC='MAXLOC X', XB=0,8,-0.1,0.1,0,0.203 /
&DEVC ID='x',QUANTITY='CONTROL VALUE',CTRL_ID='x proc'/
&DEVC ID='flag val',QUANTITY='CONTROL',CTRL_ID='flag'/

&DEVC ID='t', XYZ=0,0,0, QUANTITY='TIME', OUTPUT=F, SETPOINT=11.0/
&DEVC ID='HRR', QUANTITY='HRRPUV', XB=-1.,9.,-1.5,1.5,0.,3., SPATIAL_STATISTIC='VOLUME INTEGRAL',
      OUTPUT=F, SETPOINT=0.1, LATCH=F, TRIP_DIRECTION=-1/
&CTRL ID='flag', FUNCTION_TYPE='ALL', INPUT_ID='t','HRR',INITIAL_STATE=T/
&CTRL ID='x proc',FUNCTION_TYPE='MULTIPLY',INPUT_ID='xT','flag val'/

Eric, I leave it up to you how you want to proceed. If you want help changing the slope logic let me know.

[ericvmueller]

How about I implement that control logic and run through the cases on blackbird so the rest of the validation updates can continue on spark? It might take several days but then we can take a look at the scatterplot using this approach where position gets clipped to zero and decide if that gives a fair representation of the whole set.

[ericvmueller]

Cases are still running, but here is an example of how it looks for a case that does not spread

I can understand Randy's argument for using the clipped position to get a low/zero slope that would be included in the scatterplot.

However, this is how it looks for a case that does spread

First, I'm not sure why the two devices are different for non-zero values. This is a direct plot of x and xT from the code above.

Second, the issue is that before we used distance (dependent variable) bounds when computing slope, just looking between 2 m and 7 m from the ignition. That works for this case, because we would reject the last 30s of nonsense data. But it doesn't work for the case above where the zero positions are considered relevant to the calculation of slope.

Maybe we need both? We can keep the position clipped by HRRPUV but also kill the simulation if it reaches 7 or 7.5 m, so there is only residual data with zero position if it is an unsuccessful spread case. We also need to drop the lower bound on position when using slope, but can set a lower bound on time to remove pre-ignition data. Then there should always be devc data to feed to the slope metric.

[rmcdermo]

This is the point I was at where I punted. I think somehow the logic has gotten way too complicated. I think we should be able to default ROS=0, and if data exists between 2<x<7, then we compute a new slope. If there needs to be another mask, we could use it, but what gives us trouble is when we produce a nan.

[ericvmueller]

Okay then I would say to just leave the original kill function, and I'll look into the slope metric instead. It looks like only CSIRO and Catchpole that use it, so i think it would be fair to just ensure we get a value of zero from the python script for either of the two conditions you point out above.

[ericvmueller]

I was able to tweak fdsplotlib to give zero slope if there are no data within the bounds in FDS_validation_dataplot_inputs.csv, rather than giving a warning.
But I can see having this special case come back to bite us at some point where we actually do want firebot to fail. It might be worth considering having a column in FDS_validation_dataplot_inputs.csv which is 'no_data_value'. If it's not set, then cases will fail but if it is set we default to that value if there are no valid data.

As to the question of including 'no spread' cases in the scatterplot, this is what we end up with

I'm not sure why the bias is on the high side.

[rmcdermo]

Please send me your pkl file. I should be able to use that to get the bias fixed.

[rmcdermo]

Kevin here is the linear plot for Catchpole only with latest data and fdsplotlib (no culling of cases; no spread taken to be 1e-12 slope). Compare this with Eric's plot above, which you will get if you set Plot_Min=1e-12 in FDS_verification_scatterplot_inputs.csv for Rate of Spread.

[mcgratta]

Eric -- is it possible to determine whether or not the non-spreading cases are beyond the model capability and remove them? Or could we fix them? I think that the methodology for calculating bias and standard deviation is not appropriate in this cases where there are these spurious points. I suggest we filter out the cases with spread rates less than 1 mm/s until we find a way to either fix or remove them.

[ericvmueller]

Because the scatterplot lower limit is currently 1 mm/s and the non-spreading cases have a zero or very small spread rate, I think they are already removed from the plot. No?
This was a bit the idea of the 'special' plots you can find at the end of the Catchpole section in the validation guide (Fig 14.123). It highlights the fact that there are some cases which don't spread, beyond scatterplot stats, and what the parameters are that result in this.

To understand what is going wrong is more of a research question, IMO, and so the idea of the special plot is to guide where to look moving forward. For example, with the NFRL tunnel we can reproduce some of these problematic experiments, perhaps with more diagnostics. I hesitate to remove the cases from being run entirely because it hides the issue.

[mcgratta]

I made some progress on this issue. The "MF" cases are improved by using the parameters for "Pine" from the Aalto Woods cases. I'm overhauling the input files to use the new parameters.

[ericvmueller]

I also ran the cases testing the same kind of logic for clipping the fire position discussed here for CSIRO:
#15832
to make things consistent for getting ROS for the the scatterplot

But I learned the hard way that CTRL logic uses instantaneous data that can have a lot of noise. So I'm running them again with SMOOTHED inputs and then I can merge those changes as well.