Solver sets excess flags with minvisratio

[henedler]

I found a couple of cases where the DP3 solver is setting unexpected flags, I noticed this on a scalarphase solve of LBA data in a phased-up MS. Solutions in the frequency range 56-60MHz were fully flagged without a clear reason, this happened for multiple observations and observations with different total frequency coverage.
Here how the problem looks like in solution space (the MS file itself does not contain any flagged data in that frequency range):

Some testing suggests that this issue is linked to the minsivratio option of the solver, for which we use 0.3. A value of 0.16 or lower makes the problem disappear in a test case.

I created a minimal example with one timestep: https://cloud.hs.uni-hamburg.de/s/bskRjxSmP4EXEYJ

[tammojan]

Thanks for the small example! This probably has to do with the weights of the measurement set: they are smallest in the range of channels for which the solutions are flagged. Sorry for not putting labels on the axes: horizontal is channel number, vertical is WEIGHT_SPECTRUM. The channel numbers for which the solutions are flagged are 332-426.

[henedler]

Ok, yes this is before we update the weights with the bandpass, that explains the shape. So at some point the weights enter into the minvis logic?

[henedler]

Is there any news on this issue? Currently, we removed the minvisratio feature from our pipelines since it clearly introduces unwanted flags in LBA data that is not bandpass-corrected.

[aroffringa]

I looked at the minimal example, and I see data that has a weight of zero in the channels that are consider to not pass the minvisratio. DDECal seems to correctly count these channels as flagged and flags the data correspondingly. My suspicion is therefore that this is not a bug, but an issue with your data.

Your weights on the 'good' data are also really small: 1e-14. Maybe earlier processing has rounded the weights to zero somewhere...

[henedler]

The values of 1e-15 etc are normal for LBA data, that's what we get out of the LTA. Indeed, it might be that somewhere these small weights are rounded to zero, and then this leads to the flags. But there must still be some connection to the minvisratio option, since setting the value for this option affects whether the flags appear or not.

[aroffringa]

Hi Henrik,

The minvisratio just puts the FLAG column to true based on if there's enough data available. In this case there's not, according to the weights, so you're right minvisratio affects the flags, but the step itself does what it intends to do. If you wouldn't use minvisratio, you still have a lot of zero-weight data which wouldn't be used in calibration, imaging, etc.

I think what's the issue here is that the dynamic range of the weights is too large. They span 4 orders of magnitude, which means that even if the zero data was not zero but 1e-15, it would have practically no contribution in calibration and imaging compared to values with weights of 1e-11, which can also be found in your MS. I didn't think that's 'common' in LBA data, but I don't see that much LBA data -- however even if it is common I think it's incorrect, so that means that earlier processing must have done something bad to those weights (i.e., it shouldn't be normal).

I later came to the realization that the standard Dysco settings can not capture more than ~3 order of magnitude: it saves weights with 12 bits or so, which limits de dynamic range to 1:4000. So Dysco might be the another factor in this. However, I'd repeat again that Dysco rounding weight values 1:4000th to zero should never be an issue, because the weight of such visibilities was basically already zero. It really is the dynamic range of the weights where something is wrong.

For now a fix could be to add something like 1e-13 to all weights (and hope that the FLAG column capture all bad data), but clearly the resulting weights would not be optimal.