tclean: to concat or not to concat

[teuben]

Just ran into one of those notorious questions if to concat or not. In the past, with TPMS we found we should not concat, and the TPMS should be at the end of the list. With M100 there has to be a concat....

Now I'm running into the sky where e.g. in sky0.py the INT file gives identical maps if concat is used or now. But on sky4.py the qac_mac() , where no TPMS is used, there is a small (< 1%) flux difference if concat is used or not. Frustrating.

[dpetry]

If the setup of the MSs is identical and you don't need to use visweightscale, there should be no difference
and it is more efficient not to concat.
Why do you concat in this case?
Note that tclean by default does an internal conversion of the spectral frame to LSRK in order to align the grids.

[teuben]

In the past there were cases where this made a difference, but in qac_mac() it should not. So i expected an identical result when using concat or not, and to my surprise there was a small difference.Until I understand this, the "make issue12" in qac/test can be run to track this issue. the current version shows INT, TPINT and MAC image stats:

  QAC_STATS: issue12/clean3/sky_int.image_box1.fits    -217.89788960816759 -0.052325  False
  QAC_STATS: issue12/clean4/sky_int.image_box1.fits    -217.89788960816759 -0.052325  True
  QAC_STATS: issue12/clean3/sky_tpint.image_box1.fits  7261.5256019452918   0.982893  False
  QAC_STATS: issue12/clean4/sky_tpint.image_box1.fits 14446.892441834931    0.999994  True
  QAC_STATS: issue12/clean6/macint.image.pbcor         5300.8781113509658   0.712568  False
  QAC_STATS: issue12/clean7/macint.image.pbcor         5301.0425287754842   0.712558  True

Here only the flux and "sratio" are listed, but clearly showing that clean3 and 4 are the same for INT but not TPINT, and MAC.

[LydiaMF]

Here are some of my experiences for M100:
If I don't concat, feed tclean with TPms last (putting it first not tested yet), I get a segmentation fault (core dumped).
If I concat, I get errors like:
concat:
WARN MSConcat::copySpwAndPol Negative or zero total bandwidth in SPW 0 of MS to be appended.
tclean:
WARN MSTransformRegridder::combineSpwsCore SPW 4 cannot be combined with SPW 3. Non-matching ref. frame.
SEVERE MSTransformRegridder::combineSpwsCore Error combining SpWs

It works like a charm when I mstransform the INT-vis to the frequency range of the SD image (I think the main issue here is that INT-data should have only one spw, not the actual frequency range itself - but need to check) and don't concat, but mstransform takes very long. Looking at the variety of behaviours of the dataset depending on simulated/real and spectral setup, mstransform could be the safe option. I propose to put it into the first step of the DC_script right after the concat of all the INT-data, so that one does not loose time when playing with the TP2VIS step. In fact, we could use the mstransformed (and therefore slimmed) INT-vis for all combi methods in the following steps.

What do you think?

[LydiaMF]

OK, have to think of the case what to code for continuum data...

[dpetry]

@teuben , the concat warning message seems to explain what is happening: there is an SPW (0) which has zero total bandwidth.
Did you verify if this is really the case? If so, fixing the SPECTRAL_WINDOW table and giving that SPW a non-zero BW should help.
How was the MS created which causes the message? TP2vis? If so, TP2vis may not be filling the SPW table correctly.

[LydiaMF]

listobs for INT-ms:

SpwID Name #Chans Frame Ch0(MHz) ChanWid(kHz) TotBW(kHz) CtrFreq(MHz) BBC Num Corrs
0 3840 TOPO 113726.419 488.281 1875000.0 114663.6750 1 XX YY
1 3840 TOPO 111851.419 488.281 1875000.0 112788.6750 2 XX YY
2 3840 TOPO 103663.431 -488.281 1875000.0 102726.1750 3 XX YY
3 3840 TOPO 101850.931 -488.281 1875000.0 100913.6750 4 XX YY

listobs for TP-ms:

SpwID Name #Chans Frame Ch0(MHz) ChanWid(kHz) TotBW(kHz) CtrFreq(MHz) Corrs
0 bandtp 70 LSRK 114732.897 -1922.517 -134576.2 114666.5703 I

So, we have a negative bandwidth in the TP-ms. By the way, does TOPO vs LSRK pose a problem here, too?

[dpetry]

Negative is treated as zero. Yes, that sign should be flipped.
Also, the ref frame needs to be the same if you want it to be processed together with the others.

[LydiaMF]

The ref issue would require a mstransform or is there another - less time consuming - way?

[LydiaMF]

Just as a test, I fixed the bandwidth value of the TP-ms and run tclean without concat before and get a seg-fault again.

[LydiaMF]

Just looked at the output of the mstransformed INT-ms - it is still TOPO. So, TOPO and LSRK don't create an error for tclean.

[dpetry]

One step at a time. The first question to answer is: which ref frame is the TP data really in?
I suppose TOPO.
Once we know that, we can decide on transformations.

[LydiaMF]

So far it looks like the issue of tclean is the number of spws per ms-file (4 vs 1). turning all 4 spw into 1 spw already solved the issue. I just trimmed the frequency range to the relevant range for time and disk-space reasons.

[LydiaMF]

TP is in LSRK - see listobs above