Support matrix reordering on the GPU

[maddyscientist]

At the moment, xGPU copies the data back to the host in whatever format the kernel uses. This isn't necessarily the most intuitive format, as it often means a tiled-triangular order. We should support reordering on the device, and in addition to supporting the existing triangular ordering, we should also support a full matrix.

[sabourke]

From my experience upper triangular is the common format:
1,1; 1,2; 1,3; ....
2,2; 2,3; ...

Measurement Sets* and AIPS order the axes:
Baseline, Freq, Pol, (Re,Im)
going slowest to fastest. I suspect Miriad does the same.
There isn't a convention for the order of the pols. They're usually stored as
AIPS: AA BB AB BA
MS: AA AB BA BB

• Measurement Sets can abstract away the actual storage.

For us it probably makes more sense to have frequency as the first axis.

I wouldn't imagine the full matrix form is used by many.

A related issue is that the lines aren't always in nominal order. Conceptually, the nicest place to put them back in the right order would be prior to the cross multiply but the data rates are high there so post accumulation is probably the place to do it. Does it make sense to do this on the GPU too?

[maddyscientist]

If you want upper triangular order, I presume you'd want column major (as well as conjugating the matrix elements)? Row major for upper triangular order is painful (equally so for column-major for lower triangular).

[sabourke]

In general, user software doesn't store the visibilities in correlation matrix form. Each antenna pair combination ("baseline" from here on) is stored as a row in a table. There isn't a mandatory order but it would usually be:
1,1
1,2
1,3
2,2
2,3
3,3
So that's upper triangular, row major, right?
Yeah, they're conjugated compared to how we get them now.

How about we test out whatever baseline ordering is most convenient/efficient to implement. The main thing is to get:
An intuitive ordering of some sort (not the 2x2 thing)
The fastest changing axis as Re,Im
The second fastest changing axis as pol
Whither the slowest axis should be baseline or frequency depends on the science data product that's required. Traditionally it's baseline but there's a case to be made for frequency too.

[maddyscientist]

So I propose the following:

We presently have three orders, where TRIANGULAR_ORDER is the row-major lower-triangular

// Used to indicate matrix ordering
#define TRIANGULAR_ORDER 1000
#define REAL_IMAG_TRIANGULAR_ORDER 2000
#define REGISTER_TILE_TRIANGULAR_ORDER 3000

I sugest I also add

#define LOWER_TRIANGULAR_ORDER TRIANGULAR_ORDER
#define UPPER_TRIANGULAR_ORDER 4000

where the new lower type aliases the preexisting one, but I add it for symmetry since I would have the upper one. These use be row-major and column-major with respect to the baselines, respectively, e.g.:

1/2 i (i +1) + j and 1/2 (j + 1) j + I

where we have a conjugation of elements between the two orderings. Unless they are really needed I'm not going to add column major lower triangular or row major upper triangular order, respectively, as these orders are a pain in the butt (require counting from the end of the baselines, e.g., transform to coordinates i'=N-i, j'=N-j).

With respect to the ordering of channel / baseline / pol / complexity, we can expose this as well

#define CHAN_BASELINE_POL_CMPLX_DIM_ORDER
#define BASELINE_CHAN_POL_CMPLX_DIM_ORDER
etc.

So the different orderings that can be specified is the tensor product of the ORDER and DIM_ORDER types (perhaps I need to rethink the naming conventions).

Moreover, I'll add a couple of options to the interface:

1. What order should the kernel use when correlating?
2. What order should we convert the final correlation matrix to before dumping on the CPU?

Does that meet all of your requirements?

Last question: would you prefer lower/upper triangular order or strictly- lower/upper triangular order? e.g., with or without the autos? If you want the strictly triangular matrices we can make that an option too.

---

This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by

reply email and destroy all copies of the original message.

[sabourke]

That all sounds good. Personally we use the autocorrelations but many people do not.