Matrix transformations confusing

[will-moore]

This sentence is confusing: "The first (last) axis in a coordinate system is the top (bottom) entry in the column vector".
It reads like "first" really means "last" and "top" really means "bottom"!

I'm kinda confused by the ways that the spec represents matrices.

The example at https://ngff-spec.readthedocs.io/en/latest/specification.html#matrix-transformations could be more realistic to help illustrate the concepts.

It looks like a matrix for a 3D image (x, y, z), but a matrix for a 3D image is a 4 x 4 matrix?

E.g. what does a matrix look like for a 3D image that is rotated 30 degrees in the Z axis?

Playing around with napari's helpers...

from napari.utils.transforms import Affine

# This looks clear - if I want to scale a 4D image by 2 in x and y. The affine matrix is a 5 x 5 with the first 4 columns being `C, Z, Y, X`.

Affine(scale=[1,1,2,2]).affine_matrix
array([[1., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0.],
       [0., 0., 2., 0., 0.],
       [0., 0., 0., 2., 0.],
       [0., 0., 0., 0., 1.]])


# This looks like a rotation matrix for a 2D image, with a 3 x 3 matrix and the first columns being `Y, X`.

>>> Affine(rotate=30).affine_matrix
array([[ 0.8660254, -0.5      ,  0.       ],
       [ 0.5      ,  0.8660254,  0.       ],
       [ 0.       ,  0.       ,  1.       ]])


# If I have an image that is `C, Y, X`, and I want to rotate it in the Z axis (X and Y coordinates change) this looks like the correct matrix - 4 x 4 matrix with the first 3 columns being `C, Y, X`.

Affine(rotate=[0, 0, 30]).affine_matrix
array([[ 1.       ,  0.       ,  0.       ,  0.       ],
       [ 0.       ,  0.8660254, -0.5      ,  0.       ],
       [ 0.       ,  0.5      ,  0.8660254,  0.       ],
       [ 0.       ,  0.       ,  0.       ,  1.       ]])

# The `rotate=[0, 0, 30]` represents [yaw, pitch, roll]`.

# Same result with:

Affine(rotate=30).expand_dims([0]).affine_matrix
array([[ 1.       ,  0.       ,  0.       ,  0.       ],
       [ 0.       ,  0.8660254, -0.5      ,  0.       ],
       [ 0.       ,  0.5      ,  0.8660254,  0.       ],
       [ 0.       ,  0.       ,  0.       ,  1.       ]])

Some examples like this would help to clarify.

[jo-mueller]

Hi @will-moore , thanks for the feedback. I'll also slip these changes into my upcoming PR to the ngff repo which contains an updated proposal and replies to the reviews (including yours). If it's confusing here, then it's confusing there as well and some more examples could go a long way.

One thing to point out from my side: If you'd agree with the page layout I would go ahead and merge #4 so that other PRs can concern themselves with smaller bits and pieces of the future page layout, i.e.

• best schema parser for webpage?
• Changelog layout?
• Semantic line breaks (Use semantic line breaks #1)

As soon as there is movement on the RFC5 review process, the work focus will probably shift here. By then it would be cool to be able to separate work on the page layout from work on the spec contents.

[will-moore]

Yes, makes sense to merge #4, then focus on tweaks and updates.

[jo-mueller]

Update on this end: Check out the preview on the full spec document (0.6dev2 if merged). Contains a lot of examples pulled into the text from json files!

Related PR

[will-moore]

The transforms examples are looking great and certainly help a lot.
However, I still think there's room for confusion which could be improved with more realistic examples (see discussion over at will-moore/napari-ome-zarr#2).

For example, none of the examples have a c axis whereas in practise many images are multi-channel.

Expressing scale as e.g. [1.0, 1.0, 0.42, 0.42, 0.42] for a 5D image makes it clear that the c axis is included, even though it will never be scaled.
The same is true for affine and rotation matrices.

If we provided an example of an affine matrix for a 5D image rotated around the Z axis, that could clarify a lot.

Even an affine transform for a CYX image that scaled the x axis by 2 would help. What's the affine transform that is the equivalent of a scale transform with "scale": [1, 1, 2]:

Does it look like this (which is what I would expect)

[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 2, 0],

or like this:

[2, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 1, 0],

which is what I think is being described in the spec?

x = 1*i + 2*j + 3
y = 4*i + 5*j + 6

[ 1 2 3 ][ i ]   [ x ]
[ 4 5 6 ][ j ] = [ y ]
[ 0 0 1 ][ 1 ]   [ 1 ]

[jo-mueller]

For example, none of the examples have a c axis whereas in practise many images are multi-channel.

Fair, I think that could be included.

which is what I think is being described in the spec?

Not quite. The example you are quoting is a 2d image (YX( without channel axis- The affine matrix there composes some other transforms, but only in 2d:

$$\left[\begin{matrix} a_{11} & a_{12} & t_1\\\ a_{21} & a_{22} & t_2\\\ \end{matrix} \right]$$

where $t_1$ and $t_2$ are the translation components and the rest is some mix of rotation/shear/scale.

If the image were CYX, the respective transform parameters should look like this, which corresponds to a scale of 1 in the channel axis.

$$\left[\begin{matrix} 1 & 0 & 0 & 0\\\ 0 & a_{11} & a_{12} & t_1\\\ 0 & a_{21} & a_{22} & t_2\\\ \end{matrix} \right]$$

To circle back, for CYX and only scale ["y": 1, "x": 2], the affine looks like this:

[1, 0, 0, 0],
[0, 1, 0, 0],
[0, 0, 2, 0],

What is pointed out in the spec is the form of the affine matrix in homogeneous form, which has that extra [0, ..., 0, 1] row at the bottom to make the matrix square.

[will-moore]

That's great and what I assumed.
So, if it's only a 2D image with scale ["y": 1, "x": 2] to convert from ["j", "i"] to ["y", "x"] then this would be

[ 1 0 0 ][ j ]   [ y ]
[ 0 2 0 ][ i ] = [ x ]
[ 0 0 1 ][ 1 ]   [ 1 ]

y = 1*j + 0*i + 0
x = 0*j + 2*i + 0

But that's quite different from what's in the spec, with x and y in a different order, same for j and i:

[ 1 2 3 ][ i ]   [ x ]
[ 4 5 6 ][ j ] = [ y ]
[ 0 0 1 ][ 1 ]   [ 1 ]

x = 1*i + 2*j + 3
y = 4*i + 5*j + 6

[jo-mueller]

I think the axes names i and j are a bit arbitrary in the examples and not used consistently throughout the document. Neither are x and y, but they also don't have to be. The only thing that's important (and maybe not explicit enough) is whether these dimensions are discrete or not.

That being said, I think both examples you quote are valid? I don't exactly see what's wrong with the second example, except for the fact that the axes names are used differently? 🤔

[will-moore]

That example has "axes": [{"name": "y"}, {"name": "x"}] so you can't just switch x and y in the matrix.
If you use the spec example then you'd end up with a matrix like this for scale of x: 2 and y:1:

[ 2 0 0 ][ i ]   [ x ]
[ 0 1 0 ][ j ] = [ y ]
[ 0 0 1 ][ 1 ]   [ 1 ]

and that is not correct.
The order of the axes is critical and not arbitrary. The 2 examples I put there are not the same since they give you different results.

I also just noticed that Example 2 under https://ngff-spec-preview.readthedocs.io/en/latest/#affine has axes order:

{ "name": "xyz", "axes": [{"name": "x"}, {"name": "y"}, {"name": "z"}] }

The spec says:

"If there are three spatial axes where two correspond to the image plane ("yx") and images are stacked along the other (anisotropic) axis ("z"), the spatial axes SHOULD be ordered as "zyx"

although this is stated under https://ngff-spec-preview.readthedocs.io/en/latest/#multiscales-metadata instead of under "axes" section where I was looking for it.
I realise this isn't a MUST, but it's nicer if our examples consistently correspond to the SHOULD.

[jo-mueller]

Ah ok, now I see what you mean. The quotes above just show the transformations themselves and not the corresponding coordinateSystems.

The spec says:

"If there are three spatial axes where two correspond to the image plane ("yx") and images are stacked along the other (anisotropic) axis ("z"), the spatial axes SHOULD be ordered as "zyx"

although this is stated under https://ngff-spec-preview.readthedocs.io/en/latest/#multiscales-metadata instead of under "axes" section where I was looking for it.

Uff, that's well hidden. Will change right away 👍

[jo-mueller]

Ok, check again. I have altered the example of the affine2d3d a bit, it now reads like this:

An example with two dimensional inputs and three dimensional outputs. The affine transformation adds a translation by 1 along the new z-axis.

Note that the order of the axes can in general be determined by the application or user. These axes relate to the memory or on-disk order insofar as the last dimension is contiguous when the zarr array is c-order (the default for zarr version 2, and the only option for zarr version 3).

```json
{
  "coordinateSystems": [
    { "name": "ij", "axes": [{"name": "i"}, {"name": "j"}] },
    { "name": "zyx", "axes": [{"name": "z"}, {"name": "y"}, {"name": "x"}] }
  ],
  "coordinateTransformations": [ 
    {
      "type": "affine",
      "affine": [[1, 0, 0], [2, 3, 4], [5, 6, 7]],
      "input": "ij",
      "output": "zyx"
    }
  ]
}
```

defines the function:

```
z = 0*i + 0*j + 1
y = 3*i + 4*j + 2
x = 6*i + 7*j + 5
```
it is equivalent to this matrix-vector multiplication in homogeneous coordinates:

```
[ 1 0 0 ][ 1 ]   [ z ]
[ 2 3 4 ][ i ] = [ y ]
[ 5 6 7 ][ j ]   [ x ]
[ 0 0 1 ]        [ 1 ]
```

where the last row [0 0 1] is omitted in the JSON representation.`

Is there a rule to how projecting into a different coordinate system is done? I think it is a reasonable way of doing it to prepend the new axis to the transformation matrix (in this case [1 0 0]) because we are adding a z coordinate, which should be the leading axis. Is this generally a good way of doing it? Should there be a statement about it? 🤔

[will-moore]

Thanks - both Examples are looking good now 👍

[jo-mueller]

Cool. Mind if I close the issue?