How to embedding the transformation into deep neural layers

[ShuchaoPangUNSW]

Hi Koen,

   I have read your some papers about bspline transformation for medical image analysis. However, I'm still not sure how to change the bspline transformation operations into a neural network layer for training together. Can you show me how to do that? I sent emails to your student address, but it failed. I think you changed your email address.

Thanks a lot.

Yours sincerely,
Shuchao

[keppenhof]

Hi Shuchao,

The Gryds transformations are not suitable for use inside neural networks because they are based on numpy and scipy code. They cannot be used in backpropagation during training.

If you want to have a deformable transformation inside your network that can be trained, I suggest looking at spatial transformer networks for the deep learning framework of your choice:

• a TensorFlow example: https://towardsdatascience.com/implementing-spatial-transformer-network-stn-in-tensorflow-bf0dc5055cd5
• a PyTorch example: https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html

I hope this helps you a bit further.
Best regards,

Koen

[ShuchaoPangUNSW]

Hi Koen, Thank you very much for your kind reply. Yes, you’re right. The Gryds transformation version is prepared to augment data for deep learning. If I want to use bspline transformation for building a new network, as you said I have to write a bspline transformation layer. Actually, I had built a model based on STN, however, as you know, the STN is a affine transformation, not a bspline transformation. So, the results said it is only a global transformation. So, If I want to have a local transformation, I have to build a new STN layer using like bspline transformation. Based on your great work, i.e., Fast contour propagation for MR-guided prostate radiotherapy using convolutional neural networks, I found you exploited this bspline transformation based layer and trained it together with other CNN layers, so I send you this email to ask for your experience. Can you help with it? Thanks a lot. Yours sincerely, Shuchao From: Koen ***@***.***> Sent: Monday, 5 July 2021 3:54 PM To: tueimage/gryds ***@***.***> Cc: Shuchao Pang ***@***.***>; Author ***@***.***> Subject: Re: [tueimage/gryds] How to embedding the transformation into deep neural layers (#5) Hi Shuchao, The Gryds transformations are not suitable for use inside neural networks because they are based on numpy and scipy code. They cannot be used in backpropagation during training. If you want to have a deformable transformation inside your network that can be trained, I suggest looking at spatial transformer networks for the deep learning framework of your choice: * a TensorFlow example: https://towardsdatascience.com/implementing-spatial-transformer-network-stn-in-tensorflow-bf0dc5055cd5 * a PyTorch example: https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html I hope this helps you a bit further. Best regards, Koen — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#5 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AR4S7XMBGZNLMOEAQTSWBOTTWFCJDANCNFSM47ZYTHAQ>.

[keppenhof]

In the paper you mentioned we actually did not use a B-spline transformation inside the network, we used a dense deformation field of the same dimensions as the image. The network in the paper outputs that vector field as an N_dim x N_x x N_y tensor (the green block in figure 2) and then uses that to reinterpolate the image. Sidenote: the network is trained on B-spline transformations but no B-spline transform is done inside the network.

Maybe a 2D example to make it more clear:

Let's say we have an image I(x) of shape N_x x N_y and a vector field u(x) with a vector for every pixel in that image, i.e. something that can be represented by a 2 x N_x x N_y tensor. You can write an interpolation layer that reinterpolates the image based on the vector field, i.e. it makes a new (transformed) image J(x) = I(x + u(x)).

For the paper we used Theano (which is no longer supported), but for instance PyTorch has a function for this: torchvision.transforms.functional_tensor._apply_grid_transform(image, vector_field). I'm sure something similar can be done in TensorFlow.