is there an extra surgery image input in the prediction phase?

[qihanzhang-zach]

can i ask you 2 questions?

1. in the train phase, as we know, there is a surgery image as input(x0); then we add noises on it, and get xT ; finally we denoise xT, and get x_0_pred; i wanna know if the x0 and x0_pred(or its corresponding ground truth image) are the same image?
2. in the prediction phase, is there an extra surgery image input, or just generate image out of text(surgery phase and tool text)?

[YFrisch]

Hello!

1. in the train phase, as we know, there is a surgery image as input(x0); then we add noises on it, and get xT ; finally we denoise xT, and get x_0_pred;     i wanna know if the x0 and x0_pred(or its corresponding ground truth image) are the same image?

During training, the denoising UNet is trained to predict the noise eps_t between x_t and x_t-1 (following the original DDPM paper). However, there are alternatives for training the model, such as directly predicting x_t-1 or x_0, which we did not explore here.

2. in the prediction phase, is there an extra surgery image input, or just generate image out of text(surgery phase and tool text)?

In this repository, we are conditioning the denoising process on binary label vectors, representing the surgical phase and the instruments used.

I hope this answers your questions. Let me know if you have more!