Am I using MAGE properly?

[itayreznik]

Hi,
I have an optimization algorithm that allows generating images that are controversial (that look like 2 classes to a human). I do that with MAGE. I cloned rcg to my working directory and I also use 2 robust classifiers trained adversarially on Imagenet.
I do that by "training" the MAGE input vector's entries - optimizing the input vectors in each iteration. The loss function for this optimization contains one classifier's output probability for dog and the other classifier's output probability for cat (lets say). I want to maximize both of these probabilities.
I use MAGE purely for generating the image in each iteration of the algorithm. It works well, the images are generated and I save them in the target directory.
The thing is, I wonder if maybe I am not using it exactly like I should. Because the images generated in each iteration do not converge to images with high probabilities for the 2 classes. But I did achieve this objective when I was using a regular self conditioned gan, and not MAGE.

This is the relevant code. More specifically, lines 75,82,99.

Thank you

[LTH14]

I think you might need to give MAGE the class you want it to generate.

[itayreznik]

@LTH14 How should I do that? Mind that my objective is to generate an image that is controversial, meaning that it looks like 2 classes. I do that buy maximizing the probability of one class in one model's output and the probability of the second class in the second model's output. That way, the representation vector the is the input to MAGE is being optimized.

[LTH14]

I see -- now I understand what you want to do. However, MAGE's output is a discrete VQ token -- how do you backpropagate the gradient through that part?

[itayreznik]

I don't. I use mage only for image generation.
I have an evolutionary algorithm that has a population of representation vectors, that are the inputs to MAGE. In each generation the new population of vectors is better and better (for the purpose I described of achieving controversial images). So, I just use mage (and previously was using other more basic GANs) just for inference, just for generating the image in each iteration of the process.
The flow does work and I get generated images from MAGE, I just wanted to make sure I'm not missing something specifically about the generation of the image.

[LTH14]

Got it -- if you only generate images from MAGE I think the current flow looks correct.