regarding the consistency

[Aike2071]

Hi, sorry to bother again.
When performing image translation, the non-subject areas along the edges of the image in the A2B direction (particularly the black regions) tend to be filled with other colors, whereas in the B2A direction, the opposite occurs.
I tried adjusting lambda_cycle and lambda_idt, but it’s of little use. I have images with black regions in both domain A and B.what else can I do to maintain consistency in the edges?

[usert5432]

Hi @Aike2071,

That is an interesting problem. I think, the solution depends on whether this color filling is expected behavior or a translation error. Let me explain:

CycleGAN architectures try to find and transform systematic differences between domains. For example, if your domain B images tend to have fewer dark regions compared to domain A (even if this is not immediately obvious to human eyes), the network will try to "correct" this by filling in black regions during translation.

There are two potential approaches to address this:

1. If these are translation errors, then perhaps they can be fixed by increasing the discriminator power:

• Lowering lambda_gp
• Increasing the number of discriminator layers

2. If these are real systematic domain differences the network is trying to correct, then there is no clean solution. One can try to workaround it by reducing the power of the discriminator:

• Reducing the number of discriminator layers
• Removing batch head
• Increasing lambda_gp

There could be other possibilities, but it is hard to tell without seeing how those artifacts look like. If it is possible for you to share an example of these artifacts, perhaps, I could be able to tell which case it is.

[Aike2071]

Hi, usert5432 ,

here is an example of how the color filling looks.On the right is the translated image.The boundary of black regions of two domains has slight difference:
images in the domain A usually present non-horizontally or vertically aligned boundary while that of domain B is mostly parallel to the horizontal edge of the images.
I guess maybe this is the reason for the color filling.In this case, should I increase the discriminator power or reduce it?

[usert5432]

Hi @Aike2071,

Thank you for sharing an example. In my experience, this behavior occurs when there are systematic differences between the datasets.

images in the domain A usually present non-horizontally or vertically aligned boundary while that of domain B is mostly parallel to the horizontal edge of the images.

This sounds like the probable reason. UVCGAN2 uses much stronger discriminator, which it may detect systematic difference in the rotation angle. There are a few ways to fix the problem:

1. Add data augmentations. You can add random-rotation data augmentation to the transform_train configuration option (e.g. { 'name' : 'random-rotation', 'degrees' : 180, },, cf pretraining script). It will randomly rotate images during the training and remove any systematic rotational difference between the domains. This is probably the cleanest solution, but you may need to add different rotation angles to different domains.
2. You can also try to reduce the power of the discriminator in hopes that it will not notice rotational difference. For that, you would need to set "head_config" : "idt" and 'model_args' : { 'shrink_output' : True, }, in the configuration file.

Please, let me know if I should elaborate more on these options.

[Aike2071]

Previously I added random-rotation,degrees:45 .It worked for some samples but the problem persists. I will try with larger angles and perhaps together with reducing discriminator power.
I’m a bit confused about the shrink_output option.Could you please elaborate on what this argument is for? Your reply is appreciated!

[usert5432]

I’m a bit confused about the shrink_output option.Could you please elaborate on what this argument is for? Your reply is appreciated!

That is an unfortunate historical name that we did not revise. In short, when shrink_output is False, the discriminator acts as a feature extractor and returns a feature map of shape (512, 31, 31) (if I remember correctly). This mode of operation is useful when one has a batch head. When shrink_output is True, discriminator acts as a proper discriminator and returns a map of real/fake scores of shape (1, 31, 31).

[Aike2071]

This is clear enough.thanks for your patience!