Experimental results in paper

[VuongLong]

Dear authors, thank you for work,
I run your implementation and got lower results than ones reported in the paper. I have some questions:

1. Your results reported in the paper (average ~85.15) is selected by test-set?

2. I got average results from your implementation:

• selected by validation-set ~ 83.37
• selected by test-set ~ 84.25

3. when I turn off your seed configs and try 5 runs with random seed?

• selected by validation-set ~ 82.63
• selected by test-set ~ 83.81

Thank you for your contribution!

[SirRob1997]

A common problem here, see #12 #13 #2 #5

[Justinhzy]

Hi,
Thanks for your reported issue.

Indeed, closing the seed config will change the results. In experiments, however, this can change results so much that it is hard to select the parameters. So I fix the random seed. I am not sure if fixing random seed is common or not in DG task, but it is common in other tasks. If you want to close the seed configs, please see my updated notice which may be useful for you.

Yes, I reported the results selected by val-set. You can see my training logs in the testing section.

[gshuangchun]

Dear authors, I have the same problem. I have try 3 runs and got the average result 82.50. Besides, when I test the model you released, I got the same results you reported in art, sketch, photo. But my cartoon result is 76.75, and your report is 81.61. The environment and seed are the same of yours.@Justinhzy

[Justinhzy]

@gscahhh
Hi, do you use the exact same environment? if you use a different type of GPU or seed, the performance will definitely change and you need to tune the hyper-parameter a little bit. Please see more details in the updated notice.
I will check the cartoon resnet model.