Why do NUDGE (top-1 ref.) and (with E.S.) perform differently in Figure 3 GetOut?

[Jaraxxus-Me]

Hi,
Thanks for the great work, NUDGE.
There is one thing making me confused. I see from the code that in the GetOut environment, the beam searched top 1 clauses are the same as those provided by humans (I guess it is human_assisted). This means the two models have the same rules (C), but their performance in your paper Figure 3 GetOut seems different.
Why is this? Please correct me if I have the wrong understanding.
By the way, if we only search the top-1 ref., does that mean the weight matrix has only little influence? i.e., the performance would be similar if we provide a fixed identity matrix instead of training a W. I may have a wrong understanding.

Thank you very much.

[k4ntz]

Hi there, sorry for (very) delayed answer. It might be that there are errors in the final clauses files, I have not verified this.
Two models with the same rules can still have different weights and thus not use the rules in the same order.

"Top-1 ref." means that the refinement is done with the selection of only the top 1 best rule at each refinement step. This is done within beam search and thus has no link with the weight matrix (that is trained after the rule search).