Attention and control

[AI-ELka]

No description provided.

[josephdviviano]

can you resolve merge conflicts?

[AI-ELka]

can you resolve merge conflicts?

I noticed that Mehran's attention code was merged into main (that's where the conflict came from), so I guess I can keep this attention implementation just in case we need it. I was coded it because I wasn't able to solve a problem in the other attention implementation.

[orichardson]

@AI-ELka Can you help us determine whether or not that problem persists in the implementation that's currently on the main branch? I've forgotten the details of the issue.

[AI-ELka]

@AI-ELka Can you help us determine whether or not that problem persists in the implementation that's currently on the main branch? I've forgotten the details of the issue.

The main problem we had was that the loss remained constant (in a case where it should decrease), but after testing now with the code, this problem seems to be gone.
One thing that pops up now when testing with "uniform" (and not with "from_cpd" or "random") is an assertion error coming from:

print(f"Any unfrozen edge changed? {any_changed}")
assert any_changed, "No learnable edges changed; attention/control masks may be misapplied."

So the main problem we had seems to have been solved, but we now have this issue with the assertion error when using uniform initialization.