CumConcatLayer

[albertz]

This is for #391.

[albertz]

Implementing a test case is a bit tricky here.

I cannot use DotLayer at the moment to reduce the new dim tag (see this comment).

I also cannot really use any other layer to reduce the new dim tag, as they all would not correctly handle the extended dynamic size. Also, operations using the extended dynamic size would usually also expect all axes from the ext dyn size to be present (rec_time_dim), which is not the case inside the loop. Only DotLayer would introduce this as outlined in #391.

I also currently can not return the CumConcatLayer directly as output inside the rec layer because the rec layer logic expects that the output template inside the rec layer will be the final output with additional rec_time_dim. But this is not true for the CumConcatLayer where rec_time_dim is not added (except in the ext dyn size).

[Zettelkasten]

Looks good! I only had minor comments.
Like you suggested, I think we can merge this then and go from there. EDIT: After we fix the other issues..

[albertz]

Maybe I will directly implement all needed things in this PR here (as minimal as possible) such that the test can run. I.e. also:

• SoftmaxOverSpatialLayer with ext dyn size support (via Data get_sequence_mask_broadcast fix for uncommon cases #646) (Edit Done)
• DotLayer should check for dyn sizes on reduced axes and apply masking (and error if axes are missing) (and skip masking via some automatic logic when possible) (DotLayer reduce over dynamic axis should respect the seq mask #629) (Edit Done)
• DotLayer logic for adapting axes when optimized out (DotLayer behavior unintuitive with rec automatic optimization #569) (Edit Done)

[albertz]

I wonder a bit how to best handle such huge PRs (e.g. how to effectively do the cleanup).
I wrote that down here on Reddit as a question but not sure if that is the best place to ask such things. But I also don't really know any good place where I could ask such question. Any recommendation where to ask such things?

[albertz]

I will probably now first move everything out into a new branch, except the two new test cases and also the CumConcatLayer (which is what this PR here is about + further things we need for the test cases to pass). Because all the remaining things should always pass all the tests.

[albertz]

Wow, this was quite some effort, to get the basics ready, to continue working on this itself.

Basically:
#592 #599 #600 #601 #602 #603 #604 #605 #606 #607 #608 #609 #610 #611 #612 #613 #614 #615 #616 #617 #618 #619 #620 #621 #622 #623 #624

[albertz]

The test case is a full simple test case for auto-regressive self attention, so it covers #391. CumConcatLayer is the final missing piece now, so this finally fixes #391 (at least the basic support).