Unexpected Behavior of FourierAttention

[leventkaracan]

When we bypass the attention mechanism in the FourierCrossAttention class (returning the query unchanged), we obtain the same results. Could there be a reason for this, or have you encountered a similar issue?

[tianzhou2011]

We have never encountered this situation before. It implies that you end up with an identical mapping for attention, which is highly unlikely. Attention is a data-dependent matrix, so even if you achieve an identical mapping for one sample point, you will certainly obtain different values for varying data.

[leventkaracan]

Thank you for your prompt response. We previously tested a new attention mechanism, but it did not significantly impact the results. Subsequently, we replaced return (out, None) with return (q, None) in the FourierCrossAttention class and obtained similar results on the ETTh1 dataset (Multivariate long sequence time-series forecasting, pred_len=96). I will review it again. If this holds, it suggests that the attention mechanism does not contribute significantly to the results.

[tianzhou2011]

We did not observe this in the datasets used in our paper. However, from a design perspective, if there is a strong trend signal, such as a straight line, the attention branch may contribute minimally due to the nature of the STL framework. Just a guess.....