Question about offline calibration

@andy-yang-1 My team really appreciate your work on Double Sparsity! Thank you for everything you do!

Question - In group_channel_config.py, the scores for identifying outlier channels are decided via q * k, which is the dot product of an individual token's query and key vectors.

However, it is possible that an outlier (or feature detection) occurs via channel C' during the interaction of query vector of token T and the key vector of let's say token T-3 (and this channel C' does not show strong outlier behavior when multiplied within key-value vectors of either token T or T-3)

I am curious about how the offline calibration approach works, if only interactions between query and key vectors of the same token is considered to measure strength of outlier channels. Thanks!

[andy-yang-1]

@vnkc1 Thank you very much for your reply! I think what you said makes sense. In my experiments, I only verified the effect of q * k for the same token, but it’s very likely that cross attention might yield better results. I will try the method you suggested.