Question about token-wise cache in attention

[ywlq]

非常棒的工作！！
但是我有一些问题想问
Q1 : 我看hunyuan-clusca中对于attention部分是直接用的之前缓存的token，为什么不对attention也选择一些token进行更新呢？这样的话不就变成了只对mlp部分的token进行重新计算么
Q2：如果我要对attention也选择一些token进行重新计算的话，那么就必须完整计算所有的token才能获取对应需要的重新计算的token，这个开销要怎么处理呢？
如果有理解错了，请您指出，感谢 :)
@Shenyi-Z

[Shenyi-Z]

非常棒的工作！！ 但是我有一些问题想问 Q1 : 我看hunyuan-clusca中对于attention部分是直接用的之前缓存的token，为什么不对attention也选择一些token进行更新呢？这样的话不就变成了只对mlp部分的token进行重新计算么 Q2：如果我要对attention也选择一些token进行重新计算的话，那么就必须完整计算所有的token才能获取对应需要的重新计算的token，这个开销要怎么处理呢？ 如果有理解错了，请您指出，感谢 :) @Shenyi-Z

In the appendix of our earlier work ToCa (ICLR 2025), we discussed this issue. At that time, the conclusion was that allocating part of the computation to attention had a lower cost-effectiveness compared to allocating it to the MLP. Based on our subsequent experiments, we can further strengthen this conclusion: performing partial token computation and updates in attention will ultimately result in worse performance than fully retrieving attention outputs from the cache.

This can be understood as follows: the attention operation, unlike the MLP, does not process different tokens independently, so information from different tokens inevitably influences one another. When only partial tokens are passed into the attention operation, the total token length in attention becomes inconsistent with the model’s training conditions, and the distribution of the attention map becomes one that the model has not seen during training, leading to poorer performance.

[ywlq]

@Shenyi-Z
这是否意味着，无论是对应self-attention还是对于cross-attention，最好还是直接使用cache的token
只用在MLP部分执行token-wise的更新即可