How can I actually put it into use？

[dingjingzhen-bot]

Great job!
However, the biggest problem with this type of work lies in how to integrate it efficiently into existing inference engines, such as sglang or vllm, because the additional overhead they bring and the damage to existing features (paged attention, prefix cache) are hard to avoid, making them very useless. To address this issue, How did you solve it

[Kurt232]

I've implemented mixed attention as a feature in SGLang for RL training, and this experience inspired a solution to address this issue: for head-reallocation methods like ours (RLKV), we can maintain two separate KV cache pools in SGLang—one for full KV cache heads and one for local KV cache heads, since the critical head distribution is static.

This solution can improve the throughput of KV cache compression methods via SGLang and make our method more practical. It's essentially an engineering question rather than a research question.

In my current evaluation code, I implement inference via Transformers instead of modern efficient inference frameworks. This is because I'm currently focused on tensor parallel training in RL to discover reasoning-head distributions for larger reasoning models (e.g., 14B+).

I commit to implementing the two-cache pool feature and submitting a pull request to the official SGLang repo to improve throughput as soon as possible. I'm also open to collaborating with anyone interested in developing and integrating this feature in SGLang.

I appreciate your interest in my work!