[Feature Request] vLLM / High-Performance Inference Engine Support

[Ki6an]

hey! thanks for sharing the paper and the approach.

using this in production with the current huggingface-only inference stack is a bit tricky since it slows things down. it would be great to apply this technique with vllm / sglang instead.

the main challenge is that the per-step logit fusion (logaddexp before sampling) does not map cleanly onto vllm’s standard model or speculative decoding patterns.

do you have any plans to support vllm or similar engines, or any recommendations to make this faster? happy to help contribute if you already have a direction in mind.

[mxmcao]

Hi! Thank you for your suggestion. Yes, we do have a rather not too urgent goal of implementing the interpolation method on the lmdeploy framework (which is a simliar engine developed by Shanghai AI Lab).

Personally, I am not an expert on the detailed implementation of vllm or similar framework. However, I will look into this matter and confirm the workload of this adaptation (It would be great if you can share some tutorials for beginners to tweak the vllm engine). If it's not too difficult to implement, I will open a pr on vllm for supporting the interpolation of two models (which could be a general demand).

Will be giving you an update after the NeurIPS conference.

[Ki6an]

thanks for the reply and context.

vllm custom logits processors (this seems like the right hook to do per-step logaddexp without forking):
https://docs.vllm.ai/en/stable/features/custom_logitsprocs/

and the design note on how they are wired into the sampler:
https://docs.vllm.ai/en/latest/design/logits_processors/

looking forward to teh lmdeploy implementations, thanks.