[Proposal] refactor core for extensible router architecture

[Xunzhuo]

This PR added proposal: refactor core for extensible router architecture

[bhks]

Really great proposal !! Could not have been started by other person than you and the team.

Given your experience in AI gateway world and envoy,

1. I am wondering what prompted us to build this ?
2. From proxy perspective I believe prefix-aware and design of prefill+decode aware routing can be 2 differentiator to build this router.
3. I am also wondering how would this be different from [envoy + K8s EPP ] Implementation ? llm-d project along with other k8s sigs like lws has been making quite some progress. I guess this gateway can be coupled with those as plugin.
4. Would we be compatible with semantic router and how do we see these 2 being integrated ?
5. Would the router be infrastructure agnostic like k8s vs bare metal vs slurm ?

Thanks again, I am looking forward to this project and would love to contribute as well.