Skip to content

vLLM support #30

@coleramos425

Description

@coleramos425

We have a research team asking for vLLM support with Omniprobe, which is more complicated than it sounds because this support has a slew of Triton implications.

Based on the vLLM installation docs, install requires Triton flash attention for ROCm which is tied to a non-default (triton-mlir) triton branch. Given that supporting vLLM means changing our Triton version, and thus our LLVM version, I'm thinking may be a non-trivial upgrade.

Open to thoughts if anyone has any creative solutions on how to circumvent this...

CC: @rwvo @keithloweryamd

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions