[Inference Tasks] Plans for language model inference

[ChristianWeyer]

Great project!

Are you also thinking about an inference engine for SLMs?

[tjwald]

@ChristianWeyer Are you talking about running Qwen3 and the like?
For generation task, there are existing libraries that are very performant like vLLM, Ollama etc . Also, most of the optimization for this task is smart context generation and kv caching, which aren't something I plan to tackle anytime soon.
For other tasks, then the library does care what model you plug in, as long as you pass and get the correct inputs and outputs from the model

[ChristianWeyer]

Yeah, exactly. Was thinking about running Qwen3 et. al.

The solutions you mention above are all not-.NET-native 😉. Just as Transformers is not.

I thought one of the base ideas of your lib is to have a highly performant .NET-native solution. That's why I am asking ☺️.

[tjwald]

@ChristianWeyer if I see in the future a way to improve on vLLM or Ollama then maybe, but I will be tackling other ML tasks, and leave Text Generation for last.

[tjwald]

After reviewing some other ML tasks in the ecosystem, I will try to add support for running SLM in one way or another.
The idea is that you can use SLMs as a one part of a multi model flow, and that is the exact reason my library exists.

The idea will be to enable support via 3 executors:

• Onnx Runtime - manual generation loop.
• LlamaSharp - delegate most of the work to llama.cpp. This only supports (out of the box) specific models.
• OnnxRuntimeGenAI - use the dedicated package for this. However, this only supports specific models

This requires a lot more work and creating a new abstraction vertical which is orthogonal to the existing abstractions.
Will also require implementing an Encoder inference task support