This is a small application that runs a phi-3-vision-128k-instruct model with onnxruntime-genai-cuda.
The model should read a table from a screenshot and output it as Markdown.

Licensing

This project is distributed under the MIT License. See LICENSE for details.

However, the provided Dockerfile is based on an NVIDIA Docker image, which is distributed under its own license. By using the Docker image, you agree to comply with both the MIT license for this project and the NVIDIA license. For details on the NVIDIA license, see NVIDIA license.

Tested Environment

1. Windows 11
2. Docker version 27.3.1, build ce12230
3. DELL XPS 15 9530 with 64 GB RAM and NVIDIA GeForce RTX 4060

nvidia-smi.exe

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 561.03                 Driver Version: 561.03         CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4060 ...  WDDM  |   00000000:01:00.0 Off |                  N/A |
| N/A   54C    P3             10W /   42W |       0MiB /   8188MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

docker run -it --rm --gpus all nvidia/cuda:12.6.3-cudnn-runtime-ubuntu24.04 nvidia-smi

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.41                 Driver Version: 561.03         CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|

Setup to get started:

• Build the image with docker.build.ps1
• Run the container with docker.run.ps1

When you start the container for the first time, the model weights are downloaded from huggingface.
The weights are stored locally on your host machine via a mapped volume.