40 GB model

[DHOFM]

Hi,
thanks for your nice repo. You mention 2 3090:
`

The following hardware is needed to run different models in MiniLLM:

Model	GPU Memory Requirements	Compatible GPUs
llama-7b-4bit	6GB	RTX 2060, 3050, 3060
llama-13b-4bit	10GB	GTX 1080, RTX 2060, 3060, 3080
llama-30b-4bit	20GB	RTX 3080, A5000, 3090, 4090, V100
llama-65b-4bit	40GB	A100, 2x3090, 2x4090, A40, A6000

`

So when I try the 60B Version with 2 RTX 3090 I get an OOM - how can I use both GPUs?

Kind regards,

Dirk

[kuleshov]

Ah, sorry, I haven't implemented that feature yet :) happy to merge a PR if someone does that first

[kuleshov]

for what it's worth, i was running the 65B model on one a6000