Req for feature to add GPU VRAM and System RAM

[jkfnc]

Can you add two fields to top right, GPU VRAM and System RAM. Dynamically Highlight anymodels that can fit in GPU VRAM to green, DIMM Memory to Yellow and others as red.

[imraf]

This is a great idea, and I also want to include disk space requirements. It's a lot of data to collect so I'm adding to the table as I go and benchmark new models. However, if you have this data you can already clone and edit the .csv file directly! The web page table will update accordingly.

[jkfnc]

Just found that hugging face has this feature already, swear i did not know it before now. You need to select CPU or GPU after adding your hardware, not both at same time. Also its checking againt per model not global. There is no option in search to filter models my machine can run.

[imraf]

This is very useful! I can grab more data points from this. However, it is hard to tell which context length this is referring to (which is also a problem with the data I added myself - I have to go back to my own benchmarks and note down the context length).

[jkfnc]

yes, context length is one the big problems. i am running mistralsmall with 50k context on 3090, with flash attention on, its a lot of trial and error . these calculators https://apxml.com/tools/vram-calculator are saying it wont fit on 24GB , but it can. Also q8 kv cache can increase context length further. so lot of variables.