Question: would this work on RPIs i.e. ARM CPUs?

[stevef1uk]

Hi,

It is possible to run some LLMs on the new RPIs with 8GB RAM, but it would be nice to try a larger model running across a number of RPIs as some of us have quite a few of them available to us.

It would be nice if the tool supported this LLaMA repo: https://github.com/ggerganov/llama.cpp/tree/gg/phi-2

Regards

Steve

[X-rayLaser]

Hello.

In short, maybe. But, it won't do much good because you won't be able to run newer LLM models that have architecture different than LLama v1. If you haven't checked it out yet, there is an excellent project called petals that you should try instead: https://github.com/bigscience-workshop/petals.

Longer version.
I haven't tested this use case so I can't tell for sure. This repository actually builds on top of llama.cpp project. It uses llama.cpp repository as a git submodule vendor/llama.cpp. But, If you look at the llama.cpp folder under the vendor directory, it refers to very old checkout of LLama.cpp repository with last commit made around July 2023. Therefore, GGUF format is not supported here. Same applies for newer better quantization algorithms as well as other improvements that are now available in original LLama.cpp repo but missing here.

Also, only the most basic features are supported here. You can convert models to GGML format, quantize them and generate text with them, control generation temperature and repeat penalty, but not much more.

[stevef1uk]

Th====

Hello.

In short, maybe. But, it won't do much good because you won't be able to run newer LLM models that have architecture different than LLama v1. If you haven't checked it out yet, there is an excellent project called petals that you should try instead: https://github.com/bigscience-workshop/petals.

Longer version. I haven't tested this use case so I can't tell for sure. This repository actually builds on top of llama.cpp project. It uses llama.cpp repository as a git submodule vendor/llama.cpp. But, If you look at the llama.cpp folder under the vendor directory, it refers to very old checkout of LLama.cpp repository with last commit made around July 2023. Therefore, GGUF format is not supported here. Same applies for newer better quantization algorithms as well as other improvements that are now available in original LLama.cpp repo but missing here.

Also, only the most basic features are supported here. You can convert models to GGML format, quantize them and generate text with them, control generation temperature and repeat penalty, but not much more.

Thanks for a detailed response. I had found Petals, but the instructions did not look beginner friendly.