Running CAAL locally on Mobile or WatchOS yet?👀

[MaxSikorski]

After seeing the most recent video on the YouTube channel, Your LLM Doesn't Need Your Keys, I made a comment about running smaller models that are specifically local on edge devices like phones, watches, and of course even glasses. Has anyone played around with any of this local fine-tuning of very small models? <1B parameter models that could be easily ran on any iPhone or Pixel device or even a Apple watch.

I believe this is the future and I'm very interested if anybody else has done this. I have yet to try this out, by the way. I'll get around to trying this.

[cmac86]

@MaxSikorski, welcome!

Thanks for bringing the discussion to the GH. I haven't tried training very small models yet, but its on my radar and dev path. I think a CAAL Lite for on-device use is in the roadmap. It might be more basic and less conversational, but could work for basic needs. Then if needed maybe route back to your server running CAAL with a bigger model.

[MaxSikorski]

Glad to be here; I do like your project and need to spin up my own instance asap.
Yes, I think having some type of a small refined model on edge devices, doing everything locally will be fantastic. Keep it simple and refined. Right to the point doing local on device tasks. But we could have in the settings a toggle to say, talk back to your main server, via say, something like tail scale. And of course that would take care of some of more of the heavy workloads.

[cmac86]

CAAL is set up to support Tailscale already. So you could use it on your android device with the APK or just through mobile Chrome browser. No on device LLM yet. Toggle is a good idea as a starting point to toggle between on-device model or server model. Then next step would be a way of determining this automatically depending on the request.

[MaxSikorski]

Got it. This is great.
I could be wrong here, but I think the next step would be to begin refining a smaller model to be able to run it on any edge device. Or make a simple mobile APK. And then could easily port that over to iOS at some point. Which would be best to start first? Or is there something even more foundational and a better prerequisite to get started on?
• Run small on device local LLM
• .apk or iOS app to connect back to home server
• Or something else more foundational and a better prerequisite?

I'd like to contribute again to your project as I've started this discussion, but also I'd like to incorporate this into my own custom AI agent.

[cmac86]

.apk exists. You can download it here: https://github.com/CoreWorxLab/CAAL/releases/download/v1.4.0/CAAL-v1.4.0.apk

It’s a flutter app, so I just need to build it for iOS as well. It’s a couple versions behind as well, I need to rebuild the latest .apk

It connects back to home server. This is what nerdbeast is running on the Echo Show, you can see in the other discussion or in my last video.

I think the next step for edge device though is seeing what a <1B model is capable of. Install CAAL, pull some <1b Ollama models, test with some simple tools like weather and espn. Might need prompt adjustments. This testing can be done without mobile app though. Just through CAAL web frontend.

If this works then fine-tuning the model for the CAAL tool convention would be next along with training on tool chaining. If that works then a on-device app with specific on-device tools is possible.

[MaxSikorski]

Sounds good. Yes, I did see what Nerdbeast was running on the Echo Show and I thought that was interesting.
So I'll look at some <1B models and start playing around with that and I'll keep in the back of my mind as well to convert that flutter app to build a native iOS app as well.

Great discussion and quick summary with you today. This was great. Thank you!