Private LLM - Local AI Chatbot 17+

Pretty great, but it has some limitations

The implementation here is great, especially now that the devs have added support for several models! Works best with devices with a lot of RAM - I have an iPhone 15 Pro, which seems to be enough, but my old 12 didn't really handle it well. I just wish you could save threads like in the ChatGPT app (and maybe give specific system instructions depending on the thread?). Otherwise, pretty good and worth the price!

Also, if we could save specific temperature and top-P presets with models and maybe be able to directly type values in, that would be nice.

So much potential

I've been using this app for a couple of months at this point. When it was first released, it was a neat proof of concept, but it only supported 7B models and was overall just too simple to use effectively. With a recent update, a 13B model was released for all macs with 16GB of memory, and it makes such a huge difference! It's not quite at the same level as ChatGPT 3.5T, but it's close enough that I never use 3.5T anymore; this is my go-to. I greatly appreciate the on-device processing (hallelujah privacy!), and it doesn't even use too much power - my battery still lasts for hours and hours. The performance is also great; my base M1 Air powers right through the prompts.

I only have two complaints about the app at this point. 1) The 13B model uses about 12GB of memory by itself, which does force the use of swap on a 16GB Air. Not much the dev can do about this, but it is something to keep in mind. You'll want to close out of other programs before launching this. 2) There still is no feature that has separate conversations. If you want to start a new conversation, you need to delete the existing conversation. I'd love it if we could get separate conversations in a future update; it would make this app so much easier to use.

Overall, I love it and do not regret buying it at all. I can't wait to see what future updates bring :)

A great tool, with huge potential.

Obviously being an offline GPT limits the responses, but the fundamentals for something impressive in future less memory restricted devices eg iPhone 15 and beyond is really exciting. Offline and privacy focused are essential qualities of this app and it works quite well. A good bargain for the price with an active developer. One recommendation/piece of feedback would be to provide information on what the settings TOP-P and temperature do. If we come across an error or weird, reproducible response could you provide an option to email you a log so we can forward you the issue easily? This way you don’t need to embed any privacy violating analytics packages.

Developer Response ,

Thanks for your feedback! Indeed we're hoping the upcoming iPhone 15 series of devices will have more memory, so we can ship bigger and smarter models for newer devices (As free updates, of course!). We're in the process of adding an FAQ on our website to answer the temperature, top-p question. We've already received the question a few times by now on our support email and on discord. Perhaps I should also add a help section within the app. Thanks for the suggesting the email log idea, I'll add it to our backlog. We specifically don't embed any analytics packages in the app, that would be antithetical to the concept of privacy, which is one of our app's USPs. The only logs we get are crash-logs from iOS's off by default, and opt-in Analytics feature (Settings -> Privacy & Security -> Analytics & Improvements -> Share With App Developers). Also, responses from the LLM in the app are (intentionally) stochastic, and are often hard to reproduce.