The Hated One
patreon
How to run AI locally and 100% privately (collab with Naomi Brockwell)
🕑 Added 2024-07-23 15:00:07 +0000 UTC[This is a tutorial on how to install AI locally on your device; it's a segment I made for a video collab with Naomi Brockwell on AI privacy which will be released in the future].
Running AI locally is the most certain way to protect your privacy. To get started, you’ll only need to follow a few steps. My goal is for you to be empowered so I am gonna guide you through the process and will be holding your hand along the way. Don’t worry, this will only take a few minutes, but the payoff is totally worth it.
In its basics, running AI locally is only about two things.
Find a place where you download your favorite models from
Find a program you’ll use to interact with this model.
Think about this no differently from when you want to watch a movie. First, you find a place to the download the movie from, then you use a program to view the file so that you can watch it on your device. So take this as a form of encouragement that you can absolutely do this yourself.
Ollama
There is one quick solution that does both of these in a single shot and that is Ollama. Ollama, as this is the quickest and easiest setup and it’s available on Mac, Windows and Linux. And if you stick till the end, we’ll take it to another level where you’ll be able to do some really amazing stuff with your computer.
Step 1
Now I want you to head to https://ollama.com/ and navigate to the “Download” page. From there, select an option for your system. For Windows and Mac, this is a matter of downloading a single file. For Linux, you’ll need to run a simple command.
Once you’ve installed Ollama on your computer, congratulations! You’ve accomplished the first step as you now have an interface you can use to download and run models from. To test whether Ollama is successfully installed, open the terminal or Command Prompt on your computer and simply type “ollama” and hit enter.
In the output, you’ll see the list of available commands. The ones you’ll be using the most are “pull” or “run” commands. You’ll be using these to download and run models from Ollama’s repository to your computer.
Anytime you want Ollama to do something, you’ll type “ollama”, hit space, then type in the desired command and then hit enter.
This is all you need to know to start running AI locally. So let’s move on the second step.
Step 2
To find a model you’d like to run on your computer, head back to ollama.com and navigate to the “Models” page.
This is Ollama’s official library of many, many AI models all of which can run locally.
There is a lot of models here so how do you choose one? I am gonna teach you something cool so that you can learn to navigate these waters on your own.
With any of these models, the most relevant information for you is the model size. This is represented in billions of parameters and usually looks like this.
Llama3, as an example, has two different sizes for its model – 8b parameters and 70b paramaters. Usually, single digit-parameter models are gonna run fine on a consumer-grade laptop. The higher the number, the beefier the machine it’s gonna take. In this case, the 8b-parameter model is gonna run fine on a dedicated GPU. The 70b model will most likely need 64 gigs of RAM and a dual-GPU setup.
With other models, you also have ultra-small options, and these shouldn’t have a problem running on any machine.
When you open up a model’s page, you’ll also get a bunch of additional info about who created it and what the model is best suited for. This is what can help you determine whether the model is what you want for your use case.
So if we were to choose let’s say Aya 23, you want to select the 8b model and then copy the command ollama run aya:8b into your terminal. Hit enter to run the command and let it do its job. It’s gonna pull the model from the website, and when it’s ready, you’ll be able to chat with it. Inside the terminal.
Isn’t that neat? This setup is excellent for basic chatting, brainstorming and testing the models you downloaded this way. Anytime you want to chat with your model again, just type in “ollama run [name of the model] again.
But we can take it to another level. The biggest advantage of open source models is that you can fine-tune them using your own instructions while keeping all data private and confidential. I promised I was gonna do this for you so here we go.
To do this, we’re gonna need another tool. It’s called OpenWebUI.
OpenWebUI
OpenWebUI is a mature user-friendly interface that will allow you to communicate and tinker with local models in a very powerful way. You definitely want this tool if you want to tweak your models, create custom characters or feed the models your confidential documents. In this tutorial, I’ll only go over how to set this up for the first time. But you will learn some really cool things and you’ll take full ownership of AI technology.
Step 1
Since we already have Ollama installed and ready, all we need is a program called Docker. Docker will allow us to create a local host through which we’ll use OpenWebUI and communicate with our local AIs.
To get docker, head to the official website and navigate to the products page and Docker Desktop https://www.docker.com/products/docker-desktop/.
Download Docker for your machine. For Windows and Mac, it’s a single file that will install Docker on your device. For Linux users, you’re gonna have to follow specific instructions for your distribution.
Step 2
From there, we just follow the installation setup step by step until Docker is fully installed on our computer.
With Docker and Ollama both installed on your machine, copy the relevant command from OpenWebUI quick start guide. https://docs.openwebui.com/#quick-start-with-docker--recommended
For our case, it’s gonna be the command for “If Ollama is on your computer”. We paste this command into the terminal and run it. That should create a container inside of Docker. When you open up Docker, click on the “port 3000:8080”, which should open your default web browser with OpenWebUI running in a new window.
Feel free to install this as a progressive web app for easier access. On the first boot, you might need to create an admin account. This is a completely local account so you don’t have to worry about any data leaking.
Once you have OpenWebUI running, go to “Select a model” and you should see every AI you downloaded through Ollama. You can download and run any number of models you want provided your device has the resources and available storage space.
Remember that anytime you want to run OpenWebUI, you’ll need to have Docker and Ollama both running as well.
End
And that’s it. You are now a pro at running fully private open source AI models on your own computer. Congratulate yourself because you are now empowered to take full ownership of artificial intelligence without prying eyes stealing your data.
Comments
The Hated One
Glad you found it useful!
Ajay
Very useful guide, I did not know this was possible. Just spun up the containers on my server to access via UI and selected the gemma model. Initial results are exciting! Thank you!
The Hated One
Hey, that's great to hear! Thanks for that. Hope it was useful, if only a little.
Fan Chess
I used your tutorial and now I have private AI running. There are more sources on the net but somehow never got the chance to used them. Now I'm trying to bring it to a Ubuntu machine. I just wanted to thank you: Thanks.
