If your favourite sites, feeds and channels have been overrun with acronyms like AI, ML, GPT and LLM, and you’re wondering what all the fuss is about, you’ll get a taste of it here.

It seems like almost as soon as people realised ChatGPT was a thing, it’s become a household name. That’s because aside from the memes and the hard questions concerning its true costs (financial, environmental, legal and ethical), it’s become a useful tool for all sorts of people: from helping kids prepare for exams to generating ideas for content creators and assisting judges with cases in law.

There’s no denying the value (and controversy) ChatGPT has brought with it since it was released to the public for free. But there are a few, arguably more important things we’ve been missing: privacy, decentralisation and full control of our interactions.

Chats lacking privacy

While it’s great that ChatGPT is free to use and OpenAI evidently takes its security compliance seriously, it’s important for us to consider the privacy implications of using the free services. As OpenAI states,

Data submitted through non-API consumer services ChatGPT or DALL·E may be used to improve our models.

Call me a cynic, but I tend to think that when a company states they “may” use your data, they almost certainly will.

Luckily, for those of us who want to ensure we have full control and ownership of our AI chats, there’s Ollama: a free and open source tool for running various large language models (LLMs) locally on your own hardware.

Keeping it local with Ollama

Before we dive into setting up Ollama, a brief note on system requirements.

Ollama can be installed on Linux, macOS and Windows machines, and while you don’t need a GPU, it’s definitely recommended.

Once we start the Ollama service, it will look for a GPU for hardware acceleration. If it can’t find one, it will rely fully on the CPU. This won’t stop you trying out Ollama, but ideally you want to run these types of LLM services with the addition of a GPU. You’ll really notice the difference when it comes to response times and overall performance.

Installation

We’re opting for the Linux installation instructions. The following steps were carried out on a Debian 12 machine, but they should be transferable to most Linux distributions. We’ll fetch the Ollama binary directly from source rather than from a package manager.

While setting up the systemd service is specific to distros running systemd as their init system, this includes most distros, and the service isn’t a prerequisite to using Ollama anyway.

If you’re new to this sort of thing, you may be tempted to run the ‘one-liner’ found in the instructions, but try to avoid this if you can. It’s best practice to take a little more time and verify as we go.

Ollama binary

Head to the project’s releases page and download the Ollama binary and checksums. (Ensure you’re using the latest release version, which may not be the one shown below.)

curl -L https://github.com/ollama/ollama/releases/download/v0.1.30/ollama-linux-amd64 -o ollama-linux-amd64
curl -L https://github.com/ollama/ollama/releases/download/v0.1.30/sha256sum.txt -o sha256sum.txt

Verify the binary, scan it with your preferred AV program, then make it executable.

grep -E "ollama-linux-amd64$" sha256sum.txt | sha256sum -c
clamdscan ollama-linux-amd64
sudo chmod +x ollama-linux-amd64
sudo mv ollama-linux-amd64 /usr/bin/ollama

Ollama user

It’s always best to run services like these under a dedicated ‘system user’. We’ll create the user’s home directory at /usr/share/ollama. The user shouldn’t need a shell, so we’re going to say it can’t have one (/bin/false).

sudo useradd -r -s /bin/false -m /usr/share/ollama ollama

Systemd service

Create and enable the systemd service so that the Ollama server runs in the background after boot. Then we can simply run the Ollama binary whenever we like without having to restart the server every time.

sudo vim /etc/systemd/system/ollama.service

/etc/systemd/system/ollama.service:

[Unit]
Description = Ollama
After = network-online.target

[Service]
ExecStart = /usr/bin/ollama serve
User = ollama
Group = ollama
Restart = unless-stopped
RestartSec = 3

[Install]
WantedBy = default.target

Enable the service.

sudo systemctl daemon-reload
sudo systemctl enable ollama

NVIDIA GPU drivers

The final stage of the setup will be to ensure you have the correct GPU drivers installed if you want to use a GPU. The following steps relate to installing the proprietary NVIDIA drivers. If you already have the correct drivers installed, or you’re not going to be using a GPU, skip ahead to “Getting started”.

If you’re using the free and open source Nouveau drivers, you’ll likely run into the same issue I did: they won’t be recognised by Ollama and therefore the service will fall back to using only the CPU. So we’re going to install the better (though, unfortunately, proprietary) NVIDIA drivers instead.

But before we do, we’ll want to check which specific drivers are the correct ones for our GPU. For this, we’re going to use the nvidia-detect tool.

sudo apt install nvidia-detect

Run the tool.

nvidia-detect

Output:

Detected NVIDIA GPUs:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] [10de:1b82] (rev a1)

Checking card:  NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1)
Your card is supported by all driver versions.
Your card is also supported by the Tesla 470 drivers series.
It is recommended to install the
    nvidia-driver
package.

As we can see from the output, the tool will tell us exactly which package we should install (in my case, nvidia-driver). So let’s go ahead and install it.

sudo apt install nvidia-driver

If you were previously using the Nouveau drivers, these should be replaced by the NVIDIA drivers after a reboot. You can verify this by running the following command and looking out for “driver=nvidia” in the output.

sudo lshw -c display

Using Ollama

Start the Ollama service.

sudo systemctl start ollama

Check the service is running without any issues.

systemctl status ollama

Output:

● ollama.service - Ollama
     Loaded: loaded (/etc/systemd/system/ollama.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-03-29 08:19:02 GMT; 2min ago
   Main PID: 1642 (ollama)
      Tasks: 17 (limit: 38369)
     Memory: 756.3M
        CPU: 4.606s
     CGroup: /system.slice/ollama.service
             └─1642 /usr/bin/ollama serve

Warning: some journal files were not opened due to insufficient permissions.

The “Warning” is nothing to worry about. It’s just letting us know our regular user doesn’t have permission to view the journal files. This is expected. If the same command is run with sudo, we would see more information. The “active (running)” bit is all we’re concerned with right now.

Introductions with mistral

To get us started, let’s download one of the more popular, smaller LLMs and see what it has to say. The following command will first download the mistral model because we won’t already have it stored locally.

ollama run mistral

Once we’ve pulled down the mistral model, we’ll be given the prompt which will allow us to start chatting.

Prompt:

>>> Tell me a 100-word story about a dog named Fido.

Response:

 In the quaint town of Willowbrook, lived a loyal and spirited dog named Fido. His coat, a beautiful blend of brown and gold, shone under the sun. Though he was old, his spirit remained young at heart. Every morning, Fido would wake up early and wait by the porch for his best friend, Billy. A boy with an insatiable curiosity and a kind soul, they were inseparable. Together, they embarked on daily adventures through the woods, exploring new trails and making memories.

One day, while chasing a squirrel, Fido got lost. Billy was heartbroken but didn't give up hope. He put up posters, asked around town, and even offered a reward. After days of searching, they heard a bark - it was Fido! Overjoyed, Billy scooped up his beloved friend, their bond stronger than ever. Their tale became a beacon of loyalty and friendship in Willowbrook.

It won’t be winning any awards, but you get the idea. How about some practical advice…

Prompt:

>>> What is an easy way to find specific files and then run a command on them?

Response:

To find specific files with certain characteristics and then run a command on them, you can use a combination of commands in your terminal or command prompt. Here's a step-by-step guide using Linux as an example:

1. Use the `find` command to locate the desired files based on specifications such as file type, name, or location. For instance, to find all .txt files in the current directory and its subdirectories:

    find . -type f -name "*.txt"

2. Once you have located the files, pipe the output of the `find` command into the terminal of the command you want to execute. For example, if you wish to display the contents of all .txt files:

    find . -type f -name "*.txt" -exec cat {} \;

This command will search for all .txt files and display their contents in the terminal. Adjust the command based on your specific needs.

Coding with dolphincoder

Let’s step it up a bit and get some coding tips from the dolphincoder model.

Pull the model and run it.

ollama run dolphincoder

Prompt:

>>> Write me a Bash script for checking which devices are mounted, then listing some key details for each, such as mountpoint, used space, names etc.

Response:

#!/bin/bash

# Create the directory where we will save the details
mkdir -p /tmp/mounted_disks

# Iterate over all mounted disks and create a file for each with their details
for device in $(df | awk '{if($6 != "tmpfs" && $6 != "/" && $6 != "Mounted" ) print $NF}') ; do
    filename=$(basename "$device")
    echo "Device: $device" > /tmp/mounted_disks/"$filename".txt
    df -h "$device" >> /tmp/mounted_disks/"$filename".txt
done

# End the script
exit 0

Let’s take a look at the results.

ls /tmp/mounted_disks

Output:

1000.txt  boot.txt  dev.txt  lock.txt  run.txt  shm.txt

boot.txt:

Device: /boot
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       470M  136M  310M  31% /boot

It’s not perfect, but you get a sense of how powerful such models can be, both for beginners wanting to learn as well as seasoned professionals who might just want to save some time.

Image tasks with llava

We’re taking a visual turn with our next model: llava. This one will allow us to perform some basic tasks against images. I mean basic only in the sense that this llava model can ‘see’ images, but it can’t generate them.

So, what if we had a directory full of images, and we wanted to output a description of each image to its own text file? You can imagine how useful this could be as an aid for accessibility, such as the automated generation of alternative text.

Pull the model.

ollama run llava

Create a couple of new directories for the purpose of testing the model.

mkdir -p ~/Pictures/llava-testing/descriptions

Copy some images into the llava-testing directory, then create a Bash script for creating the image descriptions, which will be saved in the descriptions directory.

#!/bin/bash

IMG_DIR=/home/$USER/Pictures/llava-testing
DES_DIR=$IMG_DIR/descriptions

for photo in $(ls "$IMG_DIR"/*.jpg | xargs -n 1 basename); do 
	touch "$DES_DIR"/"$photo".txt && 
	ollama run llava "Print the filename and describe the photo: $IMG_DIR/$photo" > "$DES_DIR"/"$photo".txt;
done

A generated description, misty-hills.jpg.txt:

The image appears to be a composite of two photos. In the foreground, there is a photo with heavy fog or smoke, obscuring the view and creating an atmospheric effect. The fog seems to blend into the background, which contains a photo of a forested area under overcast skies, possibly during twilight or on a cloudy day. The sunlight filters through the fog and mist, casting a soft glow on the trees and landscape. The image has a moody, almost dreamlike quality due to the fog and the interplay between the two photos.

The photo, misty-hills.jpg:

Black and white photo. In the foreground, the tops of trees pushing above thick mist, which continues into the background, obscured by a whiteout formed from the mist

Again, not perfect (the model thinks there are “two photos”), but still pretty impressive, right? Of course, the more complex the image, the less likely it is that the smaller llava models will be able to provide an accurate description; the more likely it is they’ll incorrectly interpret subtle visual effects that you or I might have no issue making sense of.

As a final flourish to our fun with Ollama, we can use the power of grep to search through our image descriptions for key words or phrases. This could come in handy when trying to find older images without helpful filenames.

grep dog ~/Pictures/llava-testing/descriptions/*

Output:

descriptions/untitled.jpg.txt:- The central part of the photo features a dog standing in front of a closed wooden door. The dog looks like a medium-sized breed with a patterned coat, possibly a dachshund mix, looking towards the camera.

The photo, untitled.jpg:

Black and white photo. Dog waiting at the front door of a house, looking off to the left and positioned between two potted plants

List installed models

The following command will show us which models are currently installed at /usr/share/ollama/.ollama:

ollama list

Output:

NAME                      	ID          	SIZE  	MODIFIED     
dolphincoder:latest       	1102380927c2	9.1 GB	21 minutes ago
llava:latest              	8dd30f6b0cb1	4.7 GB	2 minutes ago
mistral:latest            	61e88e884507	4.1 GB	40 minutes ago

What next?

Now you have the tool, it’s about how you choose to use it. Once set up, Ollama is just a simple command away from helping you with any number of textual (and some visual) projects.

I’d love to know how you’re using Ollama, so reach out and let me know on Mastodon or via email.