Ahmad Mujahid/Blog
← Blog

Install Ollama Under Docker

Ahmad Mujahid Abdurrahman

Install Ollama Under Docker

Install Docker Engine

Update package indices and install the core Docker runtime components:

sudo apt update && sudo apt install -y curl docker.io util-linux-extra

Enable the Docker service to automatically start on boot:

sudo systemctl start docker
sudo systemctl enable docker

Create the Docker group

sudo groupadd docker

Configure non-root user permissions so you can run Docker commands without prefixing sudo:

sudo usermod -aG docker $USER

Note: You must log out and back into your Ubuntu user session for these group changes to apply.

Activate the new group membership in your current shell session:

newgrp docker

Option 1: CPU Only Installation

Install Ollama with CPU only is as simple as running the following command:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Option 2: GPU Installation (NVIDIA)

To install Ollama with GPU support, you need to install the NVIDIA Container Toolkit. Follow these steps.

Install prerequisites for NVIDIA GPU support:

sudo apt update && sudo apt install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2

Configure the NVIDIA package repositories and install the NVIDIA Container Toolkit:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

Update the package lists and install the NVIDIA Container Toolkit:

sudo apt update

Install the NVIDIA Container Toolkit and its dependencies:

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
  sudo apt install -y \
      nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

Source: NVIDIA Container Toolkit

Install Ollama with GPU support in Docker:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Run Ollama

docker exec -it ollama ollama run llama2

List of avalilable models can be found at Ollama Models.