Run Llama 3 Locally with Ollama: A Step-by-Step Self-Hosted AI Guide

Introduction

Running large language models locally has become a cornerstone of privacy-conscious AI usage. By hosting Llama 3 on your own hardware, you eliminate data leakage to third-party APIs, gain full control over model versions, and avoid per-token costs. Ollama simplifies this process by packaging models into easy-to-run containers, but a robust setup requires more than just pulling a model—it demands careful configuration of storage, networking, and security.

In this guide, you will learn how to deploy Ollama v0.33.2 (the latest official release as of August 27, 2026) using Docker Compose, pull Llama 3 models, and expose the service securely. We will cover hardware prerequisites, step-by-step installation, advanced optimizations like reverse proxying and backups, and common pitfalls with their solutions. By the end, you will have a production-ready local AI endpoint that respects your privacy and runs reliably.

The guide is written for homelab enthusiasts and self-hosters with intermediate Linux and Docker skills. Every command is copy-paste ready, and all configuration files are complete—no placeholders or omissions. Let's get started.

Prerequisites

Before you begin, ensure your hardware and software meet the minimum requirements. The table below outlines typical expectations—actual performance varies based on model size, concurrent requests, and storage speed.

Important: Always verify your Docker installation with docker run hello-world before proceeding. If you have an NVIDIA GPU, install the NVIDIA Container Toolkit (see NVIDIA's official documentation) to enable GPU passthrough.

Step-by-Step Installation

Step 1: Create Project Directory and Environment File

First, set up a dedicated directory for your Ollama deployment and a .env file to store secrets and version variables. Never commit the .env file to Git—it contains sensitive values.

mkdir -p ~/ollama && cd ~/ollama && touch .env

Now edit .env with your preferred editor (e.g., nano .env) and add the following content:

OLLAMA_VERSION=0.33.2
OLLAMA_MODEL_DIR=./models
OLLAMA_HOST=0.0.0.0
OLLAMA_PORT=11434
POSTGRES_PASSWORD=change_me_strong_password

Replace change_me_strong_password with a long random string (use openssl rand -base64 32). This password is used later if you add a database for model metadata—though Ollama itself doesn't require PostgreSQL, we include it here for future extensibility. The OLLAMA_MODEL_DIR points to a local folder where models will be stored.

Security note: Add .env to your .gitignore if you use Git, and never share it publicly.

Step 2: Create Docker Compose File

Create a docker-compose.yml file in the same directory. This file defines the Ollama service, its volumes, and port mappings. We use the latest tag as a fallback, but pin to the verified version via the environment variable.

version: '3.8'

services:
  ollama:
    image: ollama/ollama:${OLLAMA_VERSION:-latest}
    container_name: ollama
    restart: unless-stopped
    ports:
      - "${OLLAMA_PORT:-11434}:11434"
    volumes:
      - ${OLLAMA_MODEL_DIR:-./models}:/root/.ollama
    environment:
      - OLLAMA_HOST=${OLLAMA_HOST:-0.0.0.0}
      - OLLAMA_KEEP_ALIVE=5m
      - OLLAMA_NUM_PARALLEL=1
    # Uncomment the following lines if you have an NVIDIA GPU and installed the toolkit
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

Note: The version 0.33.2 is verified as of this writing. If you see a newer release on the official GitHub releases page, update the OLLAMA_VERSION in .env accordingly. The latest tag is used as fallback, but always check the official repository before pinning.

Step 3: Start the Ollama Container

Pull the image and start the container in detached mode.

docker compose up -d

Verify the container is running:

docker ps --filter name=ollama

You should see a status of Up with port 11434 mapped.

Step 4: Pull Llama 3 Model

Now pull the Llama 3 model. The default is llama3:latest (8B parameters). For larger variants, use llama3:70b or llama3:8b-instruct.

docker exec -it ollama ollama pull llama3:latest

This command downloads the model from the Ollama registry. The download size is approximately 4.7 GB for the 8B model. Wait for the success message.

Step 5: Test Inference

Run a quick prompt to verify the model works:

docker exec -it ollama ollama run llama3:latest "Hello, how are you?"

You should see a text response. If you get an error, refer to the troubleshooting section.

Step 6: Expose Ollama to the Network (Optional)

By default, Ollama listens on 0.0.0.0 inside the container, but Docker only exposes it on the host's localhost. To access from other devices on your LAN, modify the ports mapping in docker-compose.yml to "0.0.0.0:11434:11434" and restart:

sed -i 's/"${OLLAMA_PORT:-11434}:11434"/"0.0.0.0:${OLLAMA_PORT:-11434}:11434"/' docker-compose.yml && docker compose up -d

Warning: This exposes Ollama without authentication. Do not do this on a public network. Use a reverse proxy with authentication (see Advanced Setup).

Step 7: Verify API Endpoint

Test the REST API from your host:

curl http://localhost:11434/api/generate -d '{"model": "llama3:latest", "prompt": "Hello", "stream": false}'

You should receive a JSON response with the generated text.

Step 8: Set Up a Systemd Service (Optional)

If you're not using Docker's restart policy, create a systemd service to auto-start the container on boot. Create a file /etc/systemd/system/ollama-docker.service with:

[Unit]
Description=Ollama Docker Container
Requires=docker.service
After=docker.service

[Service]
Restart=always
ExecStart=/usr/bin/docker start -a ollama
ExecStop=/usr/bin/docker stop -t 10 ollama

[Install]
WantedBy=multi-user.target

Then enable and start it:

sudo systemctl enable ollama-docker.service && sudo systemctl start ollama-docker.service

Step 9: Manage Models with Ollama CLI

List installed models:

docker exec -it ollama ollama list

Remove a model to free space:

docker exec -it ollama ollama rm llama3:latest

Step 10: Update Ollama

When a new version is released, update the OLLAMA_VERSION in .env and run:

cd ~/ollama && docker compose pull && docker compose up -d

Always read the release notes for breaking changes.

Advanced Setup / Optimization

Reverse Proxy with Caddy and SSL

To expose Ollama securely over HTTPS, use a reverse proxy like Caddy. Create a Caddyfile:

ollama.yourdomain.com {
    reverse_proxy localhost:11434
}

Run Caddy as a container or system service. Caddy automatically obtains Let's Encrypt certificates. Ensure your domain resolves to your server's IP.

Backups

Back up your models and configuration. The models are stored in ~/ollama/models. Use tar to create a backup:

tar -czf ollama-backup-$(date +%Y%m%d).tar.gz -C ~/ollama models .env

Store backups offsite or on a separate drive.

Security Hardening (Optional)

The following settings add extra security but may break functionality if not tailored. Test thoroughly before deploying.

services:
  ollama:
    read_only: true
    tmpfs:
      - /tmp
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETUID
      - SETGID

Warning: read_only: true and cap_drop can cause failures if Ollama needs to write to unexpected locations or use certain system calls. Adjust based on your image's documentation. Run docker inspect ollama to see the default user and capabilities, and modify accordingly.

Performance Tuning

• Increase OLLAMA_NUM_PARALLEL to handle multiple concurrent requests (default 1).
• Adjust OLLAMA_KEEP_ALIVE to keep models loaded longer (e.g., 30m).
• For GPU acceleration, ensure the NVIDIA toolkit is installed and uncomment the deploy section in the compose file.

Troubleshooting

Conclusion

You now have a fully functional Llama 3 instance running locally with Ollama v0.33.2. This setup gives you privacy, control, and cost savings. Remember to regularly update Ollama and your models, back up your data, and secure your endpoint if exposed. With the reverse proxy and hardening tips, you can safely integrate this into your homelab.

FAQ

Q1: Can I run Llama 3 on a Raspberry Pi?

Yes, but performance will be limited. The 8B model requires ~8 GB RAM, which is possible on a Pi 5 with 8 GB, but inference will be slow (typically several seconds per token). Use llama3:8b-instruct-q4_0 for a smaller quantized version. Expect CPU-bound performance; a GPU is recommended for real-time use.

Q2: How do I change the model version?

Edit the docker-compose.yml or use the CLI: docker exec -it ollama ollama pull llama3:70b to download a different variant. To make it the default, restart the container with OLLAMA_MODEL environment variable set, or specify the model name in API calls.

Q3: Is Ollama free?

Yes, Ollama itself is free and open-source. The Llama 3 models are also free to use under the Llama 3 community license. You only pay for your electricity and hardware.

Q4: Can I use Ollama with other LLM frameworks like LangChain?

Absolutely. Ollama provides an OpenAI-compatible API at /v1. You can point LangChain or other tools to http://localhost:11434/v1 with an API key of ollama. See the Ollama documentation for details.

Q5: How do I secure Ollama if I expose it to the internet?

Use a reverse proxy with TLS and add authentication, such as Basic Auth or OAuth2 proxy. Never expose the raw Docker port. Also consider firewall rules to restrict access to specific IPs. Ollama itself lacks built-in authentication, so the proxy is your first line of defense.