PrivateGPT on Your Own Hardware: A Complete Local Document RAG Deployment Guide

Why Run PrivateGPT Locally?

Sending internal documents to cloud-based AI services like ChatGPT or Claude means your data traverses third-party servers, often for model training or retention. For legal firms, medical practices, or anyone handling sensitive contracts, this is a non-starter. PrivateGPT is an open-source project that runs a Retrieval-Augmented Generation (RAG) pipeline entirely on your own hardware. It ingests PDFs, Markdown, and text files, chunks them, embeds them into a local vector database (Chroma), and answers questions strictly based on that ingested context. No data leaves your network.

This guide walks you through a production-grade deployment using Docker Compose. You will set up the API server, configure a local LLM (Large Language Model) via Ollama, and connect the default UI. We will cover environment variable management, storage persistence, reverse proxy termination with SSL, and a troubleshooting table for the most common pitfalls. By the end, you will have a private, queryable document repository accessible from your LAN or the internet.

What You Will Learn

• How to structure a Docker Compose stack for PrivateGPT and Ollama.
• How to configure the settings.yaml and .env files without hardcoding secrets.
• How to expose the service securely with Caddy and a free SSL certificate.
• How to diagnose and fix common issues like model download failures and CUDA errors.

Prerequisites

Before starting, ensure your hardware and OS meet these requirements. The numbers below are typical ranges based on community reports, not strict limits. Actual consumption depends on your document volume and model size.

Check your user ID and group ID — you will need these for volume permissions later. Run:

id -u && id -g

Step-by-Step Deployment

1. Create Project Directory and .env File

Create a dedicated directory and a .env file to hold all variables, including secrets. This file must never be committed to Git.

mkdir -p ~/privategpt && cd ~/privategpt && touch .env

Edit .env with nano .env and add the following content. Replace the placeholder password with a strong, unique one:

# PrivateGPT Configuration
PRIVATE_GPT_VERSION=0.6.0
OLLAMA_VERSION=0.3.9

# API Key for PrivateGPT (used for basic auth)
PRIVATE_GPT_API_KEY=change_this_to_a_long_random_string

# Ollama Model Name
OLLAMA_MODEL=llama3.2:3b

# Host Ports
PRIVATE_GPT_PORT=8000
OLLAMA_PORT=11434

# Timezone
tz=UTC

Warning: The .env file contains credentials. Add .env to your .gitignore file if you plan to version this project. Never push it to a public repository.

2. Create Docker Compose File

Create a docker-compose.yml file. This defines two services: ollama for model management and privategpt for the application. We use named volumes for persistence.

services:
  ollama:
    image: ollama/ollama:${OLLAMA_VERSION:-latest}
    container_name: ollama
    restart: unless-stopped
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "${OLLAMA_PORT:-11434}:11434"
    environment:
      - OLLAMA_KEEP_ALIVE=24h
    # Optional: Enable GPU acceleration. Comment this out if you do not have an NVIDIA GPU.
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: all
    #           capabilities: [gpu]

  privategpt:
    image: zylon-ai/private-gpt:${PRIVATE_GPT_VERSION:-latest}
    container_name: privategpt
    restart: unless-stopped
    depends_on:
      - ollama
    ports:
      - "${PRIVATE_GPT_PORT:-8000}:8080"
    environment:
      - PGPT_MODE=api
      - PGPT_PROFILES=ollama
      - OLLAMA_BASE_URL=http://ollama:11434
      - OLLAMA_MODEL=${OLLAMA_MODEL:-llama3.2:3b}
      - PGPT_API_KEY=${PRIVATE_GPT_API_KEY}
    volumes:
      - privategpt_data:/app/private_gpt/data
      - ./local_docs:/app/local_docs:ro
    command: ["make", "run"]

volumes:
  ollama_data:
  privategpt_data:

Version Pin Warning: The versions in the .env file (0.6.0 and 0.3.9) were current at the time of writing. Check the official GitHub releases pages for PrivateGPT and Ollama before pinning — the versions above may be outdated by now.

3. Prepare the Local Documents Directory

Create the directory for your documents. The compose file mounts this read-only into the container.

mkdir -p ~/privategpt/local_docs && cd ~/privategpt

Place your PDFs, .txt, and .md files in this directory. Note: PrivateGPT does not watch this folder dynamically. You must ingest files via the API or CLI after adding them.

4. Pull and Start the Containers

Start the stack. This will pull images and start the services in detached mode.

docker compose up -d

Check the logs to ensure both containers start cleanly.

docker compose logs -f

Wait for the message indicating the API is running. Then, pull your desired model inside the Ollama container. The default model is llama3.2:3b. To change it, edit the .env file and restart. To pull the model manually:

docker exec -it ollama ollama pull llama3.2:3b

5. Ingest Documents via the API

PrivateGPT requires an API key for requests. Use the key you set in .env. First, test the health endpoint:

curl -H "Authorization: Bearer ${PRIVATE_GPT_API_KEY}" http://localhost:${PRIVATE_GPT_PORT}/v1/health

To ingest a file, use the upload endpoint. Replace yourfile.pdf with the actual filename inside the local_docs directory.

curl -X POST "http://localhost:${PRIVATE_GPT_PORT}/v1/ingest/file" \
  -H "Authorization: Bearer ${PRIVATE_GPT_API_KEY}" \
  -F "file=@local_docs/yourfile.pdf"

Repeat this for each file. If you have many files, consider scripting a loop.

6. Ask Questions via the API

Now you can ask questions. The response will include source documents and scores.

curl -X POST "http://localhost:${PRIVATE_GPT_PORT}/v1/chat/completions" \
  -H "Authorization: Bearer ${PRIVATE_GPT_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "What is the main topic of the document?"}],
    "stream": false
  }'

7. Access the Built-in UI (Optional)

PrivateGPT includes a simple UI for testing. By default, our compose file sets PGPT_MODE=api, which disables the UI. To enable it, change the environment variable to ui and restart. Note: The UI is not production-grade; it is for experimentation. For production, use the API or a custom frontend.

# Edit docker-compose.yml and change PGPT_MODE=ui, then run:
docker compose up -d --force-recreate

Open http://localhost:8000 in your browser. The UI will prompt for the API key.

8. Verify Data Persistence

To ensure your vector database survives container recreation, check the Docker volume.

docker volume inspect privategpt_privategpt_data

The Mountpoint path on your host is where the data lives. Do not modify it directly while the container is running.

Advanced Setup & Optimization

Reverse Proxy with SSL (Caddy)

Expose the service securely using Caddy as a reverse proxy. Caddy automatically obtains and renews Let's Encrypt certificates. Create a Caddyfile on the host:

your.domain.com {
    reverse_proxy localhost:8000
}

Then run Caddy with Docker. Add this service to your existing docker-compose.yml:

  caddy:
    image: caddy:2.8-alpine
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config

volumes:
  caddy_data:
  caddy_config:

Replace your.domain.com with your actual domain. Ensure your DNS A record points to your server's public IP. Caddy will handle SSL automatically. Do not expose ports 8000 or 11434 to the public internet — keep them behind the proxy or firewall.

Backup Strategy

Back up the two named volumes and your .env file. You can use tar or a tool like restic. To create a simple archive:

docker run --rm -v privategpt_ollama_data:/data -v $(pwd):/backup alpine tar czf /backup/ollama_backup.tar.gz -C /data . && docker run --rm -v privategpt_privategpt_data:/data -v $(pwd):/backup alpine tar czf /backup/privategpt_backup.tar.gz -C /data .

Store these archives offsite. Test restoration periodically.

Optional Hardening

The following security measures are optional and require adaptation to your specific environment. Copying them blindly may break the containers. Add these to your service definitions in docker-compose.yml only if you understand the implications.

  privategpt:
    security_opt:
      - no-new-privileges:true
    read_only: true
    tmpfs:
      - /tmp
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE

• read_only: Makes the container's filesystem read-only. PrivateGPT needs write access to /app/private_gpt/data (which is a volume, so it remains writable) and /tmp (provided by tmpfs).
• cap_drop: ALL: Removes all kernel capabilities. This may prevent the container from binding to ports below 1024 or accessing certain network features. The NET_BIND_SERVICE cap is added back to allow binding to port 80 if you remove the host port mapping.

Warning: Test these hardening options in a staging environment first. Some images require additional capabilities or specific system calls. If the container fails to start, remove the hardening block and re-evaluate.

Troubleshooting Common Issues

Frequently Asked Questions

1. Can I use a different LLM backend instead of Ollama?

Yes. PrivateGPT supports multiple backends including LlamaCPP and OpenAI-compatible APIs. To use a different backend, you must change the PGPT_PROFILES environment variable and adjust the settings.yaml file. The official documentation provides examples for each profile. Ollama is the simplest for local operation because it handles model management and GPU acceleration automatically.

2. How do I update PrivateGPT or Ollama to a newer version?

Edit the .env file and change the version numbers. Then run docker compose pull and docker compose up -d. This will recreate the containers with the new images. Your data in the named volumes will persist. Always read the changelog for breaking changes before upgrading, especially concerning the vector database schema.

3. Is it possible to run PrivateGPT on a Raspberry Pi?

It is possible but not recommended for anything beyond testing. ARM64 images exist for Ollama and PrivateGPT. However, embedding and inference on a Pi's CPU will be extremely slow. A question that takes 2 seconds on a desktop x86 CPU might take over a minute on a Pi. Stick to a machine with at least 8 GB of RAM and a modern x86 processor for usable performance.

4. How does PrivateGPT ensure my data is private?

All processing happens locally. The model runs in Ollama on your machine, and the vector database is stored on your local disk. No API calls are made to external services. The only network traffic is inbound requests to the API. To further protect the API, you should place it behind a reverse proxy with authentication and use HTTPS. The API key you set in .env acts as a basic access control mechanism.

5. What file formats are supported for ingestion?

PrivateGPT natively supports PDF, Markdown, and plain text files. For other formats like Word or HTML, you need to convert them to one of the supported formats first. You can use tools like pandoc or LibreOffice headless mode for conversion. The ingestion process extracts text, splits it into chunks, and embeds each chunk. Binary formats like images or scanned PDFs require OCR, which is not built into PrivateGPT by default.