Docker Deployment

Quick Start

llms.py is available as pre-built Docker images with full support for docker compose.

Using Pre-built Images

Pull and run the latest image:

docker pull ghcr.io/servicestack/llms:latest
docker run -p 8000:8000 -e OPENROUTER_API_KEY="your-key" ghcr.io/servicestack/llms:latest

Using docker compose (Recommended)

Create a .env file with your API keys:

OPENROUTER_API_KEY=sk-or-...
GROQ_API_KEY=gsk_...
GOOGLE_FREE_API_KEY=AIza...
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
GROK_API_KEY=xai-...
DASHSCOPE_API_KEY=sk-...
ZAI_API_KEY=sk-...
MISTRAL_API_KEY=...

Start the service:

docker compose up -d

Access the UI at http://localhost:8000

Docker Images

Available Images

Pre-built images are automatically published to GitHub Container Registry:

Latest stable: ghcr.io/servicestack/llms:latest
Specific version: ghcr.io/servicestack/llms:v2.0.30
Main branch: ghcr.io/servicestack/llms:main

Multi-Architecture Support

Images support:

linux/amd64 (Intel/AMD x86_64)
linux/arm64 (ARM64/Apple Silicon)

Docker automatically pulls the correct image for your platform.

Usage Examples

Basic Server

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

With Multiple API Keys

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="sk-or-..." \
  -e GROQ_API_KEY="gsk_..." \
  -e GOOGLE_FREE_API_KEY="AIza..." \
  -e ANTHROPIC_API_KEY="sk-ant-..." \
  ghcr.io/servicestack/llms:latest

With Persistent Storage

docker run -p 8000:8000 \
  -v llms-data:/home/llms/.llms \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

CLI Usage

Use the Docker container for CLI commands:

# Single query
docker run --rm \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest \
  llms "What is the capital of France?"

# List models
docker run --rm \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest \
  llms --list

# Check provider
docker run --rm \
  -e GROQ_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest \
  llms --check groq

Custom Port

Run on a different port:

docker run -p 3000:8000 \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

With Verbose Logging

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="your-key" \
  -e VERBOSE=1 \
  ghcr.io/servicestack/llms:latest

Docker Compose

Using Pre-built Image

The default docker-compose.yml uses the pre-built image:

version: '3.8'

services:
  llms:
    image: ghcr.io/servicestack/llms:latest
    ports:
      - "8000:8000"
    environment:
      - OPENROUTER_API_KEY=${OPENROUTER_API_KEY}
      - GROQ_API_KEY=${GROQ_API_KEY}
      - GOOGLE_FREE_API_KEY=${GOOGLE_FREE_API_KEY}
    volumes:
      - llms-data:/home/llms/.llms
    restart: unless-stopped

volumes:
  llms-data:

Commands:

# Start services
docker compose up -d

# View logs
docker compose logs -f

# Stop services
docker compose down

Building from Source

If you've cloned the repository, use docker-compose.local.yml:

# Build and start
docker compose -f docker-compose.local.yml up -d --build

# View logs
docker compose -f docker-compose.local.yml logs -f

# Stop
docker compose -f docker-compose.local.yml down

Data Persistence

The container stores configuration and analytics in /home/llms/.llms.

On first run, default llms.json and ui.json files are automatically created.

Named Volume (Recommended)

docker run -p 8000:8000 \
  -v llms-data:/home/llms/.llms \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

Local Directory

docker run -p 8000:8000 \
  -v $(pwd)/llms-config:/home/llms/.llms \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

Custom Configuration

Method 1: Mount a Directory

Create a directory with your config files:

mkdir -p config
# Add your custom llms.json and ui.json to config/

Mount the directory:

docker run -p 8000:8000 \
  -v $(pwd)/config:/home/llms/.llms \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

With docker compose:

volumes:
  - ./config:/home/llms/.llms

Method 2: Mount Individual Files

docker run -p 8000:8000 \
  -v $(pwd)/my-llms.json:/home/llms/.llms/llms.json:ro \
  -v $(pwd)/my-ui.json:/home/llms/.llms/ui.json:ro \
  -e OPENROUTER_API_KEY="your-key" \
  ghcr.io/servicestack/llms:latest

Method 3: Extract Default Configs

Run container to initialize defaults:

docker run --rm \
  -v llms-data:/home/llms/.llms \
  ghcr.io/servicestack/llms:latest \
  llms --init

Extract the configs:

docker run -d --name llms-temp -v llms-data:/home/llms/.llms ghcr.io/servicestack/llms:latest sleep 60
docker cp llms-temp:/home/llms/.llms/llms.json ./llms.json
docker cp llms-temp:/home/llms/.llms/ui.json ./ui.json
docker rm -f llms-temp

Edit and copy back:

# After editing
docker run -d --name llms-temp -v llms-data:/home/llms/.llms ghcr.io/servicestack/llms:latest sleep 60
docker cp ./llms.json llms-temp:/home/llms/.llms/llms.json
docker cp ./ui.json llms-temp:/home/llms/.llms/ui.json
docker rm -f llms-temp

Building Locally

Using Build Script

./docker-build.sh

Manual Build

docker build -t llms-py:latest .

With Custom Tag

./docker-build.sh v2.0.30

Run Local Build

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="your-key" \
  llms-py:latest

Health Checks

The Docker image includes health checks:

Check Container Health

docker ps

Look for health status in the STATUS column.

View Health Details

docker inspect --format='{{json .State.Health}}' container-name | jq

Security

The container:

Runs as non-root user (UID 1000)
Only exposes port 8000
No unnecessary packages installed
Multi-stage build reduces attack surface

Permissions

If mounting local directories, ensure they're writable:

mkdir -p llms-config
chmod 777 llms-config  # Or more restrictive with correct ownership

Troubleshooting

Container Won't Start

Check logs:

docker logs container-name

Port Already in Use

Change the host port:

docker run -p 3000:8000 ...

API Keys Not Working

Verify environment variables:

docker exec container-name env | grep API_KEY

Permission Issues

The container runs as user llms (UID 1000). Ensure mounted directories have correct permissions.

Production Deployment

For production:

Use HTTPS: Put behind reverse proxy (nginx, Traefik)
Persistent Storage: Use named volumes or bind mounts
Restart Policy: Use restart: unless-stopped
Resource Limits: Set CPU and memory limits
Monitoring: Monitor container health
Backups: Regular backups of /home/llms/.llms

Example with resource limits:

services:
  llms:
    image: ghcr.io/servicestack/llms:latest
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          memory: 512M
    restart: unless-stopped

Docker Deployment

GitHub OAuth

Configuration

CLI Reference

On this page