Docker Deployment

The Docker image bundles llms.py with everything its extensions need — Python, bun, the .NET SDK and typst — so nothing has to be installed on the host beyond Docker itself.

If you just want it running, use the one-line installer:

curl -fsSL https://llmspy.org/install.sh | bash

The rest of this page covers running the image directly.

Quick Start

Using Pre-built Images

docker pull ghcr.io/servicestack/llms:latest

docker run -d --name llms \
  -p 127.0.0.1:8000:8000 \
  -v ~/.llms:/home/llms/.llms \
  --add-host=host.docker.internal:host-gateway \
  -e OPENROUTER_API_KEY="sk-or-..." \
  ghcr.io/servicestack/llms:latest

Then open http://localhost:8000

Using docker compose (Recommended)

1. Create a .env file with your API keys — one VAR=value per line, no quotes, no spaces around =:

OPENROUTER_API_KEY=sk-or-...
GROQ_API_KEY=gsk_...
GOOGLE_API_KEY=AIza...
ANTHROPIC_API_KEY=sk-ant-...
OPENAI_API_KEY=sk-...
XAI_API_KEY=xai-...
DASHSCOPE_API_KEY=sk-...
ZHIPU_API_KEY=...
MISTRAL_API_KEY=...

2. Start the service:

docker compose up -d

3. Access the UI at http://localhost:8000

docker-compose.yml uses env_file: .env, so every key in that file is passed through without needing an entry per provider.

API Key Environment Variables

Each provider reads a specific environment variable, defined by its env entry in providers.json. The current set:

.env.example in the repo has the full list with links to where you get each key. llms setup --list prints the same table annotated with which ones you already have configured.

Docker Images

Available Images

Published to GitHub Container Registry and Docker Hub on every push to main and every v* tag:

• Latest stable: ghcr.io/servicestack/llms:latest
• Specific version: ghcr.io/servicestack/llms:4.0.10
• Minor series: ghcr.io/servicestack/llms:4.0
• Main branch: ghcr.io/servicestack/llms:main

Multi-Architecture Support

Images support linux/amd64 and linux/arm64; Docker pulls the right one for your platform automatically.

Usage Examples

CLI Usage

Mount your config directory so the container has your providers and keys:

# Single query
docker run --rm -v ~/.llms:/home/llms/.llms \
  --entrypoint llms ghcr.io/servicestack/llms:latest \
  "What is the capital of France?"

# List models
docker run --rm -v ~/.llms:/home/llms/.llms \
  --entrypoint llms ghcr.io/servicestack/llms:latest ls

# Check a provider
docker run --rm -e GROQ_API_KEY="gsk_..." \
  --entrypoint llms ghcr.io/servicestack/llms:latest --check groq

The one-line installer wraps all of this behind an llms command, so llms ls and llms --check groq work directly.

Custom Port

docker run -p 3000:8000 -v ~/.llms:/home/llms/.llms ghcr.io/servicestack/llms:latest

With Verbose Logging

docker run -p 8000:8000 -v ~/.llms:/home/llms/.llms \
  ghcr.io/servicestack/llms:latest \
  llms --serve 8000 --verbose

Logging

llms reads two environment variables, both off by default:

docker run -e VERBOSE=1 -e DEBUG=1 -p 8000:8000 \
  -v ~/.llms:/home/llms/.llms ghcr.io/servicestack/llms:latest

With the installer the llms command does it for you:

llms up --verbose      # request logging
llms up --debug        # verbose + debug
llms up --debug -f     # ...and follow the logs
llms restart --debug   # turn it on for a running server
llms logs -f           # follow the logs
llms status            # shows the active log level

Voice Input

The microphone lives in the browser, not the container — the chat UI records audio and POSTs it to /transcribe, so no device passthrough is needed. The container's only job is turning that audio into text.

The voice extension's api mode needs nothing installed, so it works in the container out of the box. Add a key and restart:

echo 'GROQ_API_KEY=gsk_...' >> ~/.llms/.env
llms restart

Configure the provider and model under defaults in ~/.llms/llms.json:

{
  "defaults": {
    "voice": {
      "provider": "groq",
      "model": "whisper-large-v3",
      "language": "en"
    }
  }
}

Or point it at a local speech-to-text server — no key required. From inside the container the host is host.docker.internal:

{
  "defaults": {
    "voice": {
      "url": "http://host.docker.internal:8001/v1/audio/transcriptions",
      "model": "Systran/faster-whisper-small"
    }
  }
}

This is the same configuration llms.py uses everywhere, not something Docker-specific — see Voice Input for every setting, the LLMS_TRANSCRIBE_* environment overrides, and the other modes.

Check which mode was selected:

llms restart --verbose && llms logs | grep -i voice

Using api for voice: groq [llms.json] model=whisper-large-v3 [llms.json]

The image also ships ffmpeg, which the voxtype and transcribe modes need to convert the browser's webm recording. voxtype requires a graphical desktop session so it never applies in a container; transcribe is available if you mount your own script at /usr/local/bin/transcribe.

The Microphone button is missing

Browsers only expose getUserMedia in a secure context: HTTPS, or http://localhost / http://127.0.0.1. The default llms up binds to 127.0.0.1, so it works.

If you set LLMS_BIND=0.0.0.0 and browse to http://192.168.x.x:8000, the browser silently withholds the microphone API and no button appears — nothing to do with Docker or your configuration. Reach it over an SSH tunnel (ssh -L 8000:localhost:8000 host) or put it behind a TLS-terminating reverse proxy.

Reaching Services on the Host

Inside a container localhost refers to the container, not your machine. To use a local Ollama or LM Studio, start the container with:

--add-host=host.docker.internal:host-gateway

and point the provider at host.docker.internal in llms.json:

{
  "providers": {
    "ollama": {
      "enabled": true,
      "api": "http://host.docker.internal:11434"
    }
  }
}

llms setup makes this rewrite for you when you enable a local provider.

Data Persistence

The container stores configuration and analytics data in /home/llms/.llms, and creates default llms.json and providers.json there on first run.

Bind Mount (Recommended)

Keeps the files editable on the host:

docker run -p 8000:8000 -v ~/.llms:/home/llms/.llms ghcr.io/servicestack/llms:latest

Named Volume

docker run -p 8000:8000 -v llms-data:/home/llms/.llms ghcr.io/servicestack/llms:latest

Custom Configuration Files

Method 1: Mount a Local Directory (Recommended)

Extract the defaults, edit them, then mount the directory:

./docker-extract-configs.sh config
# edit config/llms.json
docker run -p 8000:8000 -v $(pwd)/config:/home/llms/.llms ghcr.io/servicestack/llms:latest

Or with docker compose:

volumes:
  - ./config:/home/llms/.llms

Method 2: Mount Individual Config Files

Mount specific files read-only to pin them:

docker run -p 8000:8000 \
  -v $(pwd)/my-llms.json:/home/llms/.llms/llms.json:ro \
  -v $(pwd)/my-providers-extra.json:/home/llms/.llms/providers-extra.json:ro \
  ghcr.io/servicestack/llms:latest

What Can You Customize?

In llms.json:

• Enable/disable providers
• Add or remove models
• Configure API endpoints
• Set pricing information
• Customize default chat templates
• Configure provider-specific settings

In providers-extra.json:

• Additional providers and models

Example: Custom Provider Configuration

{
  "defaults": {
    "text": {
      "model": "llama-3.3-70b-versatile",
      "messages": [
        {
          "role": "user",
          "content": ""
        }
      ]
    }
  },
  "providers": {
    "groq": {
      "enabled": true
    }
  }
}

docker run -p 8000:8000 \
  -v $(pwd)/custom-llms.json:/home/llms/.llms/llms.json:ro \
  -e GROQ_API_KEY="gsk_..." \
  ghcr.io/servicestack/llms:latest

Building Locally

./docker-build.sh                  # builds llms-py:latest
./docker-build.sh v4.0.10          # with a tag

docker compose -f docker-compose.local.yml up -d --build

Testing an Image

scripts/test-docker.sh in the repo pulls the image and exercises it end to end — metadata, multi-arch manifest, non-root user, toolchain versions, llms --init, llms ls, a live HTTP server and Docker's own health check:

./scripts/test-docker.sh                        # test the published latest
./scripts/test-docker.sh --image llms-py:dev    # test a local build
./scripts/test-docker.sh --quick                # skip the server tests

It exits non-zero if any check fails, so it works as a release gate.

Health Checks

The image includes a HEALTHCHECK. To see it:

docker ps                                                  # STATUS column
docker inspect --format='{{json .State.Health}}' llms | jq

Troubleshooting

Container Won't Start

docker logs llms

Permission Issues (Linux)

The container runs as UID 1000. If your UID is different, bind-mounted config won't be writable — run the container as yourself:

docker run --user $(id -u):$(id -g) -v ~/.llms:/home/llms/.llms ...

Port Already in Use

docker run -p 3000:8000 ...

API Keys Not Working

docker exec llms env | grep -i api_key

Note that --env-file does no quote stripping — GROQ_API_KEY="gsk_..." in a .env file passes the quotes through as part of the key. Leave them off.

Security Considerations

• Container runs as a non-root user (UID 1000)
• Only port 8000 is exposed; publish it on 127.0.0.1 to keep it off your network
• Multi-stage build keeps the final image small
• Images are published with build provenance attestations