To keep the core lightweight while enabling limitless enhancements, the flexible Extensions system (inspired by ComfyUI Custom Nodes) can be used to add features, register new provider implementations, extend, replace, and customize the UI with your own custom features.

Installation

Extensions can be installed from GitHub or by creating a local folder:

Local: Simply create a folder in ~/.llms/extensions/my_extension
GitHub: Clone GitHub Repo extensions into ~/.llms/extensions e.g:

git clone https://github.com/user/my_extension ~/.llms/extensions/my_extension

How it Works (Server)

Extensions are Python modules that plug into the server lifecycle using hooks defined in their __init__.py:

Hook	Purpose
`__parser__(parser)`	Add custom CLI arguments for custom CLI functionality
`__install__(ctx)`	Enhance the server instance (routes, providers, filters, etc.)
`__run__(ctx)`	Execute custom logic when running in CLI mode (to implement `__parser__` args)

The ctx parameter provides access to the ExtensionContext
The __install__ hook runs after providers are configured but before server is run

How it Works (UI)

Extensions can also include frontend components:

Placement: Add a ui folder within your extension directory
Access: Static files in this folder are automatically served at /ext/<extension_name>/*
Integration: Create a ui/index.mjs file. This is the entry point and must export an install function:

// Vue 3 Component
const MyComponent = {
    template: `...`
}

// ui/index.mjs
export default {
    install(ctx) {
        // Register or replace components, add routes, etc.
        ctx.components({ MyComponent })
    }
}

Register GET and POST API Endpoints

Custom server API endpoints are registered at /ext/<extension_name>/*

xmas init.py:

Example of creating a simple extension that adds a greet endpoint to the server:

def install(ctx):
    # Load greetings from extension's `./ui/greetings.json`
    greetings_path = Path(__file__).parent / 'ui' / 'greetings.json'
    if greetings_path.exists():
        with open(greetings_path) as f:
            greetings = json.load(f)
    else:
        greetings = ["Merry Christmas!"]

    count = 0
    async def greet(request):
        nonlocal count
        name = request.query.get('name')
        if not name:
            data = await request.post()
            name = data.get('name')

        if not name:
            name = 'Stranger'

        greeting = greetings[count % len(greetings)]
        count += 1
        return web.json_response({"result":f"Hello {name}, {greeting}"})

    # Extension endpoints registered at /ext/xmas/*
    ctx.add_get("greet", greet)   # GET  /ext/xmas/greet
    ctx.add_post("greet", greet)  # POST /ext/xmas/greet

The xmas/ui/greetings.json static file is automatically available from /ext/xmas/greetings.json.

Example of using making Chat Completion requests inside an endpoint:

    async def story(request):
        model = request.query.get("model")
        chat = ctx.chat_request(template="text", model=model, text="Tell me a short Christmas tale")
        response = await ctx.chat_completion(chat)
        return web.json_response(response)

    ctx.add_get("story", story)  # GET /ext/xmas/story

The chat_request uses chat templates configured in your llms.json defaults configuration:

chat - Chat template for text requests (default)
image - Chat template for image input requests
audio - Chat template for audio input requests
file - Chat template for file input requests (e.g. PDFs)
out:image - Chat template for image generation requests
out:audio - Chat template for audio generation requests

Finally, connect it to the install hook to be able to run the extension:

# register install extension handler
__install__ = install

Example endpoint returning user-specific data

system_tools init.py:

This extension first checks if user is signed in (Github OAuth), then returns prompts for that user if exists, otherwise returns default prompts from all users or default prompts in this extension.

def install(ctx):
    def get_user_prompts(request):
        candidate_paths = []        
        username = ctx.get_username(request) # check if user is signed in
        if username:
            # if signed in (Github OAuth), return the prompts for this user if exists
            candidate_paths.append(os.path.join(Path.home(), ".llms", "user", username, "system_prompts", "prompts.json"))
        # return default prompts for all users if exists
        candidate_paths.append(os.path.join(Path.home(), ".llms", "user", "default", "system_prompts", "prompts.json"))
        # otherwise return the default prompts from this repo
        candidate_paths.append(os.path.join(ctx.path, "ui", "prompts.json"))

        # iterate all candidate paths and when exists return its json
        for path in candidate_paths:
            if os.path.exists(path):
                with open(path, encoding="utf-8") as f:
                    txt = f.read()
                    return json.loads(txt)
        return default_prompts

    # API Handler to get prompts
    async def get_prompts(request):
        prompts_json = get_user_prompts(request)
        return web.json_response(prompts_json)

    # Extension endpoint registered at /ext/system_prompts/prompts.json
    ctx.add_get("prompts.json", get_prompts)

Register custom tool with 3rd Party extension

Example of creating a tool extension that adds a web search tool to the server. Uses requirements.txt to install 3rd Party ddgs dependency.

duckduckgo init.py

from ddgs import DDGS
from typing import Any, Dict

def web_search(query: str, max_results: int | None = 10, page: int = 1) -> Dict[str, Any]:
    """
    Perform a web search using DuckDuckGo.
    """

    try:
        results = []
        with DDGS() as ddgs:
            # text() returns an iterator
            for r in ddgs.text(query, max_results=max_results):
                results.append(r)
        return {"query": query, "results": results}
    except Exception as e:
        return {"query": query, "error": str(e)}

def install(ctx):
    ctx.register_tool(web_search)

__install__ = install

See Tool Support Docs and core_tools implementation for more examples.

Custom Provider Implementation example

providers/openrouter.py

Example of creating a custom provider that extends the GeneratorBase class to add support for image generation in OpenRouter.

def install(ctx):
    from llms.main import GeneratorBase

    # https://openrouter.ai/docs/guides/overview/multimodal/image-generation
    class OpenRouterGenerator(GeneratorBase):
        sdk = "openrouter/image"

        def __init__(self, **kwargs):
            super().__init__(**kwargs)

        def to_response(self, response, chat, started_at):
            # go through all image responses and save them to cache
            for choice in response["choices"]:
                if "message" in choice and "images" in choice["message"]:
                    for image in choice["message"]["images"]:
                        if choice["message"]["content"] == "":
                            choice["message"]["content"] = self.default_content
                        if "image_url" in image:
                            data_uri = image["image_url"]["url"]
                            if data_uri.startswith("data:"):
                                parts = data_uri.split(",", 1)
                                ext = parts[0].split(";")[0].split("/")[1]
                                base64_data = parts[1]
                                model = chat["model"].split("/")[-1]
                                filename = f"{model}-{choice['index']}.{ext}"
                                info = {
                                    "model": model,
                                    "prompt": ctx.last_user_prompt(chat),
                                }
                                relative_url, info = ctx.save_image_to_cache(base64_data, filename, info)
                                image["image_url"]["url"] = relative_url

            return response

        async def chat(self, chat, provider=None):
            headers = self.get_headers(provider, chat)
            if provider is not None:
                chat["model"] = provider.provider_model(chat["model"]) or chat["model"]

            started_at = time.time()
            if ctx.MOCK:
                print("Mocking OpenRouterGenerator")
                text = ctx.text_from_file(f"{ctx.MOCK_DIR}/openrouter-image.json")
                return ctx.log_json(self.to_response(json.loads(text), chat, started_at))
            else:
                chat_url = provider.chat_url
                chat = await self.process_chat(chat, provider_id=self.id)
                ctx.log(f"POST {chat_url}")
                ctx.log(provider.chat_summary(chat))
                # remove metadata if any (conflicts with some providers, e.g. Z.ai)
                chat.pop("metadata", None)

                async with aiohttp.ClientSession() as session, session.post(
                    chat_url,
                    headers=headers,
                    data=json.dumps(chat),
                    timeout=aiohttp.ClientTimeout(total=300),
                ) as response:
                    return ctx.log_json(self.to_response(await self.response_json(response), chat, started_at))

    ctx.add_provider(OpenRouterGenerator)


__install__ = install

This new implementation can be used by registering it as the image modality whose npm matches the providers sdk in llms.json, e.g:

{
    "openrouter": {
        "enabled": true,
        "id": "openrouter",
        "modalities": {
            "image": {
                "name": "OpenRouter Image",
                "npm": "openrouter/image"
            }
        }
    }
}

Find more Provider implementations in the providers extension.

Server Extensions API

Server Extensions allow you to extend the functionality of the LLM Server by registering new providers, UI extensions, HTTP routes, and hooking into the chat pipeline.

The Public API surface is exposed via the ExtensionContext class in main.py which provides access to the Server's functionality.

// __init__.py
def install(ctx):
    # ctx is an instance of ExtensionContext
    ...

Logging & Debugging

Methods for logging information to the console with the extension name prefix.

`log(message)`

Log a message to stdout if verbose mode is enabled.

message: str - The message to log.

`log_json(obj)`

Log a JSON object to stdout if verbose mode is enabled.

obj: Any - The object to serialize and log.

`dbg(message)`

Log a debug message to stdout if debug mode is enabled.

message: str - The debug message.

`err(message, e)`

Log an error message and exception trace.

message: str - The error description.
e: Exception - The exception object.

Registration & Configuration

Methods to register various extension components.

`add_provider(provider)`

provider: class - The provider class to register.

`register_ui_extension(index)`

index: str - Relative path to the index file (e.g. "index.html" or "app.mjs") within the extension directory.

`register_tool(func, tool_def=None)`

func: callable - The Python function to register.
tool_def: dict (Optional) - Manual tool definition. If None, it's generated from the function signature.