> ## Documentation Index
> Fetch the complete documentation index at: https://docs.metrixllm.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Providers

> Connect OpenAI-compatible endpoints — Ollama, vLLM, LM Studio, LocalAI, and more

MetrixLLM supports any OpenAI-compatible endpoint as a custom provider. This lets you route traffic to local models (Ollama, vLLM, LM Studio) or third-party APIs that implement the OpenAI chat completions format.

## What is a Custom Provider?

A custom provider is a per-workspace configuration that tells the MetrixLLM gateway to route requests to your own endpoint instead of built-in providers. Custom providers:

* Are **per-workspace** — each workspace can have its own custom endpoints
* Support the **OpenAI chat completions** format (`/v1/chat/completions`)
* Can be **local** (no API key needed) or **remote** (with authentication)
* Appear alongside built-in providers in routing configuration
* Support **model registration** so you can track costs and capabilities

## Supported Servers

Any server implementing the OpenAI API format works:

| Server          | Default URL                      | Notes                      |
| --------------- | -------------------------------- | -------------------------- |
| **Ollama**      | `http://localhost:11434/v1`      | Local models via Ollama    |
| **vLLM**        | `http://localhost:8000/v1`       | High-throughput serving    |
| **LM Studio**   | `http://localhost:1234/v1`       | Desktop GUI for local LLMs |
| **LocalAI**     | `http://localhost:8080/v1`       | Drop-in OpenAI replacement |
| **Together AI** | `https://api.together.xyz/v1`    | Cloud inference            |
| **Groq Cloud**  | `https://api.groq.com/openai/v1` | Fast cloud inference       |
| **OpenRouter**  | `https://openrouter.ai/api/v1`   | Multi-model router         |

## Setup

### 1. Add a Custom Provider

Navigate to **Settings > Custom Providers** in your workspace dashboard and click **Add provider**.

Fill in:

* **Provider Name** — A human-readable label (e.g., "My Local Llama")
* **Base URL** — The server's base URL (e.g., `http://localhost:11434/v1`)
* **API Key** — Optional. Leave empty for local servers without authentication

### 2. Register Models

After adding a provider, expand it and click **Add model**. For each model:

* **Model ID** — The model name the server expects (e.g., `llama-3.3-70b`)
* **Display Name** — A human-friendly name (e.g., "Llama 3.3 70B (Local)")
* **Context Window** — Maximum context length (e.g., `128K`)

### 3. Test the Connection

Click **Test** on your provider to verify connectivity. The gateway will hit the `/v1/models` endpoint to confirm the server is reachable and list available models.

## Using Custom Providers

### Via the Gateway API

Send requests to the MetrixLLM gateway with your registered model name:

```bash theme={null}
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer sk-metrix-your-key" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3.3-70b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'
```

The gateway automatically routes to your custom provider based on the model name.

### Via the Dashboard Playground

1. Go to **Playground**
2. Select your custom model from the model dropdown
3. Send a test message

### Via the Drop-in Endpoint

Use the OpenAI SDK with the MetrixLLM base URL:

```python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="sk-metrix-your-key"
)

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

## How Routing Works

When a request arrives at the gateway:

1. The gateway looks up the model name in `admin_models` (built-in providers)
2. If not found, it checks your workspace's custom providers
3. If a match is found, the request is forwarded to your custom endpoint
4. The response is parsed using OpenAI format and returned to the client

Custom providers use **BYOK mode** by default — your endpoint's API key is used directly (if configured).

## Caching and Invalidation

Custom provider configurations are cached in the gateway for **60 seconds**. When you update a provider in the dashboard, the cache is invalidated immediately via NATS pub/sub.

## Limitations

* Custom providers only support the **chat completions** format (`/v1/chat/completions`)
* Embedding endpoints are not yet supported for custom providers
* Web search injection is not available for custom provider models
* Custom provider models are workspace-scoped — other workspaces cannot see them

## API Reference

### List Custom Providers

```
GET /api/workspaces/:workspaceId/custom-providers
```

### Create Custom Provider

```
POST /api/workspaces/:workspaceId/custom-providers
Body: { "name": "My Llama", "baseUrl": "http://localhost:11434/v1", "apiKey": "optional" }
```

### Update Custom Provider

```
PUT /api/workspaces/:workspaceId/custom-providers/:providerId
Body: { "name": "Updated Name", "baseUrl": "http://new-url:8000/v1" }
```

### Delete Custom Provider

```
DELETE /api/workspaces/:workspaceId/custom-providers/:providerId
```

### Test Connection

```
POST /api/workspaces/:workspaceId/custom-providers/:providerId/test
```

### Add Model

```
POST /api/workspaces/:workspaceId/custom-providers/:providerId/models
Body: { "modelName": "llama-3.3-70b", "displayName": "Llama 3.3 70B", "contextWindow": "128K" }
```

### Delete Model

```
DELETE /api/workspaces/:workspaceId/custom-providers/:providerId/models/:modelId
```