Routing & Fallbacks

Provider outages, latency spikes, and rate limits are inevitable. MetrixLLM’s routing engine lets your application survive these issues seamlessly without any changes to your code.

How it works

When you send a request to the gateway, MetrixLLM evaluates your configured routing rules for that specific endpoint (e.g. /v1/chat/completions). Instead of routing directly to a single model, MetrixLLM tries providers sequentially based on your fallback chain.

Routing modes

Every request can specify a mode parameter to control how the gateway authenticates with providers:

Mode	Description	Balance deduction	Use case
`"auto"` (default)	Prefers your BYOK key if configured, falls back to platform key	BYOK: none. Native: yes	Most users — automatic cost optimization
`"byok"`	Forces your own API key for every provider	None — your key, your cost	Cost-sensitive workloads, compliance
`"native"`	Forces MetrixLLM’s platform key	Yes — deducted from workspace credits	Quick testing, no provider accounts needed

How auto mode works

If you have a BYOK key configured for the requested provider, the gateway uses it (no charge).
If no BYOK key is configured, the gateway uses the platform key (deducted from workspace credits).
If your workspace has insufficient credits and you have a BYOK key, the gateway automatically uses your BYOK key to avoid a payment error.

Custom providers

Requests routed to Custom Providers (Ollama, vLLM, LM Studio, etc.) are always BYOK regardless of the mode parameter. These requests never deduct workspace credits since they use your own infrastructure.

{
  "model": "llama-3.3-70b-versatile",
  "mode": "byok",
  "messages": [{"role": "user", "content": "Hello!"}]
}

Configuring a fallback chain

Open Routing Rules

In your workspace, go to Routing Rules and click New Rule.

Add primary provider

Select the primary provider and model (e.g. openai, gpt-4o). This is the first provider MetrixLLM will always try.

Add fallback providers

Click Add Fallback and select your secondary provider (e.g. anthropic, claude-opus-4-5). You can add as many fallbacks as you want.

Configure retries

Set the Max Retries and select which HTTP status codes should trigger a retry (e.g. 429 Too Many Requests, 500 Internal Server Error).

Retry Configuration

MetrixLLM supports automatic retries with configurable parameters:

Parameter	Default	Description
`enabled`	`false`	Enable/disable automatic retries
`max_retries`	`2`	Maximum number of retry attempts per provider
`backoff_ms`	`500`	Milliseconds to wait between retries
`retry_on_statuses`	`[429, 500, 502, 503, 504]`	HTTP status codes that trigger a retry

Retries happen within the same provider before falling back to the next provider in the chain.

What your application sees

Your application does not need to handle the fallback logic. If openai is down and returns a 502 Bad Gateway, MetrixLLM automatically catches the error, retries the exact same prompt with anthropic, and returns the successful Anthropic response back to your app. Your code receives a normal HTTP 200 response, exactly as if the first provider had succeeded.

Response headers

To help you understand what happened under the hood, the gateway injects metadata headers into every response:

Header	Description
`X-Metrix-Provider`	The provider that ultimately served the request
`X-Metrix-Model`	The exact model used
`X-Metrix-Fallback`	`true` if a fallback provider was used
`X-Metrix-Retry-Count`	The total number of retries attempted across the chain
`X-Metrix-Retry-Status`	The outcome of retry attempts: `"success"` if a retry succeeded, `"not_triggered"` if no retries occurred
`X-Metrix-Cache-Status`	`"HIT"` or `"MISS"` indicating cache behavior

Customizing behavior per request

If you want to override the dashboard routing rules for a specific request, you can use the prompt_id mechanism (see Prompt Management) or define custom endpoint targets in the Custom Provider settings.

Programmatic routing example

You can inspect response headers to understand routing decisions:

const response = await fetch('https://gateway.metrixllm.com/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer sk-metrix-...',
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: 'Hello!' }],
  }),
});

// Check which provider served the request
const provider = response.headers.get('X-Metrix-Provider');
const model = response.headers.get('X-Metrix-Model');
const retryCount = response.headers.get('X-Metrix-Retry-Count');
const retryStatus = response.headers.get('X-Metrix-Retry-Status');
const wasFallback = response.headers.get('X-Metrix-Fallback');

console.log(`Served by: ${provider}/${model}`);
console.log(`Retries: ${retryCount}, Status: ${retryStatus}`);
console.log(`Fallback used: ${wasFallback}`);

Overview

Connect

Build

Observe

Optimize

Manage

Gateway API Reference

Routing & Fallbacks

How it works

Routing modes

How auto mode works

Custom providers

Configuring a fallback chain

Retry Configuration

What your application sees

Response headers

Customizing behavior per request

Programmatic routing example

​How it works

​Routing modes

​How auto mode works

​Custom providers

​Configuring a fallback chain

​Retry Configuration

​What your application sees

​Response headers

​Customizing behavior per request

​Programmatic routing example

How it works

Routing modes

How auto mode works

Custom providers

Configuring a fallback chain

Retry Configuration

What your application sees

Response headers

Customizing behavior per request

Programmatic routing example