Key Concepts

Architecture overview

Your Application
       │
       ▼
  MetrixLLM Gateway (:8000)
       │
       ├── Route matching
       ├── Rate limiting
       ├── Budget enforcement
       ├── Smart caching
       ├── Fallback chains
       │
       ▼
  Provider API (OpenAI, Anthropic, Google, ...)
       │
       ▼
  Response → Gateway → Your Application
       │
       └── Logged to Request Logs

The gateway sits between your application and AI providers, adding observability, reliability, and cost controls without modifying the underlying API contract.

Workspace

A Workspace is the top-level isolated environment in MetrixLLM. Everything belongs to a workspace — there is no cross-workspace data sharing. Each workspace contains its own:

API keys
Provider credentials (BYOK)
Routing rules and model configurations
Guardrails and budget limits
Request logs and analytics
Team members and roles
Billing and credit balance

A typical organization structure:

Organization: Acme Corp
├── Workspace: Production
│   ├── API Keys: prod_key_1, prod_key_2
│   ├── Providers: OpenAI, Anthropic
│   ├── Members: 5 (2 admins, 3 viewers)
│   └── Tier: Team
├── Workspace: Staging
│   ├── API Keys: staging_key_1
│   ├── Providers: OpenAI (test key)
│   ├── Members: 3
│   └── Tier: Pro
└── Workspace: Development
    ├── API Keys: dev_key_1
    ├── Providers: OpenAI (test key)
    ├── Members: 8
    └── Tier: Free

Workspaces map to deployment environments. Data isolation ensures a developer viewing Development logs never sees Production user prompts.

API Key

A MetrixLLM API key (mtx_...) authenticates requests to the gateway. Keys are scoped to a single workspace and are never shared across environments. Unlike provider API keys (like OpenAI’s sk-...), MetrixLLM keys do not directly call any AI provider — they authenticate you to the gateway, which then uses your workspace’s stored provider credentials.

Key lifecycle

Create — Generate a key in Settings → API Keys
Use — Pass as Authorization: Bearer mtx_... header
Rotate — Revoke old keys and create new ones periodically
Revoke — Immediately invalidate compromised keys

API keys are hashed before storage. The full key is only shown once at creation time.

Provider & BYOK

MetrixLLM uses a Bring Your Own Key (BYOK) model. You configure your own API keys for OpenAI, Anthropic, Google, and other providers in your workspace settings. MetrixLLM stores them encrypted and uses them when routing requests on your behalf.

Supported providers

Provider	Models	Native Web Search
OpenAI	GPT-4o, GPT-4.1, o3, o4-mini	Yes
Anthropic	Claude Opus 4, Sonnet 4	Yes
Google	Gemini 2.5 Pro, Flash	Yes
xAI	Grok-3	No
Groq	Llama 3.3 70B	No
Together	Various open models	No

MetrixLLM does not mark up the cost of model calls. You pay your provider directly at their standard rates.

Route

A Route is a named gateway endpoint path (e.g. /v1/chat/completions). You can configure routing rules, retry logic, rate limits, and model fallback chains per route.

Default route

Every workspace starts with a “Default Route” that handles all /v1/* requests. You can create additional routes to segment traffic by use case:

/v1/chat/completions  → Default Route (handles all chat models)
/v1/completions       → Default Route (handles all completion models)

Routing Rule

A Routing Rule defines how requests for a given route are handled. It contains:

Ordered model list — The fallback chain (first model tried, then fallbacks)
Retry configuration — Max retries, backoff timing, status codes to retry on
Rate limits — Per-route request throttling
Budget caps — Per-request and monthly cost limits
Load balancer config — Round-robin, weighted, or random across multiple targets

Fallback chain example

Primary:     openai/gpt-4o
Fallback 1:  anthropic/claude-sonnet-4
Fallback 2:  google/gemini-2.5-flash

If OpenAI returns a 5xx or times out, the gateway automatically retries with Anthropic, then Google.

Request ID

Every request gets a unique ID in the format req_xxxxxxxx. It appears in the X-Request-Id response header and in your Request Logs. Use it to correlate a specific call across your application logs and MetrixLLM’s logs.

{
  "id": "req_aK3x9mP2",
  "choices": [...],
  "usage": { "prompt_tokens": 42, "completion_tokens": 128 }
}

Session

A Session is a group of related requests tagged with the same metrix-session-id header. Use sessions to correlate multi-turn conversations or multi-step agent workflows in the Request Logs.

# All requests with this header are grouped under one session
headers = {"metrix-session-id": "user_123_conversation_456"}

Sessions appear as a collapsible group in the Request Logs, making it easy to trace an entire conversation.

Cache

The Semantic Cache stores LLM responses and serves them for future requests that are identical or semantically similar. It is enabled per-workspace in settings.

How it works

A request arrives at the gateway
The cache checks for a semantically similar previous request
If found, the cached response is returned immediately (zero latency, zero cost)
If not found, the request proceeds to the provider and the response is cached

When to use caching

Good for: Repeated similar queries, FAQ-style responses, template-based outputs
Not ideal for: Highly dynamic content, real-time data, personalization-heavy prompts

Smart caching requires Pro plan or higher. Enable it in Workspace Settings → Smart Caching.

Guardrails

Guardrails are rules that validate and filter both inputs and outputs of LLM requests. They run at the gateway level before the request reaches the provider and after the response returns.

Types of guardrails

Type	When	Purpose
Input guardrails	Before provider call	Block harmful prompts, validate structure, enforce length limits
Output guardrails	After provider call	Filter sensitive data, validate format, check for policy violations

Guardrails require Pro plan or higher.

Evaluations

Evaluations measure how well your prompts and models perform against test datasets. Run evaluations in the dashboard to compare model outputs, measure quality scores, and track improvements over time. Requires Pro plan or higher.

Custom Properties

Attach arbitrary metadata to requests using the metrix-properties header. Custom properties appear in your Request Logs and can be used for filtering and analysis.

headers = {
    "metrix-properties": json.dumps({
        "user_id": "usr_abc123",
        "feature": "onboarding_flow",
        "environment": "production"
    })
}

Tier & Plan

Every workspace is assigned a tier that controls feature access and usage limits:

Tier	Requests/mo	Key Features
Free	25,000	BYOK only, playground, basic analytics
Pro	500,000	Native billing, caching, guardrails, evaluations
Team	2,000,000	Custom roles, audit logs, SSO
Enterprise	Unlimited	Dedicated engineer, SLA, custom retention

See Billing for the full feature comparison.

Durable Objects (Edge)

MetrixLLM’s Cloudflare Worker edge network uses Durable Objects to store per-workspace state (API keys, balance, configuration) at the edge. This enables:

Sub-millisecond key validation at the edge
Local rate limiting without round-trips to the origin
Budget enforcement even if the origin is temporarily unreachable

Durable Objects sync with the backend via NATS and internal API calls. You don’t interact with them directly — they’re an internal implementation detail.

NATS

NATS is the message bus used for real-time configuration propagation between the backend and edge gateway. When you update workspace settings (API keys, routing rules, cache toggle), the change propagates via NATS subjects:

workspace.{id}.config_updated  → Edge gateways pick up new config
workspace.{id}.ip_allowlist    → Edge gateways refresh IP rules
workspace.{id}.models          → Edge gateways refresh model list

If NATS is unavailable, the gateway falls back to database polling with a short delay.

Best practices

Start with Free — Prototype and validate your integration before committing to a paid plan.
Use sessions for multi-turn conversations — they make debugging much easier.
Enable caching for repetitive workloads — it can reduce costs by 30-60%.
Set budget alerts before going live — avoid surprise bills from runaway scripts.
Use custom properties to tag requests by user, feature, or environment — essential for cost attribution.
Rotate API keys periodically — revoke unused keys in Settings → API Keys.

Common pitfalls

Free tier is BYOK only. You must configure your own provider API keys. Pro+ supports native billing with no markup.

Workspace data is fully isolated. You cannot share API keys, logs, or configurations across workspaces.
Cache invalidation is not instant. If you change a prompt, cached responses from the old version may still be served briefly.
Sessions require the metrix-session-id header. Without it, multi-turn conversations appear as separate requests in logs.
Request IDs are per-gateway-call, not per-provider-call. If a fallback triggers, the same request ID appears in both the failed and successful log entries.

Overview

Connect

Build

Observe

Optimize

Manage

Gateway API Reference

Architecture overview

Workspace

API Key

Key lifecycle

Provider & BYOK

Supported providers

Route

Default route

Routing Rule

Fallback chain example

Request ID

Session

Cache

How it works

When to use caching

Guardrails

Types of guardrails

Evaluations

Custom Properties

Tier & Plan

Durable Objects (Edge)

NATS

Best practices

Common pitfalls

​Architecture overview

​Workspace

​API Key

​Key lifecycle

​Provider & BYOK

​Supported providers

​Route

​Default route

​Routing Rule

​Fallback chain example

​Request ID

​Session

​Cache

​How it works

​When to use caching

​Guardrails

​Types of guardrails

​Evaluations

​Custom Properties

​Tier & Plan

​Durable Objects (Edge)

​NATS

​Best practices

​Common pitfalls

Architecture overview

Workspace

API Key

Key lifecycle

Provider & BYOK

Supported providers

Route

Default route

Routing Rule

Fallback chain example

Request ID

Session

Cache

How it works

When to use caching

Guardrails

Types of guardrails

Evaluations

Custom Properties

Tier & Plan

Durable Objects (Edge)

NATS

Best practices

Common pitfalls