Rate Limits & Budgets

Because MetrixLLM acts as the central choke point for all LLM traffic in your organization, it is the best place to enforce limits. If a rogue script or malicious user hits your application, the gateway will block the traffic before it ever reaches OpenAI or Anthropic, saving you from a massive bill.

Rate Limits

You can configure two types of rate limits:

RPM (Requests Per Minute): The maximum number of HTTP calls allowed.
TPM (Tokens Per Minute): The maximum number of estimated tokens allowed.
Cost (USD): The maximum spend allowed within the window.

When a rate limit is exceeded, the gateway immediately returns a 429 Too Many Requests status code. You can apply rate limits at two scopes:

Workspace scope: Applies globally to all traffic in the workspace.
Route scope: Configured within a Routing Rule to limit traffic to a specific endpoint (e.g. /v1/chat/completions).

Rate Limit Response Headers

Every response from the gateway includes rate limit headers so you can track usage in real-time:

Header	Description
`X-RateLimit-{type}-Limit`	The maximum allowed value for this limit type (e.g. `X-RateLimit-requests-Limit: 100`)
`X-RateLimit-{type}-Remaining`	How many requests/tokens/cost remain in the current window
`X-RateLimit-Reset`	Seconds until the current rate limit window resets

The {type} suffix corresponds to the limit type: requests, tokens, or cost. Example response headers:

X-RateLimit-requests-Limit: 100
X-RateLimit-requests-Remaining: 42
X-RateLimit-Reset: 34
X-RateLimit-tokens-Limit: 100000
X-RateLimit-tokens-Remaining: 58420

429 Response Format

When a rate limit is exceeded, the gateway returns a JSON error body:

{
  "error": "Rate limit exceeded: Workspace RPM Limit (global)",
  "limit": 100,
  "current": 101,
  "window": "fixed_minute",
  "rule_id": "abc123",
  "retry_after": 45
}

Field	Description
`error`	Human-readable message with the rule name and type
`limit`	The configured maximum for this rule
`current`	The current usage count (requests, tokens, or cost)
`window`	The time window type (`fixed_minute`, `fixed_hour`, `fixed_day`, etc.)
`rule_id`	The unique ID of the rate limit rule that was triggered
`retry_after`	Seconds to wait before retrying (when available)

Budget Caps

A budget cap is a hard limit on USD spend over a specific time period.

Navigate to Settings → Budgets.
Set a monthly limit (e.g. $500.00).
Set an alert threshold (e.g. 80%).

When the workspace spend hits the alert threshold, the gateway fires the budget.threshold webhook (see Alerts). When the total spend hits the hard limit ($500.00), the gateway completely stops processing requests. All new traffic will receive a 402 Payment Required response until the budget resets or is increased.

Rate Limit Actions

When a rate limit is hit, the gateway can take different actions based on your configuration:

Action	Behavior
`reject` (default)	Returns `429 Too Many Requests` immediately
`throttle`	Adds a 500ms delay before processing the request
`fallback`	Automatically switches to a fallback model if configured

Overview

Connect

Build

Observe

Optimize

Manage

Gateway API Reference

Rate Limits & Budgets

Rate Limits

Rate Limit Response Headers

429 Response Format

Budget Caps

Rate Limit Actions

​Rate Limits

​Rate Limit Response Headers

​429 Response Format

​Budget Caps

​Rate Limit Actions

Rate Limits

Rate Limit Response Headers

429 Response Format

Budget Caps

Rate Limit Actions