Semantic Caching

MetrixLLM’s Semantic Cache stores responses from your LLM providers. When a new request arrives, the gateway generates an embedding of the prompt and checks the cache. If an exact match or a highly similar prompt is found, MetrixLLM returns the cached response instantly without calling the upstream provider.

​

Enabling the cache

Caching requires zero code changes in your application. Once enabled, the gateway automatically intercepts and caches responses for all requests passing through that workspace.

​

How matching works

MetrixLLM supports two types of cache matching:

​

Exact match

The fastest path. If the incoming prompt is character-for-character identical to a cached prompt (including system message, model, and temperature), the cached response is returned immediately with zero embedding overhead.

​

Semantic similarity

For prompts that aren’t exact matches but are semantically equivalent, the gateway uses vector embeddings to find similar cached responses. This handles cases like:

• Rephrased questions (“What is the capital of France?” vs “Which city is the capital of France?”)
• Minor wording differences
• Different instruction phrasings with identical intent

​

Configuration options

​

Tuning the similarity threshold

​

Identifying cache hits

When a response is served from the cache, your application receives it just like a normal response, but it will be much faster. You can definitively identify a cache hit by checking the response headers:

​

Viewing cache analytics

The dashboard highlights exactly how much money and time caching saves you. Navigate to Cost Analytics in your dashboard. At the top of the page, the Cache Savings card shows:

• The number of requests served from cache
• The total tokens saved
• The estimated USD saved based on the model’s pricing
• The average latency reduction