Semantic caching for LLMs with Upstash Vector and Bun

A standard key-value cache only hits when two users type the exact same character string. If one user asks "largest city in USA by population" and the next asks "which is the most populated city in the USA?", an exact-match lookup misses and your backend pays full LLM latency and token cost a second time for the same answer.

A semantic cache embeds each incoming prompt into a high-dimensional vector, queries an Upstash Vector index for a cached neighbor above a similarity threshold, and returns the stored completion in milliseconds when a match exists. In RAG pipelines and documentation Q&A bots where users ask the same handful of questions in dozens of ways, that lookup avoids calling OpenAI or another model provider on every duplicate query.

Close

Create the Upstash Vector index and Elysia app

First, create a vector database in the Upstash console and select one of the built-in embedding models during creation. The embedding model you pick sets the latency and recall tradeoff for every cache lookup: smaller-dimension models such as bge-small-en-v1.5 keep embedding latency low, while larger-dimension models trade a few extra milliseconds for tighter semantic accuracy.

Next, scaffold an Elysia server on Bun and install @upstash/semantic-cache alongside @upstash/vector:

bun create elysia app

bun install @upstash/semantic-cache @upstash/vector

Copy the REST URL and token from your Upstash Vector dashboard into a .env file in the project root:

UPSTASH_VECTOR_REST_URL=https://example.upstash.io
UPSTASH_VECTOR_REST_TOKEN=your_secret_token_here

Store and retrieve completions by vector proximity

Initialize the Upstash Index with your environment credentials and wrap it in SemanticCache with minProximity set to 0.85:

import { SemanticCache } from "@upstash/semantic-cache";
import { Index } from "@upstash/vector";

const DBConfig = {
  url: process.env.UPSTASH_VECTOR_REST_URL,
  token: process.env.UPSTASH_VECTOR_REST_TOKEN,
};

// 👇 your vector database
const index = new Index(DBConfig);

// 👇 your semantic cache
const semanticCache = new SemanticCache({ index, minProximity: 0.85 });

async function runCache() {
  await semanticCache.set("largest city in USA by population", "New York");

  await delay(1000);

  const result = await semanticCache.get(
    "which is the most populated city in the USA?",
  );

}

function delay(ms: number) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

runCache();

Run the script with Bun:

 bun ./src/{PATH_TO_YOUR_FILE}

Even though the lookup string "which is the most populated city in the USA?" or "Largest population in USA by city" shares only a few words with "largest city in USA by population", both embeddings land above the 0.85 proximity threshold and resolve to "New York".

Close

Two operational details in that script matter when you move it into an Elysia route handler. The minProximity parameter ranges from 0 to 1, where 1.00 requires an exact vector match; setting it too low risks returning a cached answer for a neighboring question with a different subject, while setting it too close to 1.00 causes reworded duplicates to miss. Newly written vectors also take a brief moment to index before they appear in similarity searches, which is why the standalone test waits one second between semanticCache.set and semanticCache.get. In production, check semanticCache.get at the start of the request, call your LLM only on a cache miss, and write the new completion back with semanticCache.set for subsequent callers.