TanStack
AI
Middleware

Built-in Middleware

Built-in middleware caches a tool result and redacts text from a reply. Use it when the same tool call repeats, or when a reply must hide a value.

TanStack AI ships ready-made middleware. Add one to the middleware array of chat():

MiddlewareImportWhat it does
toolCacheMiddleware@tanstack/ai/middlewaresCaches tool results by tool name and arguments
contentGuardMiddleware@tanstack/ai/middlewaresRedacts, changes, or blocks streamed text
otelMiddleware@tanstack/ai/middlewares/otelSends OpenTelemetry spans and GenAI metrics

Quick setup

Cache the results of a tool for 60 seconds:

ts
// api/chat.ts
import {
  chat,
  chatParamsFromRequest,
  toolDefinition,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import { toolCacheMiddleware } from "@tanstack/ai/middlewares";
import { openaiText } from "@tanstack/ai-openai";
import { z } from "zod";

const getWeather = toolDefinition({
  name: "getWeather",
  description: "Get the current weather for a city",
  inputSchema: z.object({ city: z.string() }),
}).server(async ({ city }) => {
  console.log(`getWeather ran for ${city}`);
  return { city, temperature: 21 };
});

// Create the cache once, so all requests share it.
const weatherCache = toolCacheMiddleware({
  ttl: 60_000,
  toolNames: ["getWeather"],
});

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: openaiText("gpt-6.1-sol"),
    messages,
    threadId,
    runId,
    tools: [getWeather],
    middleware: [weatherCache],
  });

  return toServerSentEventsResponse(stream);
}

Ask about the weather in Paris. Then start a new chat and ask again. The server console shows getWeather ran for Paris only one time. The second answer comes from the cache.

Each built-in is a normal ChatMiddleware. You can combine them with your own middleware in the same array.

toolCacheMiddleware

toolCacheMiddleware caches tool results by tool name and arguments. When the model calls a tool with the same name and arguments again, the middleware returns the cached result. The tool does not run again.

ts
import { chat } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { toolCacheMiddleware } from "@tanstack/ai/middlewares";
import { weatherTool, stockTool } from "./tools";

const stream = chat({
  adapter: openaiText("gpt-6.1-sol"),
  messages: [{ role: "user", content: "What's the weather in Paris?" }],
  threadId: "thread-1",
  tools: [weatherTool, stockTool],
  middleware: [
    toolCacheMiddleware({
      ttl: 60_000, // Entries expire after 60 seconds
      maxSize: 50, // Keep at most 50 entries (LRU eviction)
      toolNames: ["getWeather"], // Cache only these tools
    }),
  ],
});

What it does:

  • It caches only successful tool calls. It never stores errors.
  • A cache hit returns { type: 'skip', result } from onBeforeToolCall.
  • When the default storage reaches maxSize, it removes the oldest entry (LRU).
  • A cache hit moves the entry to the most recently used position.

Note

Identical calls in one model turn run at the same time, so both of them miss the cache. To let the second call use the first result, set toolExecution: 'sequential' on chat().

Ignore some arguments in the cache key

Use keyFn when some arguments must not change the key. Here the cache ignores the page number:

ts
import { toolCacheMiddleware } from "@tanstack/ai/middlewares";

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === "object" && value !== null;
}

toolCacheMiddleware({
  keyFn: (toolName, args) => {
    // `args` is `unknown`, so narrow it with a type guard first.
    if (!isRecord(args)) return JSON.stringify([toolName, args]);
    const { page, ...rest } = args;
    return JSON.stringify([toolName, rest]);
  },
});

Custom storage

The default cache lives in memory, inside one toolCacheMiddleware() instance. Pass a storage option to use another backend, like Redis, localStorage, or a database. A shared storage also shares the cache across chat() calls.

Implement the ToolCacheStorage interface from @tanstack/ai/middlewares:

ts
import { type ToolCacheEntry, type ToolCacheStorage } from "@tanstack/ai/middlewares";

interface MyStorage extends ToolCacheStorage {
  getItem: (key: string) => ToolCacheEntry | undefined | Promise<ToolCacheEntry | undefined>;
  setItem: (key: string, value: ToolCacheEntry) => void | Promise<void>;
  deleteItem: (key: string) => void | Promise<void>;
}

// ToolCacheEntry is { result: unknown; timestamp: number }
  • Each method can return a Promise, for async backends.
  • The middleware checks the TTL. Your storage only stores and returns entries.
  • With a custom storage, the middleware ignores maxSize. The storage controls its own size.

A Redis storage:

ts
import { chat } from "@tanstack/ai";
import { createClient } from "redis";
import { toolCacheMiddleware, type ToolCacheStorage } from "@tanstack/ai/middlewares";
import { adapter, messages } from "./server";
import { weatherTool } from "./tools";

const redis = createClient();

const redisStorage: ToolCacheStorage = {
  getItem: async (key) => {
    const raw = await redis.get(`tool-cache:${key}`);
    return raw ? JSON.parse(raw) : undefined;
  },
  setItem: async (key, value) => {
    await redis.set(`tool-cache:${key}`, JSON.stringify(value));
  },
  deleteItem: async (key) => {
    await redis.del(`tool-cache:${key}`);
  },
};

const stream = chat({
  adapter,
  messages,
  threadId: "thread-1",
  tools: [weatherTool],
  middleware: [toolCacheMiddleware({ storage: redisStorage, ttl: 60_000 })],
});

To share a cache across requests in one process, create the storage once, outside the request handler:

ts
import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import {
  toolCacheMiddleware,
  type ToolCacheEntry,
  type ToolCacheStorage,
} from "@tanstack/ai/middlewares";
import { openaiText } from "@tanstack/ai-openai";
import { weatherTool } from "./tools";

const entries = new Map<string, ToolCacheEntry>();

const sharedStorage: ToolCacheStorage = {
  getItem: (key) => entries.get(key),
  setItem: (key, value) => {
    entries.set(key, value);
  },
  deleteItem: (key) => {
    entries.delete(key);
  },
};

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: openaiText("gpt-6.1-sol"),
    messages,
    threadId,
    runId,
    tools: [weatherTool],
    middleware: [toolCacheMiddleware({ storage: sharedStorage })],
  });

  return toServerSentEventsResponse(stream);
}

contentGuardMiddleware

contentGuardMiddleware filters or changes streamed text in onChunk. Use it to:

  • Redact sensitive data (SSNs, emails, API keys).
  • Filter profanity.
  • Rewrite text while it streams.

The rules apply to TEXT_MESSAGE_CONTENT chunks only. All other chunk types pass through unchanged.

ts
import { chat } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { contentGuardMiddleware } from "@tanstack/ai/middlewares";

const stream = chat({
  adapter: openaiText("gpt-6.1-sol"),
  messages: [{ role: "user", content: "Tell me about customer 123-45-6789" }],
  threadId: "thread-1",
  middleware: [
    contentGuardMiddleware({
      rules: [
        // Regex and replacement
        { pattern: /\b\d{3}-\d{2}-\d{4}\b/g, replacement: "[SSN REDACTED]" },
        // Custom transform function
        { fn: (text) => text.replaceAll("badword", "****") },
      ],
      strategy: "buffered",
    }),
  ],
});

Matching strategies

  • 'buffered' (default). The middleware collects the text and applies the rules to the settled part. It holds back the last bufferSize characters, so it still catches a pattern split across two chunks ("...123-45" then "-6789..."). It flushes the buffer when the message or the run ends. Use it for most redaction.
  • 'delta'. The middleware applies the rules to each delta alone, when it arrives. This is the fastest, but a pattern split across two chunks can get through. If each pattern always fits in one delta, you can use it.

What it does

  • A rule that does not change the text does nothing. The chunk passes through unchanged.
  • With blockOnMatch: true, the middleware drops a matched chunk (it returns null from onChunk). It does not send the redacted text.
  • onFiltered is for logs and audits. It gets the text before and after, but it does not change what goes out.

otelMiddleware

otelMiddleware sends OpenTelemetry traces and metrics for each chat() call:

  • A root span for each call.
  • A child span for each agent-loop iteration.
  • A grandchild span for each tool run.

All spans have GenAI semantic-convention attributes.

ts
import { chat } from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { otelMiddleware } from "@tanstack/ai/middlewares/otel";
import { trace, metrics } from "@opentelemetry/api";
import { messages } from "./server";

const otel = otelMiddleware({
  tracer: trace.getTracer("my-app"),
  meter: metrics.getMeter("my-app"), // optional, turns on the GenAI histograms
});

const result = await chat({
  adapter: openaiText("gpt-6.1-sol"),
  messages,
  threadId: "thread-1",
  middleware: [otel],
});

otelMiddleware needs the optional @opentelemetry/api peer dependency. It has its own options for content capture, redaction, span names, extra attributes, and lifecycle callbacks. See OpenTelemetry for the setup, the list of spans and metrics, and all options.

Write your own

The built-ins are normal ChatMiddleware objects. To write your own, see Middleware.

Advanced

Import paths

toolCacheMiddleware and contentGuardMiddleware come from the @tanstack/ai/middlewares barrel. otelMiddleware has its own subpath, @tanstack/ai/middlewares/otel. So an import from the barrel never loads @opentelemetry/api, which is an optional peer dependency.

Policies that belong to your app, like tool-call budgets, stay in your code. See the tool-call budget recipe.

toolCacheMiddleware options

OptionTypeDefaultDescription
maxSizenumber100The maximum number of entries. The oldest entry goes first (LRU). Applies to the default in-memory storage only.
ttlnumberInfinityTime to live in milliseconds. The middleware does not return expired entries.
toolNamesstring[]All toolsCache only these tools. Other tools pass through.
keyFn(toolName, args) => stringJSON.stringify([toolName, args])Builds the cache key.
storageToolCacheStorageIn-memory MapA custom storage backend. When you set it, the middleware ignores maxSize.

contentGuardMiddleware options

OptionTypeDefaultDescription
rulesContentGuardRule[]noneRequired. The rules apply in order. Each rule gets the output of the rule before it. A rule is { pattern: RegExp; replacement: string } or { fn: (text: string) => string }.
strategy'delta' | 'buffered''buffered'How the middleware matches text. See Matching strategies.
bufferSizenumber50Buffered only. The number of characters to hold back, so patterns across chunk boundaries still match. Set it to at least the length of your longest pattern. The middleware flushes the buffer at the end of the stream.
blockOnMatchbooleanfalseWhen true, drop the whole chunk if a rule changes the text.
onFiltered(info: ContentFilteredInfo) => voidnoneRuns each time a rule changes the text. Gets { messageId, original, filtered, strategy }.
Edit on GitHub
Middleware: Advanced
Locks