Mcp Server Tester

What is Mcp Server Tester?

Mcp Server Tester offers a Playwright fixture that connects to an MCP server (via stdio or HTTP) and supplies a high‑level API for calling tools, asserting responses, and generating conformance reports. It also supports data‑driven evaluation datasets where an LLM acts as a client, allowing automatic judging of tool discoverability and response quality.

How to Use Mcp Server Tester?

1. Install the package (beta) and Playwright:

   npx @gleanwork/mcp-server-tester@beta init
   npm install --save-dev @gleanwork/mcp-server-tester@beta @playwright/test
   

2. Configure the server in playwright.config.ts using the mcpConfig field (stdio or HTTP).
3. Write Playwright tests with the provided mcp fixture and custom matchers such as toContainToolText or toPassToolJudge.
4. Create eval datasets in JSON, then run them via runEvalDataset() in a test or with the CLI command mst run.
5. View reports generated by the built‑in reporter for pass/fail metrics and trace data.

Key Features

• Deterministic Playwright tests for tool calls, schema validation, and protocol conformance.
• Rich set of custom matchers for readable assertions.
• Data‑driven eval framework with JSON case definitions.
• LLM‑as‑judge support (Anthropic SDK by default) for groundedness, completeness, etc.
• Multiple client modes (api, Claude Code, Codex CLI, Cowork, ChatGPT desktop).
• Built‑in reporters and OpenTelemetry telemetry.
• CLI utilities: mst init, mst generate, mst run, mst batch, mst grade, plugin system.
• Support for both stdio and HTTP transports, including OAuth configuration.

Use Cases

• Regression testing of custom MCP tool implementations.
• Validation of tool metadata readability for autonomous LLM clients.
• Benchmarking tool discoverability across different LLM models.
• Continuous‑integration pipelines that enforce MCP spec conformance.
• Comparative evaluation of multiple server deployments via eval configs and variants.

FAQ

Q: Do I need to run an MCP server to use the tester? A: Yes, the tester connects to an existing MCP server; you point to it via the mcpConfig in Playwright config.

Q: Which Node version is required? A: Node.js 22+.

Q: How do I add LLM‑as‑judge capabilities? A: Install the Anthropic SDK (npm install --save-dev @anthropic-ai/sdk). Other SDKs can be added as optional dependencies.

Q: Can I test HTTP‑based MCP servers? A: Set transport: 'http' and provide serverUrl in the mcpConfig.

Q: What if I want to run non‑deterministic LLM client evaluations? A: Use the trials option to run a case multiple times and measure pass rates.

Q: Where are the evaluation reports stored? A: By default they are written to the .mcp-test-results directory defined in the eval config, but you can configure other result stores via plugins.

  MST (MCP Server Tester)

Testing, evals, and conformance for Model Context Protocol (MCP) servers. MST is published as @gleanwork/mcp-server-tester, and its CLI is mst.

Write deterministic Playwright tests against your MCP tools, or run data-driven eval datasets — including LLM-based evaluation of tool discoverability.

Playwright Tests

The mcp Playwright fixture connects to your MCP server (stdio or HTTP) and exposes a high-level API for calling tools and asserting responses. Custom matchers keep assertions readable.

import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';

test('read_file returns file contents', async ({ mcp }) => {
  const result = await mcp.callTool('read_file', { path: '/tmp/test.txt' });
  expect(result).toContainToolText('Hello, world');
  expect(result).not.toBeToolError();
});

test('server exposes required tools', async ({ mcp }) => {
  const tools = await mcp.listTools();
  expect(tools.map((t) => t.name)).toContain('read_file');
});

Playwright tests are fast, deterministic, and designed for CI. Use them for regression testing, schema validation, and protocol conformance. The framework includes built-in conformance checks for the MCP spec. mst generate records tool calls as a spec file you can edit (see CLI).

Available matchers:

Matcher	Description
toMatchToolResponse	Response exactly matches expected value (deep equal)
toContainToolText	Response contains expected substrings
toMatchToolSchema	Response validates against a Zod schema
toMatchToolPattern	Response matches a regex pattern
toMatchToolSnapshot	Response matches a saved baseline
toBeToolError	Response is (or is not) an error
toHaveToolResponseSize	Response size is within bounds
toSatisfyToolPredicate	Response satisfies a custom function
toHaveToolCalls	LLM called the expected tools
toHaveToolCallCount	LLM made N tool calls
toPassToolJudge	LLM evaluates response quality against a rubric

Eval Datasets

Eval datasets define cases as JSON: an input the client under test acts on, and assertions about what it did. Run them with runEvalDataset() in a Playwright test, on the api client, or with mst run on any client (Claude Code, the Codex CLI, Cowork, ChatGPT and the ChatGPT app's Codex surface). Checks on a single tool's response belong in Playwright tests, above.

{
  "name": "file-ops",
  "cases": [
    {
      "id": "read-config",
      "input": "What version is set in the app's config file?",
      "assertions": {
        "toolsTriggered": {
          "calls": [
            {
              "name": "read_file",
              "required": true
            }
          ]
        },
        "containsText": "1.0.0"
      }
    },
    {
      "id": "summarize-readme",
      "input": "Summarize the README in two sentences",
      "assertions": {
        "toolsTriggered": {
          "calls": [
            {
              "name": "read_file",
              "required": true
            }
          ]
        }
      },
      "judges": [{ "type": "rubric", "rubric": "groundedness" }]
    }
  ]
}

import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
import { loadEvalDataset, runEvalDataset } from '@gleanwork/mcp-server-tester';

test('file operations eval', async ({ mcp }, testInfo) => {
  const dataset = await loadEvalDataset('./data/evals.json');
  const result = await runEvalDataset(
    { dataset, client: 'api', model: 'claude-haiku-4-5', defaultTrials: 5 },
    { mcp, testInfo }
  );
  expect(result.passed).toBe(result.total);
});

Supported assertions:

Assertion	Description
containsText	The client's answer includes expected substrings
matchesPattern	The client's answer matches a pattern
toolsTriggered	The client called the expected tools
toolCallCount	The client made a number of tool calls in a range

Judges go in the case's judges list, beside assertions: each scores the answer against a rubric or reference, such as { "type": "rubric", "rubric": "groundedness" } or a plugin judge "acme/judge/completeness".

LLM client mode

In LLM client mode, a real LLM receives your server's tool list and a natural language prompt, then decides which tools to call. This tests whether your tool names, descriptions, and input schemas are clear enough for autonomous use — a different question from whether the tools return correct output.

A case with input runs on the client. In a Playwright test, runEvalDataset names the client and model ({ dataset, client: 'api', model: 'claude-haiku-4-5' }), and the api client uses the test's MCP connection; an eval config names them with client and model. A case can set its own client, model and clientOptions, except on the desktop clients (cowork, chatgpt, codex-app), which run a variant's cases in one app with the variant's settings: there, a case that changes them fails when its dataset loads.

{
  "id": "find-config",
  "input": "Find the application config file and return its contents",
  "assertions": {
    "toolsTriggered": {
      "calls": [
        {
          "name": "read_file",
          "required": true
        }
      ]
    }
  }
}

LLM client mode makes real API calls and produces non-deterministic results. Use trials to run a case multiple times and measure pass rate rather than expecting 100% on a single run. See the LLM Client Guide for configuration and cost management.

Installation

Requires Node.js 22+. These docs describe 2.0, which is in beta:

npm install --save-dev @gleanwork/mcp-server-tester@beta @playwright/test

npm install @gleanwork/mcp-server-tester installs 1.x, whose docs are at v1.1.1. Upgrading: Migration guide (v1.x → v2.0).

LLM-as-judge with the default Anthropic judge needs the Anthropic SDK:

npm install --save-dev @anthropic-ai/sdk

The SDK client's packages (ai and the @ai-sdk/* providers) are optional dependencies, installed by default; add them yourself if you install with --omit=optional.

Quick Start

npx @gleanwork/mcp-server-tester@beta init

The CLI wizard creates a playwright.config.ts, example tests, and a sample eval dataset configured for your server. See the CLI Guide for all options.

Configuration

Point the framework at your MCP server in playwright.config.ts:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  reporter: [['list'], ['@gleanwork/mcp-server-tester/reporters/mcpReporter']],
  projects: [
    {
      name: 'my-server',
      use: {
        mcpConfig: {
          transport: 'stdio',
          command: 'node',
          args: ['server.js'],
        },
      },
    },
  ],
});

For HTTP servers, set transport: 'http' and serverUrl. For servers that require OAuth, see the Transports Guide and CLI Guide for authentication setup, including CI/CD token management.

Plain stdio is supported by Claude CLI, Vercel AI SDK, ChatGPT (chatgpt and codex-app), and Cowork evaluation clients, including Cowork on macOS and Linux. Cowork also supports private file-backed native proxies and mixed HTTP/stdio server sets. See evaluation client transport support for client-specific fields, environment rules, and setup ownership.

Documentation

• Quick Start — detailed setup and configuration
• Assertions — all assertion types including snapshot sanitizers
• LLM Client Simulation — tool discoverability testing
• LLM Gateways — routing MST's LLM calls through a gateway
• Traces — sending each trial to an OpenTelemetry collector
• API Reference
• Transports — stdio and HTTP configuration, OAuth
• Protocol Versions — legacy and 2026-07-28, protocol matrices, era-aware conformance
• Agent Skills — testing and evaluating skills served over MCP (SEP-2640)
• CLI Commands — init, generate, login, token, auth, datasets, judges, run, grade, batch, open, cowork
• Evaluation Framework — eval configs, plugins and their extension kinds, the run format
• Built-in Judges — the built-in judges measured on MST's calibration set
• Cowork and ChatGPT Desktop — running evals on desktop clients
• Codex CLI — running evals on the Codex CLI (codex-cli), headless
• Design walkthrough and explainer — how evals are meant to run end to end, and what is built so far
• Architecture decisions — why MST is built as it is, including detached runs: a run on an environment that keeps going with the laptop off
• Glossary — the terms MST uses
• The run report: the report every eval run writes
• Development — contributing and building
• Migration Guide (v1.x → v2.0) — MCP SDK v2 and protocol versions
• Migration Guide (v0.12 → v1.0) — upgrading from pre-1.0 releases

AI Skills

Install AI skills to help your coding assistant write tests and evals, and improve tool metadata:

npx skills add -g gleanwork/mcp-server-tester

This installs skills globally so they're available across all your projects. Four skills are included:

Skill	Description
mcp-tester-guide	Reference: vocabulary, config, fixtures, matchers, eval configs, CLI, 1.x renames
write-mcp-test	Write Playwright tests that call tools directly and check them with matchers
write-mcp-eval	Write eval datasets and eval configs: cases a client and model act on, and variants
optimize-mcp-tool-metadata	Find tool-metadata changes that measurably help, with mst optimize

Compatible with Claude Code, Cursor, Windsurf, Copilot, and 40+ other AI agents.

Examples

The examples/ directory contains complete working examples:

• filesystem-server/ — Test suite for Anthropic's Filesystem MCP server: 5 Playwright tests, 11 eval dataset cases, Zod schema validation.
• sqlite-server/ — Test suite for a SQLite MCP server: 11 Playwright tests, 14 eval dataset cases.
• basic-playwright-usage/ — Minimal Playwright patterns.

Eval configs

For projects with multiple datasets, use an eval config. Dataset paths are shorthand for tagged file sources, while clients, metrics, judges, result stores, and other extensions are built-ins or come from plugins (referenced as <namespace>/<kind>/<name>, such as acme/judge/completeness):

{
  "name": "tool-selection-search",
  "datasets": [
    {
      "type": "file",
      "path": "evalsets/search.json"
    }
  ],
  "servers": {
    "prod": {
      "transport": "http",
      "serverUrl": "https://example.com/mcp"
    }
  },
  "client": "api",
  "clientOptions": {
    "provider": "anthropic"
  },
  "metrics": ["passed", "tool_count"],
  "results": {
    "store": {
      "type": "file",
      "dir": ".mcp-test-results"
    }
  }
}

Run one eval config or a bounded batch:

npx mst run \
  --config eval.json \
  --plugins ./plugins \
  --dry-run

npx mst batch \
  --config-dir configs \
  --workers 4 \
  --skip-existing \
  --dry-run

Use variants to compare server sets or client configurations. A variant can override servers, client, model, client options, tool maps, input templates, metrics, and judges. The canonical execution primitives remain EvalDataset, EvalCase, MCPConfig, and runEvalDataset.

Plugins add dataset sources, clients, judges, pairwise judges, metrics, result stores, connectors, environments and shared configs. A plugin is a plain default-exported object in ESLint's shape, and its extensions are referenced as <namespace>/<kind>/<name>. See Plugins. Secrets remain environment-variable or plugin-owned runtime inputs and do not belong in committed eval configs.

Known Limitations

These MCP protocol features are not currently supported. These are deliberate scope decisions, not bugs:

• MCP prompts (listPrompts, getPrompt)
• Server-to-client notifications
• Streaming tool responses (callTool waits for the complete response)

If any of these affect your use case, please open an issue.

License

MIT