Atla MCP Server

What is Atla MCP Server about?

The server implements the Model Context Protocol (MCP) to expose Atla's evaluation models as tools that LLMs can call. It offers two evaluation tools: evaluate_llm_response for a single criterion and evaluate_llm_response_on_multiple_criteria for batch evaluation, returning scores and textual critiques.

How to use Atla MCP Server?

1. Install: Recommended to manage the environment with uv. After installing uv, run the server with:

   ATLA_API_KEY=<your-api-key> uvx atla-mcp-server
   

2. Connect:
   • OpenAI Agents SDK – use MCPServerStdio with the same command and environment variables.
   • Claude Desktop – add a server entry to claude_desktop_config.json with the command, args, and ATLA_API_KEY.
   • Cursor – add a similar entry to .cursor/mcp.json.
3. Once the server is running, LLMs can invoke the exposed tools directly from the client interface.

Key Features

• Dual evaluation tools: single‑criterion and multi‑criterion evaluation.
• Seamless integration with popular MCP clients (OpenAI Agents SDK, Claude Desktop, Cursor).
• Simple deployment using uvx; no Docker or complex setup required.
• MIT licensed and open for contributions.

Use Cases

• Automated grading of chatbot or assistant responses during development.
• Benchmarking multiple LLMs against a consistent evaluation model.
• Integrating real‑time feedback loops in AI‑augmented workflows.
• Providing critique and scoring for user‑generated content in educational platforms.

FAQ

• Do I need an Atla account? Yes, an active Atla API key is required.
• Is the server still maintained? The repository was archived on July 21 2025 and the Atla API is no longer active.
• Can I run the server on Windows? Yes, as long as uv is installed and the environment variable ATLA_API_KEY is set.
• What programming language is used? The server is a Python package executed via uvx.
• How do I contribute? Fork the repo and follow the guidelines in CONTRIBUTING.md.

Atla MCP Server

[!CAUTION] This repository was archived on July 21, 2025. The Atla API is no longer active.

An MCP server implementation providing a standardized interface for LLMs to interact with the Atla API for state-of-the-art LLMJ evaluation.

Learn more about Atla here. Learn more about the Model Context Protocol here.

Available Tools

• evaluate_llm_response: Evaluate an LLM's response to a prompt using a given evaluation criteria. This function uses an Atla evaluation model under the hood to return a dictionary containing a score for the model's response and a textual critique containing feedback on the model's response.
• evaluate_llm_response_on_multiple_criteria: Evaluate an LLM's response to a prompt across multiple evaluation criteria. This function uses an Atla evaluation model under the hood to return a list of dictionaries, each containing an evaluation score and critique for a given criteria.

Usage

To use the MCP server, you will need an Atla API key. You can find your existing API key here or create a new one here.

Installation

We recommend using uv to manage the Python environment. See here for installation instructions.

Manually running the server

Once you have uv installed and have your Atla API key, you can manually run the MCP server using uvx (which is provided by uv):

ATLA_API_KEY=<your-api-key> uvx atla-mcp-server

Connecting to the server

Having issues or need help connecting to another client? Feel free to open an issue or contact us!

OpenAI Agents SDK

For more details on using the OpenAI Agents SDK with MCP servers, refer to the official documentation.

1. Install the OpenAI Agents SDK:

pip install openai-agents

2. Use the OpenAI Agents SDK to connect to the server:

import os

from agents import Agent
from agents.mcp import MCPServerStdio

async with MCPServerStdio(
        params={
            "command": "uvx",
            "args": ["atla-mcp-server"],
            "env": {"ATLA_API_KEY": os.environ.get("ATLA_API_KEY")}
        }
    ) as atla_mcp_server:
    ...

Claude Desktop

For more details on configuring MCP servers in Claude Desktop, refer to the official MCP quickstart guide.

1. Add the following to your claude_desktop_config.json file:

{
  "mcpServers": {
    "atla-mcp-server": {
      "command": "uvx",
      "args": ["atla-mcp-server"],
      "env": {
        "ATLA_API_KEY": "<your-atla-api-key>"
      }
    }
  }
}

2. Restart Claude Desktop to apply the changes.

You should now see options from atla-mcp-server in the list of available MCP tools.

Cursor

For more details on configuring MCP servers in Cursor, refer to the official documentation.

1. Add the following to your .cursor/mcp.json file:

{
  "mcpServers": {
    "atla-mcp-server": {
      "command": "uvx",
      "args": ["atla-mcp-server"],
      "env": {
        "ATLA_API_KEY": "<your-atla-api-key>"
      }
    }
  }
}

You should now see atla-mcp-server in the list of available MCP servers.

Contributing

Contributions are welcome! Please see the CONTRIBUTING.md file for details.

License

This project is licensed under the MIT License. See the LICENSE file for details.