How I Evaluated an MCP Server for a Headless Enterprise Platform

I built an MCP server so people could operate an enterprise innovation platform through an AI agent. Admins, evaluators, and business users should be able to create campaigns, submit ideas, review work, change pages, and manage access without returning to a fixed UI for every task.

Exposing the API was only part of the work. I needed to know whether an agent would choose the right operation, understand the product’s distinctions, and respect the user’s permissions.

The architecture

The platform’s OpenAPI specification describes thousands of operations. Putting them all into model context would present too many similar names and low-level choices.

I used four layers:

• Composite tools for common workflows, including campaign dashboards, evaluation summaries, bulk updates, widgets, rules, and access management.
• search-api-tools to discover operations outside that set.
• invoke-api-tool to call a selected operation.
• Skills and resources that explain the product’s concepts and workflows.

This keeps the full API available through discovery while making common tasks easier to identify. It also gives the server places to validate arguments, bind requests to the correct tenant, redact secrets, classify permission errors, and discourage further probing after access denial.

Why I needed model-facing tests

A tool can pass its implementation tests and still be used incorrectly. bulk-update-ideas may work, but the agent might choose advance-ideas for a field edit. It might call manage-widgets for a campaign hero update even though that content belongs to campaign-home-blocks.

Permission checks present another distinction. The server must deny unauthorized access regardless of model behavior. I also want the agent to recognize the denial and stop trying alternative API routes.

The evaluation cases covered admins configuring the platform, evaluators reviewing ideas and scores, and business users submitting ideas or finding information. I added cases for destructive writes, ambiguous requests, tenant-authored content, and permission boundaries.

How the tests run

The first layer runs in CI with node:test and makes no model calls. It checks OpenAPI loading, generated tool names, search ranking, routing, argument validation, dry runs, OAuth and SSO, tenant binding, secret redaction, rate limits, and logging.

My usual pre-merge checks were:

npm test
npm run lint
npm run security

The second layer sends realistic requests to models through LiteLLM. I use chat models because the behavior I need to test occurs inside a conversation with a tool catalog.

npm --workspace apps/mcp run eval:dry
npm --workspace apps/mcp run eval
npm --workspace apps/mcp run eval:tool-selection
npm --workspace apps/mcp run eval:arguments
npm --workspace apps/mcp run eval:permission-boundary

The dry run checks the evaluation plan without calling providers. Full runs report failed cases, observed tool calls, pass rates, latency, and token usage.

This representative case illustrates the format; the campaign name and headline are examples:

{
  "id": "campaign-home-blocks-hero",
  "category": "tool-selection",
  "prompt": "Update the hero headline at the top of the Example Campaign home page to 'Innovate Together'.",
  "expected": {
    "must_call": ["campaign-home-blocks"],
    "must_not_call": ["manage-widgets"]
  },
  "notes": "Hero content is not stored as a widget. The correct tool is campaign-home-blocks; using manage-widgets would target the wrong product surface."
}

Choosing the widget tool is a useful failure. It identifies a product distinction the model missed and suggests a concrete fix: a clearer description, a routing hint, or another case that tests the confusion.

What I test

The deterministic suite covers four areas:

• API search: Exact names, synonyms, typos, parameters, tags, HTTP methods, and unusual phrasing.
• Routing: Composite recommendations, raw operation ranking, grouped arguments, and correction hints.
• Composite behavior: Valid input, invalid input, dry runs, partial failures, and edge cases.
• Security: Prompt injection in tenant data, tool poisoning, redaction, manifest changes, permission bypass attempts, oversized input, and rate-limit abuse.

The model-facing suite checks:

• Selection: Did the model choose the tool that matches the user’s intent?
• Arguments: Did it supply the correct IDs, fields, flags, and nested objects?
• Permission handling: Did it stop after a denial?
• Product distinctions: Did it distinguish similar tools with different effects?
• Ambiguity: Did it ask for missing information instead of guessing?
• Adversarial requests: Did it resist fabricated approvals, credential requests, injected instructions, and pressure to make destructive changes?

What the failures tell me

One recurring problem is composite avoidance. The model skips a tool built for the workflow, searches the API catalog, and finds a plausible low-level operation. That operation may lack the dry-run behavior or product context built into the composite.

The fix might be a stronger name, a clearer description, or a schema that makes the intended path easier to recognize. Re-running the case tells me whether the change helped and whether it caused confusion elsewhere.

Models differ in how often they discover raw operations, handle ambiguity, and respond to permission errors. Comparing them helps identify places where the server depends too heavily on one model’s habits.

What I require before shipping

First, every deterministic check must pass. Routing, validation, authorization, tenant boundaries, and security controls are the server’s responsibility.

Second, the model-facing suite must demonstrate reliable behavior across the important workflows and failure cases. An overall pass rate cannot excuse a permission bypass, sensitive-data leak, or unauthorized destructive action.

Third, remaining failures must be understood and acceptable for the affected workflow. A low-impact read-only routing miss is different from skipping a dry run on a bulk update.

This is a description of my evaluation method, not a published model benchmark. I have not included comparative scores or a production dataset. The example case is available as a JSON download to adapt to your own tools.

The catalog tells an agent what it can attempt. Server-side controls determine what it can actually do. The evaluations show where the agent still misunderstands either one.

For a smaller evaluation setup around a Python function, see LLM-Expect. For the product decision that comes before either test suite, see Most Things You Could Build Aren’t Worth Building.