Mcp Server Evaluations

MCP Server Evaluations Skill

Systematically evaluate MCP servers to ensure they function correctly, handle errors gracefully, and meet quality standards.

Workflow

Phase 1: Environment Verification

1. Verify MCP server is running

```bash curl -s http://localhost:3030/health # Expected: 200 OK

curl -s -X POST http://localhost:3030/mcp \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","id":1,"method":"ping"}' # Expected: {"jsonrpc":"2.0","id":1,"result":{}} ```

Phase 2: Tool Discovery

1. List all available tools

``bash curl -X POST http://localhost:3030/mcp \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' ``

1. Verify tool completeness

• [ ] All OpenAPI operations exposed as tools
• [ ] Tool names follow consistent convention (e.g., getUsers, createOrder)
• [ ] Descriptions are clear and actionable
• [ ] Required vs optional parameters clearly marked
• [ ] Parameter types match OpenAPI schema

1. Document discovered tools — Create inventory of tools for systematic testing.

Phase 3: Functional Testing

For each discovered tool:

1. Basic functionality test

``bash curl -X POST http://localhost:3030/mcp \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "method": "tools/call", "params": { "name": "", "arguments": { } }, "id": 2 }' ``

1. Verify response structure

• [ ] Response contains expected data
• [ ] Data types match schema
• [ ] No unexpected null values
• [ ] Pagination works (if applicable)

1. Error handling test — Call with invalid/missing arguments:

``bash curl -X POST http://localhost:3030/mcp \ -H "Content-Type: application/json" \ -d '{ "jsonrpc": "2.0", "method": "tools/call", "params": { "name": "", "arguments": {} }, "id": 3 }' ``

1. Verify error response quality

• [ ] Error message is actionable
• [ ] Missing required parameters identified
• [ ] HTTP status codes propagated correctly

Phase 4: Question-Based Evaluation

Generate and test with realistic user questions:

1. Generate 10+ test questions covering:

• Simple single-tool queries
• Multi-step workflows requiring multiple tools
• Edge cases (empty results, large datasets)
• Error scenarios (invalid IDs, unauthorized access)

1. Execute each question through MCP client or Inspector

1. Score responses using evaluation criteria:

• Correctness: Does the answer match expected result?
• Completeness: Is all relevant information included?
• Clarity: Is the response well-structured?
• Performance: Response time within acceptable limits?

Phase 5: Quality Scoring

Calculate overall quality score:

| Category | Weight | Criteria | |----------|--------|----------| | Tool Discovery | 20% | All operations exposed, proper naming | | Basic Functionality | 30% | Valid inputs return correct responses | | Error Handling | 20% | Graceful errors with actionable messages | | Question Accuracy | 20% | Test questions answered correctly | | Performance | 10% | Response times /dev/null && echo "✓ Ping OK" || echo "✗ Ping FAILED"

3. Tools list

curl -s -X POST http://localhost:3030/mcp \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' | jq '.result.tools | length' | xargs -I {} echo "✓ {} tools discovered"

4. Sample tool call (adjust tool name and args)

curl -s -X POST http://localhost:3030/mcp \ -H "Content-Type: application/json" \ -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"listPets","arguments":{}},"id":2}' | jq '.result' > /dev/null && echo "✓ Tool call OK" || echo "✗ Tool call FAILED"

## Test Question Templates

Use these patterns to generate effective test questions:

1. **List/Query**: "Show me all [resources] that match [criteria]"
2. **Get Details**: "What are the details of [resource] with ID [id]?"
3. **Create**: "Create a new [resource] with [properties]"
4. **Update**: "Update [resource] [id] to change [field] to [value]"
5. **Delete**: "Remove [resource] with ID [id]"
6. **Aggregate**: "How many [resources] exist with [status]?"
7. **Search**: "Find [resources] where [field] contains [term]"
8. **Workflow**: "Create a [resource], then update it, then list all"

## References

For detailed documentation:
- [references/mcp-inspector-guide.md](references/mcp-inspector-guide.md) — Inspector setup & usage
- [references/evaluation-criteria.md](references/evaluation-criteria.md) — Quality metrics & scoring
- [references/question-templates.md](references/question-templates.md) — Test question generation

## Example: Petstore API Evaluation

```bash
# 1. Run health checks
curl -s http://localhost:3030/health
curl -s -X POST http://localhost:3030/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"ping"}' | jq -e '.jsonrpc == "2.0" and .result' > /dev/null && echo "✓ Ping OK" || echo "✗ Ping FAILED"

# 2. Tool discovery
curl -s -X POST http://localhost:3030/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' | jq '.result.tools'

# 3. Test questions:
# - "List all available pets"
# - "Show details of pet with ID 1"
# - "Find pets with status 'available'"
# - "Create a new pet named 'Fluffy'"

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mcp-com-ai
• Source: mcp-com-ai/mcp-server-evaluations-skills
• License: MIT
• Homepage: https://mcp.com.ai

Install and usage instructions live in the source repository linked above.