# Mcp Server Evaluations

> Test MCP servers for quality and reliability. Verify tool functionality, test error handling, generate tests, and assess response quality with no dependencies other than curl. Use this when validating MCP server implementations, testing OpenAPI-to-MCP conversions, or assessing API tool quality.

- **Type:** Skill
- **Install:** `agentstack add skill-mcp-com-ai-mcp-server-evaluations-skills-mcp-server-evaluations-skills`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [mcp-com-ai](https://agentstack.voostack.com/s/mcp-com-ai)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [mcp-com-ai](https://github.com/mcp-com-ai)
- **Source:** https://github.com/mcp-com-ai/mcp-server-evaluations-skills
- **Website:** https://mcp.com.ai

## Install

```sh
agentstack add skill-mcp-com-ai-mcp-server-evaluations-skills-mcp-server-evaluations-skills
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# MCP Server Evaluations Skill

Systematically evaluate MCP servers to ensure they function correctly, handle errors gracefully, and meet quality standards.

## Workflow

### Phase 1: Environment Verification

1. **Verify MCP server is running**
   ```bash
   curl -s http://localhost:3030/health
   # Expected: 200 OK
   
   curl -s -X POST http://localhost:3030/mcp \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","id":1,"method":"ping"}'
   # Expected: {"jsonrpc":"2.0","id":1,"result":{}}
   ```

### Phase 2: Tool Discovery

1. **List all available tools**
   ```bash
   curl -X POST http://localhost:3030/mcp \
     -H "Content-Type: application/json" \
     -d '{"jsonrpc":"2.0","method":"tools/list","id":1}'
   ```

2. **Verify tool completeness**
   - [ ] All OpenAPI operations exposed as tools
   - [ ] Tool names follow consistent convention (e.g., `getUsers`, `createOrder`)
   - [ ] Descriptions are clear and actionable
   - [ ] Required vs optional parameters clearly marked
   - [ ] Parameter types match OpenAPI schema

3. **Document discovered tools** — Create inventory of tools for systematic testing.

### Phase 3: Functional Testing

For each discovered tool:

1. **Basic functionality test**
   ```bash
   curl -X POST http://localhost:3030/mcp \
     -H "Content-Type: application/json" \
     -d '{
       "jsonrpc": "2.0",
       "method": "tools/call",
       "params": {
         "name": "",
         "arguments": {  }
       },
       "id": 2
     }'
   ```

2. **Verify response structure**
   - [ ] Response contains expected data
   - [ ] Data types match schema
   - [ ] No unexpected null values
   - [ ] Pagination works (if applicable)

3. **Error handling test** — Call with invalid/missing arguments:
   ```bash
   curl -X POST http://localhost:3030/mcp \
     -H "Content-Type: application/json" \
     -d '{
       "jsonrpc": "2.0",
       "method": "tools/call",
       "params": {
         "name": "",
         "arguments": {}
       },
       "id": 3
     }'
   ```

4. **Verify error response quality**
   - [ ] Error message is actionable
   - [ ] Missing required parameters identified
   - [ ] HTTP status codes propagated correctly

### Phase 4: Question-Based Evaluation

Generate and test with realistic user questions:

1. **Generate 10+ test questions** covering:
   - Simple single-tool queries
   - Multi-step workflows requiring multiple tools
   - Edge cases (empty results, large datasets)
   - Error scenarios (invalid IDs, unauthorized access)

2. **Execute each question** through MCP client or Inspector

3. **Score responses** using evaluation criteria:
   - **Correctness**: Does the answer match expected result?
   - **Completeness**: Is all relevant information included?
   - **Clarity**: Is the response well-structured?
   - **Performance**: Response time within acceptable limits?

### Phase 5: Quality Scoring

Calculate overall quality score:

| Category | Weight | Criteria |
|----------|--------|----------|
| Tool Discovery | 20% | All operations exposed, proper naming |
| Basic Functionality | 30% | Valid inputs return correct responses |
| Error Handling | 20% | Graceful errors with actionable messages |
| Question Accuracy | 20% | Test questions answered correctly |
| Performance | 10% | Response times  /dev/null && echo "✓ Ping OK" || echo "✗ Ping FAILED"

# 3. Tools list
curl -s -X POST http://localhost:3030/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' | jq '.result.tools | length' | xargs -I {} echo "✓ {} tools discovered"

# 4. Sample tool call (adjust tool name and args)
curl -s -X POST http://localhost:3030/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/call","params":{"name":"listPets","arguments":{}},"id":2}' | jq '.result' > /dev/null && echo "✓ Tool call OK" || echo "✗ Tool call FAILED"
```

## Test Question Templates

Use these patterns to generate effective test questions:

1. **List/Query**: "Show me all [resources] that match [criteria]"
2. **Get Details**: "What are the details of [resource] with ID [id]?"
3. **Create**: "Create a new [resource] with [properties]"
4. **Update**: "Update [resource] [id] to change [field] to [value]"
5. **Delete**: "Remove [resource] with ID [id]"
6. **Aggregate**: "How many [resources] exist with [status]?"
7. **Search**: "Find [resources] where [field] contains [term]"
8. **Workflow**: "Create a [resource], then update it, then list all"

## References

For detailed documentation:
- [references/mcp-inspector-guide.md](references/mcp-inspector-guide.md) — Inspector setup & usage
- [references/evaluation-criteria.md](references/evaluation-criteria.md) — Quality metrics & scoring
- [references/question-templates.md](references/question-templates.md) — Test question generation

## Example: Petstore API Evaluation

```bash
# 1. Run health checks
curl -s http://localhost:3030/health
curl -s -X POST http://localhost:3030/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"ping"}' | jq -e '.jsonrpc == "2.0" and .result' > /dev/null && echo "✓ Ping OK" || echo "✗ Ping FAILED"

# 2. Tool discovery
curl -s -X POST http://localhost:3030/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"tools/list","id":1}' | jq '.result.tools'

# 3. Test questions:
# - "List all available pets"
# - "Show details of pet with ID 1"
# - "Find pets with status 'available'"
# - "Create a new pet named 'Fluffy'"
```

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [mcp-com-ai](https://github.com/mcp-com-ai)
- **Source:** [mcp-com-ai/mcp-server-evaluations-skills](https://github.com/mcp-com-ai/mcp-server-evaluations-skills)
- **License:** MIT
- **Homepage:** https://mcp.com.ai

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-mcp-com-ai-mcp-server-evaluations-skills-mcp-server-evaluations-skills
- Seller: https://agentstack.voostack.com/s/mcp-com-ai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.