# MCPToolBenchPP

> MCPToolBench++ MCP Model Context Protocol Tool Use Benchmark on AI Agent and Model Tool Use Ability

- **Type:** MCP server
- **Install:** `agentstack add mcp-mcp-tool-bench-mcptoolbenchpp`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [mcp-tool-bench](https://agentstack.voostack.com/s/mcp-tool-bench)
- **Installs:** 0
- **Category:** [Integrations](https://agentstack.voostack.com/c/integrations)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [mcp-tool-bench](https://github.com/mcp-tool-bench)
- **Source:** https://github.com/mcp-tool-bench/MCPToolBenchPP
- **Website:** https://arxiv.org/abs/2508.07575

## Install

```sh
agentstack add mcp-mcp-tool-bench-mcptoolbenchpp
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# MCPToolBench++: AI Agent MCP Model Context Protocol MCP Tool Use Benchmark

[GitHub](https://github.com/mcp-tool-bench/MCPToolBenchPP)|[HuggingFace](https://huggingface.co/datasets/MCPToolBench/MCPToolBenchPP)|[ModelScope](https://www.modelscope.cn/datasets/mcptoolbench/MCPToolBenchPP)

[](https://www.deepnlp.org/store/ai-agent/benchmark/pub-mcp-tool-bench/mcptoolbenchp)[](https://www.deepnlp.org/store/ai-agent/benchmark/pub-mcp-tool-bench/mcptoolbenchpp) 

**Introduction**

MCPToolBench++ is a large-scale, multi-domain AI Agent Tool Use Benchmark. As of July 2025, this benchmark includes over 4k+ MCP Servers from more than 45 categories collected from the MCP and GitHub communities. The dataset comprises both single-step and multi-step tool calls across different categories.

Notice: This repo benchmark is still WIP and more domain dataset will be released.

**News**

[2025-10-08] Add OneKey MCP Router Support in MCP Benchmark: Use OneKey MCP Proxy Router to simplify registration and use one access key to various commercial MCPs. See detailed list in [GitHub](https://github.com/aiagenta2z/onekey_mcp_router)  (e.g. Google Maps,Google Search, Perplexity, Firecrawl, etc) and generate OneKey from [website](https://www.deepnlp.org/agent/onekey-mcp-router). 

## Performance Leaderboard

|     | Browser |      | File System |      | Search | |
| --- | ------  | ---- | ----| ---- |  --- | ---  |
|     | AST | Pass@1 | AST | Pass@1 |  AST | Pass@1  |
| GPT4o | 0.6524  |  0.2182 | 0.8863 | 0.8232 | 0.5200 | 0.4720 |
| Qwen2.5 Max | 0.7262 | 0.2749 | 0.9419 | 0.8871 | 0.6280 | 0.4600 |
| Claude Sonnet 3.7 | 0.6503 | 0.1840 | 0.8415 | 0.8183 | 0.7280 | 0.6200 |
| Kimi K2 Instruct | 0.8182 | 0.2524 | 0.9062 | 0.8772 | 0.7320 | 0.3680 |
| Qwen3 Coder | 0.8866 | 0.2925 | 0.9080 | 0.8680 | 0.7180 | 0.5227 |
| Claude Opus 4 | - | - | - | - | - | - |
| Claude Sonnet 4 | - | - | - | - | - | - |

|     | Map |      | Pay |      | Finance | |
| --- | ------  | ---- | ----| ---- |  --- | ---  |
|     | AST | Pass@1 | AST | Pass@1 |  AST | Pass@1  |
| GPT4o | 0.6120 | 0.3616 | 0.7077 | 0.5742 | 0.7200 | 0.2889 |
| Qwen2.5 Max | 0.7372 | 0.2272 | 0.6684 | 0.5277 | 0.7511 | 0.2556 |
| Claude Sonnet | 0.5820 | 0.2748 | 0.7058 | 0.5574 | 0.7400 | 0.2311 |
| Kimi K2 Instruct | 0.6088 | 0.2008 | 0.8071 | 0.6761 | 0.7156 | 0.2378 |
| Qwen3 Coder | 0.7830 | 0.3054 | 0.7240 | 0.5440 | 0.7320 | 0.2860 |
| Claude Opus 4 | - | - | - | - | - | - |
| Claude Sonnet 4 | - | - | - | - | - | - |

## Introduction

### 0. Dataset Overview

|  Category  | Number Instance | MCP Tool Count | Avg Tokens/Tool | Total Tokens  |
| ----- | ------  | ----- | ------  | ----- |
| Browser |  187 | 32 |  107.44 | 3.4k |
| File System | 241 | 11 |  143.82 |  1.6k |
| Search |  181 |  5 | 555.6 | 2.8k |
| Map | 500 | 32 | 401.28 | 13k |
| Finance | 90 | 1 | 505.0  | 0.5k |
| Pay | 310 | 6 | 656.5 | 3.9k |
| Total | 1509 | 87 | 288.3 | 25k |

### 1. Browser

The browser subset evaluates models' ability to use the web browser, typical tools include puppeteer_navigate, puppeteer_screenshot,  puppeteer_click, playwright_screenshot, playwright_navigate, etc. Agent models call the tools to nagivate to the URL, visit page, click on buttons, take screenshot of the webpage, etc.

```
Navigate to the Wikipedia website using browser and check its accessibility.
```

#### Setup MCP Servers and API

See [Browser Use MCP Setup](#1-browser-use-mcp-setup) for how to setup and start the servers.

#### Run Dataset

Once the servers are started, run below command to start evaluation.

```
## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/browser/browser_single_demo.json --category browser --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/browser/browser_0724_single_v3.json --category browser --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

```

### 2. File System

The file system mcp helps to manage your local file and directories, typical tools include: read_file/edit_File/list_directory_with_sizes/etc.

```
Read the contents of the files located at ./test_project_root/src/main.py and ./test_project_root/docs/README.md at the same time.

Provide a recursive tree view of the files and directories located at ./test_project_root/src.
```

#### Setup MCP Servers and API
See [File System MCP Setup](#2-file-system-mcp-setup) for how to setup and run MCP servers

#### Run Dataset

```
## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/file_system/filesystem_single_demo.json --category filesystem --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/file_system/filesystem_0723_single.json --category filesystem --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

```

### 3. Search

```
Find latest AI LLM and Agents related news on the web
```

The search mcp tools helps to search the web given user's query, typical servers and tools include google-web-search, google-image-search, tavily-search, tavily-extract, firecrawl-search, etc.

#### Setup MCP Servers and API
See [Search MCP Setup](#3-search-mcp-setup) for how to setup and run MCP servers

#### Run Dataset

```
## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/search/search_single_demo.json --category search --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
### Note Qwen doesn't allow tool to be named 'search'
python3 run.py --stage tool_call --input_file ./data/search/search_0725_single_v2_forqwen.json --category search --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

python3 run.py --stage tool_call --input_file ./data/search/search_0725_single_v2.json --category search --model gpt4o --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

```

### 4. Map

Map Subsets support multilingual queries to search places and routes.

```
# english
What is the current weather in Tokyo and the weather forecast for the next 5 days?
Find popular Japanese restaurants in Houston.

# french
¿Cuál es la mejor ruta para ir en bicicleta desde Tokio hasta la Torre de Tokio?

# russian
Каковы координаты адреса Санкт-Петербург, Невский проспект, 1?

```

#### Setup MCP Servers and API
See [Map MCP Setup](#4-map-mcp-setup) for how to setup and run MCP servers

```
## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/map/map_single_demo.json --category map --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus 

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/map/map_0717_single_multi_lang_500.json --category map --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

```

### 5. Pay

The pay subdataset evaluates pay related MCP servers(paypal/alipay/etc), typical tools include create_invoice, create_products, etc.

```
Create an invoice for Tech Solutions Inc. for a Consultation Service costing 150.00 USD.
```

#### Setup MCP Servers and API

The paypal and alipay MCPs are free to use, but you needs to register and setup paypal/alipay sandbox access_key with development account and setup config in mcp-marketplace UI.

See [Pay MCP Setup](#5-pay-mcp-setup) for how to setup and run MCP servers

#### Run Dataset

```
## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/pay/pay_single_demo.json --category pay --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/pay/pay_0723_single.json --category pay --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

```

### 6. Finance

```
What is the current stock price of Tesla in the US market?
What is the current stock price and market capitalization of Shell in the London Stock Exchange market?
```

#### Setup MCP Servers and API

See [Finance MCP Setup](#6-finance-mcp-setup) for how to setup and run MCP servers

#### Run Dataset

```
## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/finance/finance_single_demo.json --category finance --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/finance/finance_0724_single_v3.json --category finance --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

```

## Tutorial On How To Setup Environment and Use the MCP Benchmark

### 0. Setup

#### Install

| Method                              | Description                                             |
|-------------------------------------|---------------------------------------------------------|
| Install pypi package and cmd `mcpm` | Use CLI to start MCP Client using local mcp_config.json |
| Install from source                 | -                                                       |

**Install from pypi `mcpm`**

Install from pypi and start the mcp-client with config
```
## Basic Usage MCP Index and Search
pip install mcp-marketplace

## MCPClient Supports User Defined mcp_config.json, Just Like Other Clients Cursor/Claude
pip install mcp-marketplace[mcp_tool_use] 
```

The CLI 'mcpm' will be in python path
```
## default
mcpm run
## set client and port
mcpm run --host 0.0.0.0 --port 5000

## Use Local Config File
cd python/tests
mcpm run --port 5000 --config "./mcp_config_onekey.json"
mcpm run --port 5000 --config "./mcp_config.json"
```
Then you can visit http://0.0.0.0:5000 for web console and http://0.0.0.0:5000/mcp for mcp management

**Install from source**

Clone the repo https://github.com/mcp-tool-bench/MCPToolBenchPP

```
## dataset
git clone https://github.com/mcp-tool-bench/MCPToolBenchPP

## clone the mcp client to execute tool call
cd ./MCPToolBenchPP/mcp
## path: ./MCPToolBenchPP/mcp/mcp-marketplace
git clone https://github.com/aiagenta2z/mcp-marketplace

```

### Requirements
```
pip install python-dotenv fastapi uvicorn[standard] asyncio openai anthropic mcp mcp-marketplace uuid httpx aiofiles anthropic Jinja2
```

#### Setup Env Keys
Edit .env file
```
# ./MCPToolBenchPP/.env
vim .env
```

```txt
QWEN_API_KEY=...
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
GOOGLE_API_KEY=...
MISTRAL_API_KEY=...
KIMI_API_KEY=...
DEEPNLP_ONEKEY_ROUTER_ACCESS=....
```

**MCP OneKey Router** 

See list of support remote pure http MCP by OneKey Router [GitHub](https://github.com/aiagenta2z/onekey_mcp_router), and generate Keys in [MCP OneKey Router Website](https://www.deepnlp.org/agent/onekey-mcp-router) website.

Use OneKey MCP Router to simplified MCP access key registration and use one key to access Google Maps,Google Search,Tavily,Firecrawl,Bing Search,Bing Image Search and more MCPs for Benchmarking and daily use.

Google Maps Example:

```
export DEEPNLP_ONEKEY_ROUTER_ACCESS={your_access_key} 

## BETA_TEST_KEY_OCT_2025
```

```txt
{
	"mcpServers": {
		"deepnlp-onekey-google-maps": {
			"url": "https://agent.deepnlp.org/mcp?server_name=google-maps&onekey={BETA_TEST_KEY_OCT_2025}"
		},
	}
}
```

#### Setup Client MCP Marketplace Admin and Start Servers

Install requirements and follow the steps in https://github.com/aiagenta2z/mcp-marketplace

**Start the Server**
```
cd ./mcp/mcp-marketplace/app/mcp_tool_use
uvicorn src.app:app --port 5000
```
Visit http://127.0.0.1:5000/mcp 

**Setup Config**

Setup the mcp_config.json by visiting http://localhost:5000/mcp/config.
Change Configuration during initialization MCP_INIT_AUTO_ENABLE=True to Start all servers from mcp_config.json

```
vim ./mcp/mcp-marketplace/app/mcp_tool_use/src/constants.py
# set the variables
MCP_INIT_AUTO_ENABLE=True
```

Manage the MCP Configs Started at ./mcp/mcp-marketplace/app/mcp_tool_use/data/mcp/config/mcp_config.json

**Restart Open MCP Marketplace Client**

To make the config valid, you need to restart the server.

Visit http://127.0.0.1:5000/mcp to see if the servers are started.

### 1. Run Evaluation 

Run the browser use dataset using the qwen3-coder-plus model

####  Start Open MCP Marketplace Client to Execute Tool Call

```
cd ./mcp/mcp-marketplace/app/mcp_tool_use
uvicorn src.app:app --port 5000
```

####  Run the scripts

| parameter | description |
|  ---- | ---- |
| input_file | the json file containing examples |
| category | category of the sub dataset |
| model | the code for the LLM model to evaluate, see ./MCPToolBenchPP/src/mcp_tool_bench/global_variables.py and ./mcp_tool_bench/model_utils/model_provider.py for more details. |
| stage | 'demo', 'generation', 'tool_call', 'all' |
| metric | e.g. pass@k |
| pass_k | e.g. "1,3" comma separated pass@k value list. |
| evaluation_trial_per_task | default to 5 |
| llm_as_judge_model | the check the AST score of parameters, LLM as a judge is needed because some tools such as "search" have rewritten query, so exact match check is not possible. |

```txt
## Test Run 1 instance, Evaluate qwen3-coder-plus model and use qwen-plus as llm-as-judge
python3 run.py --stage tool_call --input_file ./data/browser/browser_single_demo.json --category browser --model qwen3-coder-plus --pass_k 1 --evaluation_trial_per_task 1 --llm_as_judge_model qwen-plus

```

**Expected Correct Output**
```
=== Running Parameters ===
input_file: ./data/browser/browser_single_demo.json
category: browser
model: qwen3-coder-plus
stage: tool_call
metric: pass_k
pass_k: 1
agent: base
mcp_config: mcp_marketplace/mcp_config.json
data_version: v0
log_file: None
evaluation_trial_per_task: 1
llm_as_judge_model: qwen-plus
===============
==================================================
Executing tool_call stage: tool calling and evaluation
==================================================

【Step 1】Tool Calling and Evaluation
------------------------------
Validation passed: EVALUATION_TRIAL_PER_TASK=1, max_pass_k=1
Loaded 1 instances of data files
Starting new benchmark run (log file: /Users/xichen.dxc/Desktop/project/gitlab/MCPToolBenchPP/logs/browser/browser_single_demo_20250802_225043.json)

Processing 1 remaining tasks...
Processing tasks:   0%|                                                                                                                        | 0/1 [00:00 and type 
Processing tasks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03test_project_root to the working directory or create a new project test_project_root from scratch
to run local file testing.

```
## under the root directory cd ./MCPToolBenchPP

mv ./data/file_system/test_project_root ./mcp/mcp-marketplace/app/mcp_tool_use

or

mkdir ./mcp/mcp-marketplace/app/mcp_tool_use/test_project_root 
create dummy files similar to ./data/filesystem/test_project_root
```

workspaceFolder should be a absolute path.e.g. : /path/to/folder/test_project_root

```
workspaceFolder=/{absolute_path_to_MCPToolBenchPP}/mcp/mcp-marketplace/app/mcp_tool_use

vim ./mcp/mcp-marketplace/app/mcp_tool_use/data/mcp/config/mcp_config.json

```

And add the below config, remember to use absolute path ${workspaceFolder}

```
{
  "mcpServers": {
      "filesystem": {
        "command": "npx",
        "args": [
          "-y",
          "@modelcontextprotocol/server-filesystem",
          "${workspaceFolder}"
        ]
      }
  }
}

```

#### Start Server and Curl if Setup

Endpoint: http://127.0.0.1:5000/api/query
```
curl -X POST -H "Content-Type: application/json" -d '{
    "server_id": "filesystem",
    "tool_name": "list_directo

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [mcp-tool-bench](https://github.com/mcp-tool-bench)
- **Source:** [mcp-tool-bench/MCPToolBenchPP](https://github.com/mcp-tool-bench/MCPToolBenchPP)
- **License:** MIT
- **Homepage:** https://arxiv.org/abs/2508.07575

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-mcp-tool-bench-mcptoolbenchpp
- Seller: https://agentstack.voostack.com/s/mcp-tool-bench
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.