MCPToolBenchPP

MCPToolBench++: AI Agent MCP Model Context Protocol MCP Tool Use Benchmark

GitHub|HuggingFace|ModelScope

[](https://www.deepnlp.org/store/ai-agent/benchmark/pub-mcp-tool-bench/mcptoolbenchp)[](https://www.deepnlp.org/store/ai-agent/benchmark/pub-mcp-tool-bench/mcptoolbenchpp)

Introduction

MCPToolBench++ is a large-scale, multi-domain AI Agent Tool Use Benchmark. As of July 2025, this benchmark includes over 4k+ MCP Servers from more than 45 categories collected from the MCP and GitHub communities. The dataset comprises both single-step and multi-step tool calls across different categories.

Notice: This repo benchmark is still WIP and more domain dataset will be released.

News

[2025-10-08] Add OneKey MCP Router Support in MCP Benchmark: Use OneKey MCP Proxy Router to simplify registration and use one access key to various commercial MCPs. See detailed list in GitHub (e.g. Google Maps,Google Search, Perplexity, Firecrawl, etc) and generate OneKey from website.

Performance Leaderboard

| | Browser | | File System | | Search | | | --- | ------ | ---- | ----| ---- | --- | --- | | | AST | Pass@1 | AST | Pass@1 | AST | Pass@1 | | GPT4o | 0.6524 | 0.2182 | 0.8863 | 0.8232 | 0.5200 | 0.4720 | | Qwen2.5 Max | 0.7262 | 0.2749 | 0.9419 | 0.8871 | 0.6280 | 0.4600 | | Claude Sonnet 3.7 | 0.6503 | 0.1840 | 0.8415 | 0.8183 | 0.7280 | 0.6200 | | Kimi K2 Instruct | 0.8182 | 0.2524 | 0.9062 | 0.8772 | 0.7320 | 0.3680 | | Qwen3 Coder | 0.8866 | 0.2925 | 0.9080 | 0.8680 | 0.7180 | 0.5227 | | Claude Opus 4 | - | - | - | - | - | - | | Claude Sonnet 4 | - | - | - | - | - | - |

| | Map | | Pay | | Finance | | | --- | ------ | ---- | ----| ---- | --- | --- | | | AST | Pass@1 | AST | Pass@1 | AST | Pass@1 | | GPT4o | 0.6120 | 0.3616 | 0.7077 | 0.5742 | 0.7200 | 0.2889 | | Qwen2.5 Max | 0.7372 | 0.2272 | 0.6684 | 0.5277 | 0.7511 | 0.2556 | | Claude Sonnet | 0.5820 | 0.2748 | 0.7058 | 0.5574 | 0.7400 | 0.2311 | | Kimi K2 Instruct | 0.6088 | 0.2008 | 0.8071 | 0.6761 | 0.7156 | 0.2378 | | Qwen3 Coder | 0.7830 | 0.3054 | 0.7240 | 0.5440 | 0.7320 | 0.2860 | | Claude Opus 4 | - | - | - | - | - | - | | Claude Sonnet 4 | - | - | - | - | - | - |

Introduction

0. Dataset Overview

| Category | Number Instance | MCP Tool Count | Avg Tokens/Tool | Total Tokens | | ----- | ------ | ----- | ------ | ----- | | Browser | 187 | 32 | 107.44 | 3.4k | | File System | 241 | 11 | 143.82 | 1.6k | | Search | 181 | 5 | 555.6 | 2.8k | | Map | 500 | 32 | 401.28 | 13k | | Finance | 90 | 1 | 505.0 | 0.5k | | Pay | 310 | 6 | 656.5 | 3.9k | | Total | 1509 | 87 | 288.3 | 25k |

1. Browser

The browser subset evaluates models' ability to use the web browser, typical tools include puppeteernavigate, puppeteerscreenshot, puppeteerclick, playwrightscreenshot, playwright_navigate, etc. Agent models call the tools to nagivate to the URL, visit page, click on buttons, take screenshot of the webpage, etc.

Navigate to the Wikipedia website using browser and check its accessibility.

Setup MCP Servers and API

See [Browser Use MCP Setup](#1-browser-use-mcp-setup) for how to setup and start the servers.

Run Dataset

Once the servers are started, run below command to start evaluation.

## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/browser/browser_single_demo.json --category browser --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/browser/browser_0724_single_v3.json --category browser --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

2. File System

The file system mcp helps to manage your local file and directories, typical tools include: readfile/editFile/listdirectorywith_sizes/etc.

Read the contents of the files located at ./test_project_root/src/main.py and ./test_project_root/docs/README.md at the same time.

Provide a recursive tree view of the files and directories located at ./test_project_root/src.

Setup MCP Servers and API

See [File System MCP Setup](#2-file-system-mcp-setup) for how to setup and run MCP servers

Run Dataset

## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/file_system/filesystem_single_demo.json --category filesystem --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/file_system/filesystem_0723_single.json --category filesystem --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

3. Search

Find latest AI LLM and Agents related news on the web

The search mcp tools helps to search the web given user's query, typical servers and tools include google-web-search, google-image-search, tavily-search, tavily-extract, firecrawl-search, etc.

Setup MCP Servers and API

See [Search MCP Setup](#3-search-mcp-setup) for how to setup and run MCP servers

Run Dataset

## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/search/search_single_demo.json --category search --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
### Note Qwen doesn't allow tool to be named 'search'
python3 run.py --stage tool_call --input_file ./data/search/search_0725_single_v2_forqwen.json --category search --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

python3 run.py --stage tool_call --input_file ./data/search/search_0725_single_v2.json --category search --model gpt4o --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

4. Map

Map Subsets support multilingual queries to search places and routes.

# english
What is the current weather in Tokyo and the weather forecast for the next 5 days?
Find popular Japanese restaurants in Houston.

# french
¿Cuál es la mejor ruta para ir en bicicleta desde Tokio hasta la Torre de Tokio?

# russian
Каковы координаты адреса Санкт-Петербург, Невский проспект, 1?

Setup MCP Servers and API

See [Map MCP Setup](#4-map-mcp-setup) for how to setup and run MCP servers

## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/map/map_single_demo.json --category map --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus 

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/map/map_0717_single_multi_lang_500.json --category map --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

5. Pay

The pay subdataset evaluates pay related MCP servers(paypal/alipay/etc), typical tools include createinvoice, createproducts, etc.

Create an invoice for Tech Solutions Inc. for a Consultation Service costing 150.00 USD.

Setup MCP Servers and API

The paypal and alipay MCPs are free to use, but you needs to register and setup paypal/alipay sandbox access_key with development account and setup config in mcp-marketplace UI.

See [Pay MCP Setup](#5-pay-mcp-setup) for how to setup and run MCP servers

Run Dataset

## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/pay/pay_single_demo.json --category pay --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/pay/pay_0723_single.json --category pay --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

6. Finance

What is the current stock price of Tesla in the US market?
What is the current stock price and market capitalization of Shell in the London Stock Exchange market?

Setup MCP Servers and API

See [Finance MCP Setup](#6-finance-mcp-setup) for how to setup and run MCP servers

Run Dataset

## Test Run 1 instance
python3 run.py --stage tool_call --input_file ./data/finance/finance_single_demo.json --category finance --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

## Run the Dataset
python3 run.py --stage tool_call --input_file ./data/finance/finance_0724_single_v3.json --category finance --model qwen3-coder-plus --pass_k 1,3 --evaluation_trial_per_task 5 --llm_as_judge_model qwen-plus

Tutorial On How To Setup Environment and Use the MCP Benchmark

0. Setup

Install

| Method | Description | |-------------------------------------|---------------------------------------------------------| | Install pypi package and cmd mcpm | Use CLI to start MCP Client using local mcp_config.json | | Install from source | - |

Install from pypi mcpm

Install from pypi and start the mcp-client with config

## Basic Usage MCP Index and Search
pip install mcp-marketplace

## MCPClient Supports User Defined mcp_config.json, Just Like Other Clients Cursor/Claude
pip install mcp-marketplace[mcp_tool_use] 

The CLI 'mcpm' will be in python path

## default
mcpm run
## set client and port
mcpm run --host 0.0.0.0 --port 5000

## Use Local Config File
cd python/tests
mcpm run --port 5000 --config "./mcp_config_onekey.json"
mcpm run --port 5000 --config "./mcp_config.json"

Then you can visit http://0.0.0.0:5000 for web console and http://0.0.0.0:5000/mcp for mcp management

Install from source

Clone the repo https://github.com/mcp-tool-bench/MCPToolBenchPP

## dataset
git clone https://github.com/mcp-tool-bench/MCPToolBenchPP

## clone the mcp client to execute tool call
cd ./MCPToolBenchPP/mcp
## path: ./MCPToolBenchPP/mcp/mcp-marketplace
git clone https://github.com/aiagenta2z/mcp-marketplace

Requirements

pip install python-dotenv fastapi uvicorn[standard] asyncio openai anthropic mcp mcp-marketplace uuid httpx aiofiles anthropic Jinja2

Setup Env Keys

Edit .env file

# ./MCPToolBenchPP/.env
vim .env

QWEN_API_KEY=...
OPENAI_API_KEY=...
ANTHROPIC_API_KEY=...
GOOGLE_API_KEY=...
MISTRAL_API_KEY=...
KIMI_API_KEY=...
DEEPNLP_ONEKEY_ROUTER_ACCESS=....

MCP OneKey Router

See list of support remote pure http MCP by OneKey Router GitHub, and generate Keys in MCP OneKey Router Website website.

Use OneKey MCP Router to simplified MCP access key registration and use one key to access Google Maps,Google Search,Tavily,Firecrawl,Bing Search,Bing Image Search and more MCPs for Benchmarking and daily use.

Google Maps Example:

export DEEPNLP_ONEKEY_ROUTER_ACCESS={your_access_key} 

## BETA_TEST_KEY_OCT_2025

{
	"mcpServers": {
		"deepnlp-onekey-google-maps": {
			"url": "https://agent.deepnlp.org/mcp?server_name=google-maps&onekey={BETA_TEST_KEY_OCT_2025}"
		},
	}
}

Setup Client MCP Marketplace Admin and Start Servers

Install requirements and follow the steps in https://github.com/aiagenta2z/mcp-marketplace

Start the Server

cd ./mcp/mcp-marketplace/app/mcp_tool_use
uvicorn src.app:app --port 5000

Visit http://127.0.0.1:5000/mcp

Setup Config

Setup the mcpconfig.json by visiting http://localhost:5000/mcp/config. Change Configuration during initialization MCPINITAUTOENABLE=True to Start all servers from mcp_config.json

vim ./mcp/mcp-marketplace/app/mcp_tool_use/src/constants.py
# set the variables
MCP_INIT_AUTO_ENABLE=True

Manage the MCP Configs Started at ./mcp/mcp-marketplace/app/mcptooluse/data/mcp/config/mcp_config.json

Restart Open MCP Marketplace Client

To make the config valid, you need to restart the server.

Visit http://127.0.0.1:5000/mcp to see if the servers are started.

1. Run Evaluation

Run the browser use dataset using the qwen3-coder-plus model

Start Open MCP Marketplace Client to Execute Tool Call

cd ./mcp/mcp-marketplace/app/mcp_tool_use
uvicorn src.app:app --port 5000

Run the scripts

| parameter | description | | ---- | ---- | | inputfile | the json file containing examples | | category | category of the sub dataset | | model | the code for the LLM model to evaluate, see ./MCPToolBenchPP/src/mcptoolbench/globalvariables.py and ./mcptoolbench/modelutils/modelprovider.py for more details. | | stage | 'demo', 'generation', 'toolcall', 'all' | | metric | e.g. pass@k | | passk | e.g. "1,3" comma separated pass@k value list. | | evaluationtrialpertask | default to 5 | | llmasjudgemodel | the check the AST score of parameters, LLM as a judge is needed because some tools such as "search" have rewritten query, so exact match check is not possible. |

## Test Run 1 instance, Evaluate qwen3-coder-plus model and use qwen-plus as llm-as-judge
python3 run.py --stage tool_call --input_file ./data/browser/browser_single_demo.json --category browser --model qwen3-coder-plus --pass_k 1 --evaluation_trial_per_task 1 --llm_as_judge_model qwen-plus

Expected Correct Output

=== Running Parameters ===
input_file: ./data/browser/browser_single_demo.json
category: browser
model: qwen3-coder-plus
stage: tool_call
metric: pass_k
pass_k: 1
agent: base
mcp_config: mcp_marketplace/mcp_config.json
data_version: v0
log_file: None
evaluation_trial_per_task: 1
llm_as_judge_model: qwen-plus
===============
==================================================
Executing tool_call stage: tool calling and evaluation
==================================================

【Step 1】Tool Calling and Evaluation
------------------------------
Validation passed: EVALUATION_TRIAL_PER_TASK=1, max_pass_k=1
Loaded 1 instances of data files
Starting new benchmark run (log file: /Users/xichen.dxc/Desktop/project/gitlab/MCPToolBenchPP/logs/browser/browser_single_demo_20250802_225043.json)

Processing 1 remaining tasks...
Processing tasks:   0%|                                                                                                                        | 0/1 [00:00 and type 
Processing tasks: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:03test_project_root to the working directory or create a new project test_project_root from scratch
to run local file testing.

under the root directory cd ./MCPToolBenchPP

mv ./data/filesystem/testprojectroot ./mcp/mcp-marketplace/app/mcptool_use

or

mkdir ./mcp/mcp-marketplace/app/mcptooluse/testprojectroot create dummy files similar to ./data/filesystem/testprojectroot

workspaceFolder should be a absolute path.e.g. : /path/to/folder/test_project_root

workspaceFolder=/{absolutepathtoMCPToolBenchPP}/mcp/mcp-marketplace/app/mcptool_use

vim ./mcp/mcp-marketplace/app/mcptooluse/data/mcp/config/mcp_config.json

And add the below config, remember to use absolute path ${workspaceFolder}

{ "mcpServers": { "filesystem": { "command": "npx", "args": [ "-y", "@modelcontextprotocol/server-filesystem", "${workspaceFolder}" ] } } }

#### Start Server and Curl if Setup

Endpoint: http://127.0.0.1:5000/api/query

curl -X POST -H "Content-Type: application/json" -d '{ "serverid": "filesystem", "toolname": "list_directo

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mcp-tool-bench
• Source: mcp-tool-bench/MCPToolBenchPP
• License: MIT
• Homepage: https://arxiv.org/abs/2508.07575

Install and usage instructions live in the source repository linked above.