# Mcp Bigquery Evals

> Read-only BigQuery MCP server with mandatory dry-run cost caps, agent-friendly structured errors, and a Spider/BIRD-style NL-to-SQL eval harness.

- **Type:** MCP server
- **Install:** `agentstack add mcp-umarfarook1-mcp-bigquery-evals`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Umarfarook1](https://agentstack.voostack.com/s/umarfarook1)
- **Installs:** 0
- **Category:** [Databases](https://agentstack.voostack.com/c/databases)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Umarfarook1](https://github.com/Umarfarook1)
- **Source:** https://github.com/Umarfarook1/mcp-bigquery-evals
- **Website:** https://pypi.org/project/mcp-bigquery-evals/

## Install

```sh
agentstack add mcp-umarfarook1-mcp-bigquery-evals
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# mcp-bigquery-evals

**The BigQuery MCP server with mandatory cost guardrails and a measurable accuracy number.**

[](https://pypi.org/project/mcp-bigquery-evals/)
[](#eval-harness)
[](https://github.com/Umarfarook1/mcp-bigquery-evals/actions/workflows/ci.yml)
[](https://pypi.org/project/mcp-bigquery-evals/)
[](LICENSE)

`uvx mcp-bigquery-evals`  ·  works with any MCP-compatible client  ·  v0.1.0

---

## Why use this over the other BigQuery MCPs

| | Most BQ MCPs | `mcp-bigquery-evals` |
|---|---|---|
| Cost guardrails | none | **mandatory** dry-run before every query, refuses if over cap |
| Quality signal | "trust me" | **live accuracy badge**, recomputed every release |
| Write operations | usually enabled | **disabled by design** (read-only) |
| Errors when things break | raw API exceptions | **7 stable error codes** an agent can switch on |
| Local dev without GCP | impossible | **in-memory sqlite-backed fake** ships in the box |

## What ships in the box

- **7 read-only MCP tools** for warehouse discovery and querying
- **Mandatory dry-run cost cap** on every `run_query` (default 100 MB scanned, about $0.0005 per query)
- **Result-set-equivalence eval harness** (Spider/BIRD methodology) with a live accuracy badge in this README
- **Structured BigQuery errors** with 7 stable codes (`invalid_sql`, `table_not_found`, `permission_denied`, `unauthenticated`, `rate_limited`, `query_timeout`, `unknown`)
- **Two BigQueryClient implementations**: `RealBigQueryClient` (production, wraps `google-cloud-bigquery`) and `FakeBigQueryClient` (in-memory, sqlite-backed, for dev and CI without GCP credentials)

## Quickstart (5 minutes)

### 1. Install

```bash
uvx mcp-bigquery-evals --help
```

First run takes about 30s while `uv` fetches dependencies; subsequent runs are instant from the local cache. Plain `pip install mcp-bigquery-evals` also works.

### 2. Authenticate to GCP

```bash
gcloud auth application-default login
```

### 3. Wire into your MCP client

Open your MCP client's server config (developer settings) and add:

```json
{
  "mcpServers": {
    "bigquery": {
      "command": "uvx",
      "args": ["mcp-bigquery-evals", "serve"],
      "env": {
        "BIGQUERY_PROJECT": "YOUR_GCP_PROJECT_ID_HERE"
      }
    }
  }
}
```

Restart your client. The MCP indicator should show "bigquery" with 7 tools.

### 4. Try it

> Using the bigquery tool, find the top 5 most-viewed Stack Overflow questions tagged 'python'.

The agent chains `list_datasets`, `list_tables`, `describe_table`, `run_query` to answer. Every `run_query` is dry-run-cost-capped before execution.

Detailed setup, troubleshooting, and the alternative `pip` install path live in [`docs/mcp_client_setup.md`](docs/mcp_client_setup.md).

## The 7 tools

| Tool | Purpose |
|---|---|
| `list_datasets()` | List all datasets in your GCP project |
| `list_tables(dataset_id)` | List tables in a dataset |
| `describe_table(table_id)` | Schema, row count, size |
| `sample_table(table_id, n=5)` | Up to n sample rows |
| `search_schema(term)` | Fuzzy-match a term against all column names |
| `estimate_cost(sql)` | Free dry-run; returns bytes_scanned and estimated USD |
| `run_query(sql, max_bytes_scanned=100MB)` | Dry-run, refuse if over cap, then execute |

All tools are read-only. There are no write operations in v1 by design. See [`docs/architecture.md`](docs/architecture.md) for the design rationale.

## Cost guardrails

Every `run_query` call dry-runs first (free) before execution. If the dry-run estimate exceeds `max_bytes_scanned`, the call returns a structured error rather than burning bytes:

```json
{
  "error": "cost_cap_exceeded",
  "would_scan": "1.4 GB",
  "cap": "100.0 MB",
  "estimated_usd": 0.007,
  "hint": "narrow your WHERE clause or pass max_bytes_scanned=1500000000 to override"
}
```

The agent reads the structured error and self-corrects (narrows the WHERE clause, raises the cap explicitly, picks a different table).

## Eval harness

Every release runs a result-set-equivalence eval suite against `bigquery-public-data` and updates the accuracy badge above. The methodology matches Spider and BIRD academic benchmarks: execute both gold and predicted SQL, compare result sets as multisets of rows (order-independent, with float tolerance, Decimal handling, NULL equality, NaN equality, ARRAY/STRUCT recursion, bool/int distinction).

Run locally:

```bash
mcp-bigquery-evals evals run --model 
```

Full methodology, golden-pairs YAML format, and how to add your own pairs: [`docs/how_evals_work.md`](docs/how_evals_work.md).

## Development

```bash
git clone https://github.com/Umarfarook1/mcp-bigquery-evals
cd mcp-bigquery-evals
python -m venv .venv && source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

pytest                    # unit tests (no GCP needed; ~160 tests)
pytest -m bq              # real-BQ integration tests (needs GCP creds)
pytest -m live            # end-to-end with real model + real BQ
```

## Contributing

Issues and PRs welcome. Highest-leverage contributions:

1. **More verified golden NL-to-SQL pairs** against `bigquery-public-data`
2. **Prompt improvements** with before/after eval numbers showing the accuracy badge moved
3. **Bug reports** with minimum reproductions

## License

MIT, see [`LICENSE`](LICENSE).

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Umarfarook1](https://github.com/Umarfarook1)
- **Source:** [Umarfarook1/mcp-bigquery-evals](https://github.com/Umarfarook1/mcp-bigquery-evals)
- **License:** MIT
- **Homepage:** https://pypi.org/project/mcp-bigquery-evals/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-umarfarook1-mcp-bigquery-evals
- Seller: https://agentstack.voostack.com/s/umarfarook1
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.