# Verity

> MCP to check and minimise false LLM claims

- **Type:** MCP server
- **Install:** `agentstack add mcp-johnnyryan-verity`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [johnnyryan](https://agentstack.voostack.com/s/johnnyryan)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [johnnyryan](https://github.com/johnnyryan)
- **Source:** https://github.com/johnnyryan/Verity

## Install

```sh
agentstack add mcp-johnnyryan-verity
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Verity (Multi agent verification MCP to minimise hallucinations)

LLMs confidently claim things that are manifestly untrue. Enforce has developed Verity, a tool that helps minimise false claims and fake sources from self-hosted LLMs. It can run on cheap, old hardware. We think it is the first MCP that combines cross-family LLM critics, NLI, deterministic arithmetic recompute, consistency sampling, perplexity, and identifies disputes among these many critics. Today, we are releasing Verity for anyone to use, test, adapt, and improve.

Verity can also produce second opinions. If you have a spare old graphics card Verity can use it to produce second opinions at the same time that your primary LLMs responds. Both answers are then considered by your primary LLM. Once adapted for your hardware, you can easily use Verity in LM Studio. We are also sharing our system prompts, which help minimise LLM mistakes even without Verity.

[](https://docs.claude.com/en/docs/claude-code/mcp)
[](https://modelcontextprotocol.io/quickstart/user)
[](https://docs.cursor.com/context/model-context-protocol)
[](https://docs.windsurf.com/windsurf/cascade/mcp)
[](https://github.com/cline/cline)
[](https://github.com/openai/codex)
[](https://zed.dev/docs/assistant/mcp)
[](https://modelcontextprotocol.io)

  

---

## Quick start for local LLM (one-click install)

Verity itself needs only Node.js 18+. Everything else is a choice of where the models run. The one-click install below uses the reference design because it is the least setup, not because it is required: your primary in LM Studio, the critics in Ollama. But any OpenAI-compatible backend can serve in either role, and Verity can run without LM Studio at all. See "Adapt for your hardware" for vLLM, TGI, llama.cpp, cloud providers, and other MCP clients.

The reference one-click install uses:

1. Node.js 18 or newer, the only hard requirement.
2. Git, used by the installer to clone the repo. Without it, download the files and run `npm install` then `npm run build` yourself.
3. LM Studio 0.3.x or newer, as the MCP client and primary LLM host. Claude Desktop or another MCP client also works, or you can skip the client and use the confidence proxy.
4. Ollama for the two critics (the Vulkan build on AMD).

Then run the installer for your platform.

### Windows

Right-click `install-verity.ps1` and choose "Run with PowerShell".

The script clones the repo to `%USERPROFILE%\Verity`, installs the Node dependencies, builds the server, and pulls the two critic models. Allow about ten minutes the first time.

### Apple Silicon Mac

Double-click `install-verity-mac.command`. First time, right-click, Open, confirm the security prompt.

Same effect as the Windows installer, into `~/Verity`. The installer also sets `CONSULT_DUAL=0`; unified memory means one GPU, so the dual-GPU second-opinion path has nothing to do.

### After the installer finishes

Open LM Studio. Settings, Model Context Protocol. Paste:

```json
{
  "mcpServers": {
    "verity": {
      "url": "http://localhost:8090/mcp",
      "timeout": 240000,
      "retries": 1
    }
  }
}
```

Load a chat model in LM Studio. 

Then go to **Start** below.

---

## Current line-up

Four roles. Each runs as a separate model. Swap any of them.

| Role     | Current model                     | Where it runs               |
|----------|-----------------------------------|-----------------------------|
| LLM      | Qwen 3.5 9B (Q4_K_M)              | Strong GPU, via LM Studio   |
| Critic A | IBM Granite 4.1 8B (Q4_K_M)       | Weak GPU, via Ollama        |
| Critic B | Ministral 3B, Mistral AI (Q4_K_M) | Weak GPU, via Ollama        |
| NLI      | DeBERTa-v3-large (ONNX)           | CPU                         |

The names will change. Treat them as placeholders. From here on the document refers to your primary LLM as the "worker", Critic A, Critic B, and the NLI check, not to any specific model. Note the spread, though: the worker is from Alibaba, the two critics from IBM and Mistral. Three makers, three sets of blind spots, which is the point rather than the particular models.

---

## Start

### Windows

Run `start-verity.ps1`. It pins Ollama to the weak GPU, brings up the Verity server on port 8090, and reports status. Close the window when done; both services keep running.

To unload: `start-verity.ps1 -Action Stop`.

### Mac

From a Terminal prompt:

```
cd ~/Verity/project
node dist/index.js
```

Stop with Ctrl-C. A double-click launcher is on the deferred list.

### Using it

Ask the worker anything. Append `/verify` to the question. Verity returns a table of critic verdicts and a verdict.

The first chat after restarting LM Studio sometimes fails to call Verity. MCP plugins register a few seconds after LM Studio's UI is ready. Wait five seconds before the first message, or send a one-word "hi" first.

---

## Adapt for your hardware

Verity assumes a strong GPU for the worker and a weak GPU for the critics. The architecture survives other shapes; you just edit a few values in `project/src/config.ts`. Every user-tunable value is marked `[ADAPT]`.

### Two NVIDIA cards

Default. Worker on the strong card via LM Studio, both critics on the older card via Ollama. No special environment variables.

### NVIDIA strong, AMD weak (the reference build)

Ollama needs its Vulkan build. The launcher pins Ollama to the AMD card via `VK_DRIVER_FILES`. Without this, Ollama lands on the NVIDIA card, the worker spills to system RAM, and every call times out.

### One GPU only

Drop the split. Put the worker and both critics under the same backend. Use smaller critics so all three models fit. Set `CONSULT_DUAL=0` to disable the dual-card second-opinion path.

### Apple Silicon

One unified memory pool. The installer sets `CONSULT_DUAL=0` for you. The NLI check runs on CPU and is fine.

### Asymmetric: one strong card, one tiny card

Drop to a one-critic panel. Edit `ALL_CRITICS` in `project/src/critics/critic-configs.ts` to a single entry. Set `MAX_UNAVAILABLE_CRITICS = 0`. You lose cross-critic disputes; the rest still works.

### Cloud model as the worker

A standard `/verify` never calls the worker; it reads only the question and the answer text. So with any cloud model you can paste the answer, append `/verify`, and the local critics do the rest, with no setup.

Deep and deeper modes do call the worker, for re-sampling, claim extraction, and regeneration. Point those at the cloud with three settings:

```
WORKER_ENDPOINT=
WORKER_API_KEY=
WORKER_MODEL=
```

| Provider | `WORKER_ENDPOINT` | Connects directly | Perplexity (logprobs) |
|----------|-------------------|-------------------|-----------------------|
| OpenAI                  | `https://api.openai.com/v1`                                 | Yes | Yes |
| Microsoft Azure OpenAI  | `https://.openai.azure.com/openai/v1/`            | Yes | Yes |
| Google Gemini           | `https://generativelanguage.googleapis.com/v1beta/openai/`  | Yes | No on this surface |
| Anthropic (Claude)      | `https://api.anthropic.com/v1/`                             | Yes | No |
| Mistral (La Plateforme) | `https://api.mistral.ai/v1`                                 | Yes | No |

Critics, NLI, recompute, and consistency work with all of these; consistency just re-samples, which any of them can do. Perplexity needs token logprobs, so it works on OpenAI and Azure OpenAI and is skipped with a note on the rest: Anthropic ignores the request and returns nothing, Gemini exposes logprobs only on its native API rather than the OpenAI surface above, and Mistral has no logprob parameter. For Azure, `WORKER_MODEL` is your deployment name, not the catalogue id. If a provider's auth does not fit a plain `Authorization: Bearer` header, or it has no OpenAI-compatible surface, put a gateway such as LiteLLM in front and point `WORKER_ENDPOINT` at that.

Two things to weigh. Cost: deeper re-samples the worker five times, so one check is five cloud generations. Privacy: a standard check keeps the answer on the machine, but deep and deeper send the question and the answer to the provider. Keep the worker local if either matters.

### Where logprobs come from

Two signals use token logprobs: the perplexity check, and the proxy below. Whether they are available depends on the model server, not on Verity. For local backends:

| Backend | Logprobs | What Verity gets |
|---------|----------|------------------|
| LM Studio                | Responses API only (`/v1/responses`)                  | Scores a re-answer; proxy works |
| llama.cpp `llama-server` | `logprobs` on chat, `n_probs` on `/completion`        | Scores the answer, roughly |
| vLLM                     | `logprobs` plus `prompt_logprobs` / `echo`            | Scores the exact answer under review |
| Hugging Face TGI         | `decoder_input_details`                               | Scores the exact answer under review |
| Ollama                   | native `/api/generate` only; the OpenAI path drops them | No proxy; perplexity only via the native API |
| Jan                      | not surfaced                                          | Nothing |

The split that matters: vLLM and TGI can score the answer a user already has; LM Studio can only score text it generates itself, so its perplexity is a re-answer, exact on a deterministic question and a near-twin otherwise. Point Verity at vLLM or TGI if you want the answer under review scored directly.

### Confidence on every answer (the proxy)

In a sealed chat window (the LM Studio app, `ollama run`) Verity cannot see an answer's tokens, so it can only score confidence when you run a deep verify. Chat through an external client and the proxy removes that limit.

Point Open Web UI, Jan, LibreChat, or AnythingLLM at the proxy (`http://localhost:1235/v1`) instead of at the backend, and start it with `npm run proxy`. A plain answer then arrives with a confidence note whenever the model was unsure; tool calls, structured output, images, and everything else pass through untouched. The per-request rules and client setup are in `docs/confidence-proxy.md`.

The proxy needs a backend that serves logprobs from a responses-style endpoint: LM Studio does, Ollama does not (see the table above).

### Settings that almost always need tuning

| setting                              | What to tune                                       |
|-----------------------------------|----------------------------------------------------|
| `WORKER_MODEL_NAME`               | Match whatever you run in LM Studio                |
| `CRITIC_A_MODEL`, `CRITIC_B_MODEL`| Whatever fits the weak GPU                         |
| `CRITIC_TIMEOUT_MS`               | 45 s default; lower on faster hardware             |
| `PIPELINE_TIMEOUT_MS`             | Roughly three times the slowest critic             |
| `WARN_SEVERITY_THRESHOLD`         | Tighten if critics are quiet                       |
| `FAIL_SEVERITY_THRESHOLD`         | Loosen if critics are noisy                        |

Critic prompts in `project/src/prompts.ts` are the second-biggest lever after model choice.

### What stays the same

MCP wiring, aggregator rules, dispute detection, the recompute pass, the NLI check, consistency and perplexity. Pure logic; no GPU dependency.

---

## Commands

Type any of these after a worker reply.

### Depth

| Command         | What runs                                                   | Time   |
|-----------------|-------------------------------------------------------------|--------|
| `/verify`       | Two critics, NLI claim check, recompute                     | 3-5 s  |
| `/verifydeep`   | Standard, plus 2-sample consistency, advisory uncertainty   | ~20 s  |
| `/verifydeeper` | Standard, plus 5-sample consistency, advisory uncertainty   | ~40 s  |

### Context

| Command                | What it does                                          |
|------------------------|-------------------------------------------------------|
| `/verify`              | Minimal context. Question and answer only.           |
| `/verify with context` | Worker passes the relevant prior messages.           |
| `/verify full`         | Worker passes the whole visible conversation.        |

### Modifiers

| Command                | Effect                                                |
|------------------------|-------------------------------------------------------|
| `/verify no-nli`       | Skip the NLI claim check.                            |
| `/verify as code`      | Force task_type=code.                                |
| `/verify as prose`     | Force task_type=prose.                               |
| `/verify as reasoning` | Force task_type=reasoning.                           |

Modifiers stack. `/verifydeeper as code no-nli with context` is valid.

### Second opinion

| Command           | Effect                                                                  |
|-------------------|-------------------------------------------------------------------------|
| `/second`         | Two cross-family models answer the same question. A third pass compares them. |
| `/verify /second` | Both run. `/second` first, then `/verify`.                              |

---

## How a verdict is built

Five checks, fired in parallel.

1. **Critic A and Critic B.** Two smaller LLMs read the worker's answer and return structured JSON: verdict, severity, concerns, suggested fixes.
2. **NLI claim check.** Each factual claim is paired with the prior context. A 0.4 B encoder transformer (not an LLM) labels each pair as entailment, contradiction, or neutral. Runs on CPU.
3. **Recompute pass.** Pure code. Pulls arithmetic and unit conversions out of the answer, evaluates them, flags mismatches. 100% precision when it fires.
4. **Consistency** (deep modes only). Re-asks the worker N times at temperature 0.7. Compares each re-sample against the original.
5. **Perplexity** (deep modes only, advisory). Reads the worker's own token probabilities and flags low-confidence spans as model uncertainty. A nudge, not a vote: it never moves the verdict on its own, and it cannot catch a confident, fluent error. The consistency check guards against that.

Each check has a different failure profile. That is the point. Two LLMs from similar training data tend to be wrong about the same things; when they agree, they often agree wrong. The NLI classifier was trained on entailment labels, not helpfulness preferences. The recompute pass has no bias profile at all because it is not statistical. When two layers built on different machinery agree on a flaw, the signal is strong.

The aggregator combines the first four into one of: pass, warn, fail, error. Perplexity rides alongside as an advisory note and never changes the verdict.

A separate disputes table is computed after the verdict. It surfaces concerns one critic raised but not the other. The user sees disagreement even when the headline verdict is pass.

---

## Reference machine and current setup

The reference build is a 2021 PC. NVIDIA RTX 5070 Ti (16 GB, 2025) for the worker. AMD Radeon RX 5700 XT (8 GB, 2019) for the critics. CPU runs the NLI classifier.

VRAM use, current line-up:

| Role     | Size  | VRAM    | Device                          |
|----------|-------|---------|---------------------------------|
| Worker   | 9 B   | ~5.5 GB | Strong GPU, LM Studio           |
| Critic A | 8 B   | ~5.3 GB | Weak GPU, Ollama                |
| Critic B | 3 B   | ~2.5 GB | Weak GPU, Ollama                |
| NLI      | 0.4 B | ~1 GB   | CPU (ONNX Runtime)              |

The strong GPU uses about a third of its memory. The two critics now fill most of the weak card, roughly 8 GB of models on an 8 GB card, so KV-cache headroom is slim. Drop Critic A to a q3 quant if Ollama starts evicting.

Verity wants two things from the critics: different training data than the worker, and small enough to share the weak GPU. Family diversity matters more than size. Two sm

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [johnnyryan](https://github.com/johnnyryan)
- **Source:** [johnnyryan/Verity](https://github.com/johnnyryan/Verity)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-johnnyryan-verity
- Seller: https://agentstack.voostack.com/s/johnnyryan
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.