# Token Compressor

> Token compressor

- **Type:** MCP server
- **Install:** `agentstack add mcp-base76-research-lab-token-compressor`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [base76-research-lab](https://agentstack.voostack.com/s/base76-research-lab)
- **Installs:** 0
- **Category:** [Integrations](https://agentstack.voostack.com/c/integrations)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [base76-research-lab](https://github.com/base76-research-lab)
- **Source:** https://github.com/base76-research-lab/token-compressor

## Install

```sh
agentstack add mcp-base76-research-lab-token-compressor
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# token-compressor 
## Reduce LLM prompt tokens by 30–70% while preserving semantic meaning.

mcp-name: io.github.base76-research-lab/token-compressor

**Semantic prompt compression for LLM workflows. Reduce token usage by 40–60% without losing meaning.**

[](LICENSE)
[](https://ollama.com)
[](https://modelcontextprotocol.io)

Built by [Base76 Research Lab](https://base76.se) — research into epistemic AI architecture.

---

## Live demo

Intent Compiler MVP is now live and uses this project as part of the idea -> spec -> compressed output flow:

- Live: https://intent-compiler-mvp.pages.dev
- Product repo: https://github.com/base76-research-lab/token-compressor

---

## What it does

token-compressor is a two-stage pipeline that compresses prompts before they reach an LLM:

1. **LLM compression** — a local model (llama3.2:1b via Ollama) rewrites the prompt to its semantic minimum, preserving all conditionals and negations
2. **Embedding validation** — cosine similarity between original and compressed embeddings must exceed a threshold (default: 0.85) — if not, the original is sent unchanged

The result: shorter prompts, lower costs, same intent.

```
Input prompt (300 tokens)
        ↓
  LLM compresses
        ↓
  Embedding validates (cosine ≥ 0.85?)
        ↓
  Pass → compressed (120 tokens)   Fail → original (300 tokens)
```

**Key design principle:** conditionality is never sacrificed. If your prompt says "only do X if Y", that constraint survives compression.

---

## Requirements

- Python 3.10+
- [Ollama](https://ollama.com) running locally
- Two models pulled:

```bash
ollama pull llama3.2:1b
ollama pull nomic-embed-text
```

- Python dependencies:

```bash
pip install ollama numpy
```

---

## Quick start

```python
from compressor import LLMCompressEmbedValidate

pipeline = LLMCompressEmbedValidate()
result = pipeline.process("Your prompt text here...")

print(result.output_text)   # compressed (or original if validation failed)
print(result.report())      # MODE / COVERAGE / TOKENS saved
```

**Result object:**

| Field | Description |
|-------|-------------|
| `output_text` | Text to send to your LLM |
| `mode` | `compressed` / `raw_fallback` / `skipped` |
| `coverage` | Cosine similarity (0.0–1.0) |
| `tokens_in` | Estimated input tokens |
| `tokens_out` | Estimated output tokens |
| `tokens_saved` | Difference |

---

## CLI usage

```bash
echo "Your long prompt here..." | python3 cli.py
```

Output: compressed text on stdout, stats on stderr.

---

## Claude Code hook (recommended setup)

Add to your `~/.claude/settings.json` under `hooks → UserPromptSubmit`:

```json
{
  "type": "command",
  "command": "echo \"${CLAUDE_USER_PROMPT:-}\" | python3 /path/to/token-compressor/cli.py > /tmp/compressed_prompt.txt 2>/tmp/compress.log || true"
}
```

This runs on every prompt submission and writes the compressed version to a temp file, which can be injected back into context via a second hook or MCP server.

---

## MCP server

The MCP server exposes compression as a tool callable from Claude Code and any MCP-compatible client.

**Install:**

```bash
pip install token-compressor-mcp
```

**Tool:** `compress_prompt`
- Input: `text` (string)
- Output: compressed text + stats footer

**Claude Code MCP config** (`~/.claude/settings.json`):

```json
{
  "mcpServers": {
    "token-compressor": {
      "command": "uvx",
      "args": ["token-compressor-mcp"]
    }
  }
}
```

Or from source:

```json
{
  "mcpServers": {
    "token-compressor": {
      "command": "python3",
      "args": ["-m", "token_compressor_mcp"],
      "cwd": "/path/to/token-compressor"
    }
  }
}
```

---

## Configuration

```python
pipeline = LLMCompressEmbedValidate(
    threshold=0.85,          # cosine similarity floor (lower = more aggressive)
    min_tokens=80,           # skip pipeline below this (not worth compressing)
    compress_model="llama3.2:1b",
    embed_model="nomic-embed-text",
)
```

---

## How it works

**Stage 1 — LLM compression**

The compression prompt instructs the model to:
- Preserve all conditionals (`if`, `only if`, `unless`, `when`, `but only`)
- Preserve all negations
- Remove filler, hedging, redundancy
- Target 40–60% of original length

**Stage 2 — Embedding validation**

Computes cosine similarity between the original and compressed text using `nomic-embed-text`. If similarity falls below threshold, the original is returned unchanged. This prevents silent meaning loss.

---

## Results

Tested across Swedish and English prompts, technical and natural language:

| Input | Tokens in | Tokens out | Saved |
|-------|-----------|------------|-------|
| Research abstract (EN) | 89 | 38 | 57% |
| Session intent (SV) | 32 | 18 | 44% |
| Technical instruction | 47 | 22 | 53% |
| Short command ( Wikström, B. (2026). *When Alignment Reduces Uncertainty: Epistemic Variance
> Collapse and Its Implications for Metacognitive AI.*
> DOI: [10.5281/zenodo.18731535](https://doi.org/10.5281/zenodo.18731535)

Part of the [Base76 Research Lab](https://base76.se) toolchain for epistemic AI infrastructure.

---

## License

MIT — Base76 Research Lab, Sweden

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [base76-research-lab](https://github.com/base76-research-lab)
- **Source:** [base76-research-lab/token-compressor](https://github.com/base76-research-lab/token-compressor)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-base76-research-lab-token-compressor
- Seller: https://agentstack.voostack.com/s/base76-research-lab
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.