# Logzip

> Compress logs for LLM analysis — Rust-powered, Python API. 40-60% token savings.

- **Type:** MCP server
- **Install:** `agentstack add mcp-nailshakurov-logzip`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [NailShakurov](https://agentstack.voostack.com/s/nailshakurov)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [NailShakurov](https://github.com/NailShakurov)
- **Source:** https://github.com/NailShakurov/logzip
- **Website:** https://pypi.org/project/logzip/

## Install

```sh
agentstack add mcp-nailshakurov-logzip
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# logzip (Rust)

[](https://pypi.org/project/logzip/)
[](https://pypi.org/project/logzip/)
[](https://pypi.org/project/logzip/)
[](https://opensource.org/licenses/MIT)
[](https://www.rust-lang.org/)

Compress logs **before** sending to LLM. Powered by Rust & PyO3.

```text
raw log → [logzip compress] → compressed text → LLM (Claude Code / Cursor / API)
```

### Before / After

**Raw Log (Uvicorn):**
```text
INFO: 127.0.0.1:45678 - "GET /api/v1/status HTTP/1.1" 200 OK
INFO: 127.0.0.1:45679 - "GET /api/v1/status HTTP/1.1" 200 OK
... (100 similar lines) ...
```

**logzip output:**
```text
--- PREFIX ---
INFO: 127.0.0.1:
--- LEGEND ---
#0# = - "GET /api/v1/status HTTP/1.1" 200 OK
--- BODY ---
45678 #0#
45679 #0#
...
```

Typical savings: **52–58%** on structured logs (systemd, uvicorn, docker).  
Anomalies and unique lines stay uncompressed — visible at a glance in the BODY.

Compression is **lossy-semantic by default** (sub-second timestamps trimmed, whitespace collapsed — meaning preserved). Use `--lossless` for a byte-exact roundtrip.

### Why use logzip? (RAG & LLM)

When working with logs in LLMs (Claude, GPT, RAG systems), you face two problems:
1. **Context Limit**: Logs are huge. A 10MB log is ~2.5M tokens.
2. **Noise**: 90% of the log consists of repeating `INFO` and identical requests that drown out the real error.

`logzip` is well-suited for **RAG pipelines**: it compresses the context before sending it to the model, saving money on tokens and increasing answer accuracy by highlighting anomalies.

---

## Performance (7.96 MB Log, ~2M tokens)

Benchmarked on a real 7.96 MB production log.

### logzip modes

| Mode | CLI | Time (ms) | Size (KB) | Saved (%) | Output type |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **fast** | `--quality fast` | ~200 | ~4,900 | ~40% | text/LLM |
| **balanced** | `--quality balanced` | 404 | 3,928 | 52% | text/LLM |
| **balanced + 2 passes** ★ | `--quality balanced --bpe-passes 2` | 418 | 3,404 | **58%** | text/LLM |
| **max** | `--quality max` | ~1,600 | ≤ 3,404 | **≥ 58%** | text/LLM |

★ **Recommended default.** A second compression pass finds repeated token sequences in already-compressed text — 14 ms overhead, 7% more savings vs `balanced`.

`--quality max` runs an auto-search over several `(legend, passes)` configurations and returns the smallest output. It includes `(128, 2)` — the recommended config — in its grid, so it can never lose to `balanced --bpe-passes 2`, but it costs ~4× the time. Use it when you want the best ratio and runtime doesn't matter; otherwise stick with the recommended default. An explicit `--bpe-passes N` disables the search and pins a single config.

### vs. binary compressors (for context)

| Tool | Time (ms) | Size (KB) | Saved (%) | LLM-readable? |
| :--- | :--- | :--- | :--- | :--- |
| lz4 | 6 | 1,280 | 84% | No |
| zstd (lvl 3) | 14 | 819 | 90% | No |
| zlib (lvl 6) | 69 | 840 | 90% | No |
| **logzip (recommended)** | 418 | 3,404 | 58% | **Yes** |

Binary compressors produce opaque binary blobs — LLMs cannot read them. logzip trades ~30% size for fully human- and LLM-readable output.

Token estimation: 1 token ≈ 4 characters (rough estimate for English-like logs).

### Economic Impact

```text
┌──────────────────────────────────────────────────────────┐
│  logzip Savings (7.96 MB Production Log)                 │
├──────────────────────────────────────────────────────────┤
│  Raw Size:        8,151 KB  (~1,990,000 tokens)          │
│  After balanced:  3,928 KB  (~959,000 tokens,  -52%)     │
│  After 2 passes:  3,404 KB  (~831,000 tokens,  -58%)     │
├──────────────────────────────────────────────────────────┤
│  Cost Before:     $5.97                                  │
│  Cost After:      $2.49      (Claude 3.5 Sonnet Input)   │
│  LLM Efficiency:  2.4x larger context for the same price │
└──────────────────────────────────────────────────────────┘
```

---

## Install

**Python API + `logzip-py` CLI:**
```bash
pip install logzip
```

**Rust CLI + MCP Server:**
```bash
cargo install logzip
```

## CLI

Two CLIs are available. Both provide `compress` and `decompress` subcommands with identical flags.

**Rust binary** (`cargo install logzip` → `logzip`):
```bash
# stdin → stdout
logzip compress  compressed.txt

# save + show stats
logzip compress --stats -i app.log -o app.logzip

# lossless timestamps: keep full sub-second precision (default trims to milliseconds)
logzip compress --exact-timestamps -i app.log -o app.logzip

# byte-exact roundtrip: exact timestamps + preserved whitespace/indentation
logzip compress --lossless -i app.log -o app.logzip

# decompress
logzip decompress -i app.logzip
```

**Python CLI** (`pip install logzip` → `logzip-py`):
```bash
# same flags as above, plus:

# explicit profile (otherwise auto-detected)
logzip-py compress --profile journalctl  "Use logzip to analyze `/var/log/syslog`"

**Option B — `analyze_logs` prompt (Claude Code):**
```
/mcp → logzip → analyze_logs → path: /var/log/syslog
```
This compresses the log server-side and drops an SRE-ready context into the conversation.

**Option C — install the `log-analysis` skill (Claude Code, recommended):**

The skill makes Claude automatically reach for logzip whenever you mention a log file — no explicit instruction needed.

```bash
# 1. Register this repo as a single-plugin marketplace
#    (reads .claude-plugin/marketplace.json from the repo root)
claude plugin marketplace add NailShakurov/logzip

# 2. Install the logzip plugin from that marketplace
claude plugin install logzip@logzip
```

The install target is `@` — both are `logzip` here (the marketplace name comes from the `name` field in `marketplace.json`, not the repo path). To pull later updates, run `claude plugin marketplace update logzip`.

After that, asking "what's in `/var/log/syslog`?" is enough — Claude calls `get_stats` and `compress_tail` on its own.

### Security

The MCP server only reads files inside directories specified via `--allow-dir`.  
If no `--allow-dir` is given, defaults to the current working directory.  
All paths are canonicalized before comparison to prevent path traversal attacks.

---

## Through the eyes of an LLM

Unlike `gzip/zstd` which produce binary noise, `logzip` produces **structured text**. The model reads the legend once and works with the compressed body directly — it doesn't need to expand every token to understand the log.

**Input for LLM:**
> This is a compressed log. Rules: `#0#` is replaced by `GET /api/v1/status`.
>
> --- BODY ---
> 12:00:01 #0# 200 OK
> 12:00:02 #0# 500 ERR  B

    subgraph pipe["Compression Pipeline"]
        B["① Profile Detection\njournalctl · docker · uvicorn · nodejs · plain"]
        C["② Normalizer\nANSI · timestamps · hex zeros · common prefix"]
        D["③ Frequency Analysis\nparallel n-gram counting — rayon"]
        E["④ Preserve Filter\nUUID · IPv4 · hex≥16 · custom regex\nkeeps diagnostic IDs in body"]
        F["⑤ Greedy Legend Builder\nO(N) positional index — up to 512 entries"]
        G["⑥ AhoCorasick Substitution\nsingle-pass k-way merge"]
        H{bpe_passes > 0?}
        I["⑦ Recursive BPE\n2nd-pass on compressed body"]
        J["⑧ Template Extraction\nstructural repeats → &tag = value"]

        B --> C --> D --> E --> F --> G --> H
        H -->|yes| I --> J
        H -->|no| J
    end

    J --> K([CompressResult])
    K --> L[body]
    K --> M[legend]
    K --> N[templates]
    K --> O[stats]
```

1. **Normalizer**: Collapses ANSI, timestamps, IPs, and common prefixes.
2. **Frequency Analysis**: Parallel n-gram counting using `rayon`.
3. **Preserve Filter**: Skips UUID, IPv4, long hex, and custom patterns — keeps them visible in the body for LLM analysis.
4. **Greedy Legend**: Optimized selection using a positional index (O(N)).
5. **Direct Replacement**: Fast substitution without re-scanning.
6. **Second Pass**: Compresses repeated token sequences in the already-compressed body.
7. **Templates**: Structural template extraction.

### Safety First
- **Pure Rust**: Core logic is 100% Rust.
- **Zero `unsafe`**: The codebase contains **no unsafe blocks**, ensuring memory safety within the Python runtime.
- **Stress-tested**: Handled multi-GB logs without memory leaks or crashes.

## Reproducibility

Want to verify our benchmarks? Run the included script:
```bash
python benchmark.py
```

## Roadmap

Priority:
- [ ] Streaming mode for multi-GB logs

Planned:
- [x] MCP server for Claude Code
- [ ] Suffix automaton for arbitrary repetition search

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [NailShakurov](https://github.com/NailShakurov)
- **Source:** [NailShakurov/logzip](https://github.com/NailShakurov/logzip)
- **License:** MIT
- **Homepage:** https://pypi.org/project/logzip/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-nailshakurov-logzip
- Seller: https://agentstack.voostack.com/s/nailshakurov
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.