Logzip

logzip (Rust)

[](https://pypi.org/project/logzip/) [](https://pypi.org/project/logzip/) [](https://pypi.org/project/logzip/) [](https://opensource.org/licenses/MIT) [](https://www.rust-lang.org/)

Compress logs before sending to LLM. Powered by Rust & PyO3.

raw log → [logzip compress] → compressed text → LLM (Claude Code / Cursor / API)

Before / After

Raw Log (Uvicorn):

INFO: 127.0.0.1:45678 - "GET /api/v1/status HTTP/1.1" 200 OK
INFO: 127.0.0.1:45679 - "GET /api/v1/status HTTP/1.1" 200 OK
... (100 similar lines) ...

logzip output:

--- PREFIX ---
INFO: 127.0.0.1:
--- LEGEND ---
#0# = - "GET /api/v1/status HTTP/1.1" 200 OK
--- BODY ---
45678 #0#
45679 #0#
...

Typical savings: 52–58% on structured logs (systemd, uvicorn, docker). Anomalies and unique lines stay uncompressed — visible at a glance in the BODY.

Compression is lossy-semantic by default (sub-second timestamps trimmed, whitespace collapsed — meaning preserved). Use --lossless for a byte-exact roundtrip.

Why use logzip? (RAG & LLM)

When working with logs in LLMs (Claude, GPT, RAG systems), you face two problems:

1. Context Limit: Logs are huge. A 10MB log is ~2.5M tokens.
2. Noise: 90% of the log consists of repeating INFO and identical requests that drown out the real error.

logzip is well-suited for RAG pipelines: it compresses the context before sending it to the model, saving money on tokens and increasing answer accuracy by highlighting anomalies.

---

Performance (7.96 MB Log, ~2M tokens)

Benchmarked on a real 7.96 MB production log.

logzip modes

| Mode | CLI | Time (ms) | Size (KB) | Saved (%) | Output type | | :--- | :--- | :--- | :--- | :--- | :--- | | fast | --quality fast | ~200 | ~4,900 | ~40% | text/LLM | | balanced | --quality balanced | 404 | 3,928 | 52% | text/LLM | | balanced + 2 passes ★ | --quality balanced --bpe-passes 2 | 418 | 3,404 | 58% | text/LLM | | max | --quality max | ~1,600 | ≤ 3,404 | ≥ 58% | text/LLM |

★ Recommended default. A second compression pass finds repeated token sequences in already-compressed text — 14 ms overhead, 7% more savings vs balanced.

--quality max runs an auto-search over several (legend, passes) configurations and returns the smallest output. It includes (128, 2) — the recommended config — in its grid, so it can never lose to balanced --bpe-passes 2, but it costs ~4× the time. Use it when you want the best ratio and runtime doesn't matter; otherwise stick with the recommended default. An explicit --bpe-passes N disables the search and pins a single config.

vs. binary compressors (for context)

| Tool | Time (ms) | Size (KB) | Saved (%) | LLM-readable? | | :--- | :--- | :--- | :--- | :--- | | lz4 | 6 | 1,280 | 84% | No | | zstd (lvl 3) | 14 | 819 | 90% | No | | zlib (lvl 6) | 69 | 840 | 90% | No | | logzip (recommended) | 418 | 3,404 | 58% | Yes |

Binary compressors produce opaque binary blobs — LLMs cannot read them. logzip trades ~30% size for fully human- and LLM-readable output.

Token estimation: 1 token ≈ 4 characters (rough estimate for English-like logs).

Economic Impact

┌──────────────────────────────────────────────────────────┐
│  logzip Savings (7.96 MB Production Log)                 │
├──────────────────────────────────────────────────────────┤
│  Raw Size:        8,151 KB  (~1,990,000 tokens)          │
│  After balanced:  3,928 KB  (~959,000 tokens,  -52%)     │
│  After 2 passes:  3,404 KB  (~831,000 tokens,  -58%)     │
├──────────────────────────────────────────────────────────┤
│  Cost Before:     $5.97                                  │
│  Cost After:      $2.49      (Claude 3.5 Sonnet Input)   │
│  LLM Efficiency:  2.4x larger context for the same price │
└──────────────────────────────────────────────────────────┘

---

Install

Python API + logzip-py CLI:

pip install logzip

Rust CLI + MCP Server:

cargo install logzip

CLI

Two CLIs are available. Both provide compress and decompress subcommands with identical flags.

Rust binary (cargo install logzip → logzip):

# stdin → stdout
logzip compress  compressed.txt

# save + show stats
logzip compress --stats -i app.log -o app.logzip

# lossless timestamps: keep full sub-second precision (default trims to milliseconds)
logzip compress --exact-timestamps -i app.log -o app.logzip

# byte-exact roundtrip: exact timestamps + preserved whitespace/indentation
logzip compress --lossless -i app.log -o app.logzip

# decompress
logzip decompress -i app.logzip

Python CLI (pip install logzip → logzip-py):

# same flags as above, plus:

# explicit profile (otherwise auto-detected)
logzip-py compress --profile journalctl  "Use logzip to analyze `/var/log/syslog`"

**Option B — `analyze_logs` prompt (Claude Code):**

/mcp → logzip → analyze_logs → path: /var/log/syslog

This compresses the log server-side and drops an SRE-ready context into the conversation.

**Option C — install the `log-analysis` skill (Claude Code, recommended):**

The skill makes Claude automatically reach for logzip whenever you mention a log file — no explicit instruction needed.

```bash
# 1. Register this repo as a single-plugin marketplace
#    (reads .claude-plugin/marketplace.json from the repo root)
claude plugin marketplace add NailShakurov/logzip

# 2. Install the logzip plugin from that marketplace
claude plugin install logzip@logzip

The install target is @ — both are logzip here (the marketplace name comes from the name field in marketplace.json, not the repo path). To pull later updates, run claude plugin marketplace update logzip.

After that, asking "what's in /var/log/syslog?" is enough — Claude calls get_stats and compress_tail on its own.

Security

The MCP server only reads files inside directories specified via --allow-dir. If no --allow-dir is given, defaults to the current working directory. All paths are canonicalized before comparison to prevent path traversal attacks.

---

Through the eyes of an LLM

Unlike gzip/zstd which produce binary noise, logzip produces structured text. The model reads the legend once and works with the compressed body directly — it doesn't need to expand every token to understand the log.

Input for LLM: > This is a compressed log. Rules: #0# is replaced by GET /api/v1/status. > > --- BODY --- > 12:00:01 #0# 200 OK > 12:00:02 #0# 500 ERR B

subgraph pipe["Compression Pipeline"] B["① Profile Detection\njournalctl · docker · uvicorn · nodejs · plain"] C["② Normalizer\nANSI · timestamps · hex zeros · common prefix"] D["③ Frequency Analysis\nparallel n-gram counting — rayon"] E["④ Preserve Filter\nUUID · IPv4 · hex≥16 · custom regex\nkeeps diagnostic IDs in body"] F["⑤ Greedy Legend Builder\nO(N) positional index — up to 512 entries"] G["⑥ AhoCorasick Substitution\nsingle-pass k-way merge"] H{bpe_passes > 0?} I["⑦ Recursive BPE\n2nd-pass on compressed body"] J["⑧ Template Extraction\nstructural repeats → &tag = value"]

B --> C --> D --> E --> F --> G --> H H -->|yes| I --> J H -->|no| J end

J --> K([CompressResult]) K --> L[body] K --> M[legend] K --> N[templates] K --> O[stats]

1. **Normalizer**: Collapses ANSI, timestamps, IPs, and common prefixes.
2. **Frequency Analysis**: Parallel n-gram counting using `rayon`.
3. **Preserve Filter**: Skips UUID, IPv4, long hex, and custom patterns — keeps them visible in the body for LLM analysis.
4. **Greedy Legend**: Optimized selection using a positional index (O(N)).
5. **Direct Replacement**: Fast substitution without re-scanning.
6. **Second Pass**: Compresses repeated token sequences in the already-compressed body.
7. **Templates**: Structural template extraction.

### Safety First
- **Pure Rust**: Core logic is 100% Rust.
- **Zero `unsafe`**: The codebase contains **no unsafe blocks**, ensuring memory safety within the Python runtime.
- **Stress-tested**: Handled multi-GB logs without memory leaks or crashes.

## Reproducibility

Want to verify our benchmarks? Run the included script:
```bash
python benchmark.py

Roadmap

Priority:

• [ ] Streaming mode for multi-GB logs

Planned:

• [x] MCP server for Claude Code
• [ ] Suffix automaton for arbitrary repetition search

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: NailShakurov
• Source: NailShakurov/logzip
• License: MIT
• Homepage: https://pypi.org/project/logzip/

Install and usage instructions live in the source repository linked above.