# Qulib

> Honest QA gap analysis for deployed web apps. Auth-aware scanning, real accessibility checks, and an MCP server for AI agents.

- **Type:** MCP server
- **Install:** `agentstack add mcp-tapeshn-qulib`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [TapeshN](https://agentstack.voostack.com/s/tapeshn)
- **Installs:** 0
- **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [TapeshN](https://github.com/TapeshN)
- **Source:** https://github.com/TapeshN/qulib

## Install

```sh
agentstack add mcp-tapeshn-qulib
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Qulib

**Release confidence for deployed web apps — the one question: should we ship?**

Qulib fuses live-app quality evidence, automation maturity, and API coverage into a single scored verdict: **ship / caution / hold / block**. It prefers **honest uncertainty** over fake confidence: if auth blocks the crawl, coverage is thin, or data is incomplete, the report says so.

**Design line:** AI should explore unknown gaps; **deterministic checks** (crawl, axe, links, console) should scale. Cost Intelligence tracks LLM usage so repeated reasoning can graduate into checks you own in CI.

On npm: **`@qulib/core`** (engine + CLI `qulib`) and **`@qulib/mcp`** (MCP server for AI agents).

[](https://www.npmjs.com/package/@qulib/core)
[](https://www.npmjs.com/package/@qulib/mcp)
[](LICENSE)
[](https://github.com/TapeshN/qulib/actions/workflows/ci.yml)

**Status:** [`@qulib/core`](https://www.npmjs.com/package/@qulib/core) and [`@qulib/mcp`](https://www.npmjs.com/package/@qulib/mcp) **v0.13.0** are published on npm — the release-confidence verdict **gates CI** (`--fail-on` / `--min-score`), the judge tools are now on the CLI (`qulib score-decisions` / `qulib score-bug-report`), and **`qulib validate`** grades a deployed app against a supplied spec (PRD / requirements) — *does it do what it was supposed to do?* See [`roadmap.json`](./roadmap.json) for shipped capabilities and the trust / release-confidence path to v1.0.0.

---

## What Qulib does

- Crawls deployed apps (anonymous or authenticated via Playwright)
- Runs **axe-core** accessibility checks (WCAG 2 A/AA)
- Detects broken links, console errors, navigation failures
- Computes **release confidence** (0–100) with a **coverage floor** when too few pages were scanned
- Emits **JSON** and **Markdown** reports (or **ephemeral** JSON on stdout)
- **Auth-aware:** optional `detect-auth`, `explore-auth`, form-login, and storage-state flows
- **Cost Intelligence** (optional block on gap analysis): token usage, budget warnings vs per-call output ceiling, prompt fingerprints, maturity hints, conversion recommendations

---

## Packages

| Package | Purpose |
|---------|---------|
| [`@qulib/core`](./packages/core) | Analyzer engine and CLI (`qulib`) |
| [`@qulib/mcp`](./packages/mcp) | MCP server exposing Qulib to AI clients |

---

## Open-core — what's MIT, what's hosted

Qulib is **genuinely open**, not a teaser. `@qulib/core` and `@qulib/mcp` are MIT-licensed, run **locally**, and need **no Qulib account** — bring your own `ANTHROPIC_API_KEY` for the optional LLM paths and every tool works. The verdict engine, the schemas, the deterministic collectors, the judge harness, and the MCP server are all here, complete, with **nothing gated behind a paywall and no crippled code paths**.

What is **not** in this repo, by design, is the part you can't meaningfully open-source: the **calibrated benchmark packs** (continuously-tuned weights and rubrics), the **cross-project signal** that sharpens the judges over time, and the **hosted service** (managed runs, history, team, CI-gate-as-a-service). The open tool ships an honest **baseline** rubric — useful standalone and clearly labelled as baseline; the hosted tier is the calibrated, networked upgrade. **We gate on data and service, never on crippled code.**

---

## Quick start (CLI)

**Release confidence — the flagship command:**

```bash
npx @qulib/core confidence --url https://example.com
# or equivalently: npx @qulib/core release-confidence --url https://example.com
```

Returns a verdict (`ship` / `caution` / `hold` / `block`) with a 0–100 score, top risks, and recommended next checks.

**Sample output:**

```json
{
  "verdict": "caution",
  "confidenceScore": 54,
  "level": 3,
  "label": "L3 — moderate confidence, known risks",
  "topRisks": [
    "Low crawl coverage (2 pages scanned)",
    "No CI integration detected"
  ],
  "recommendedNextChecks": [
    "Add a CI pipeline that runs qulib on each deploy",
    "Increase crawl depth or provide auth credentials"
  ],
  "honestyNotes": [
    "API coverage: not_applicable (no API endpoints found — excluded from score)"
  ]
}
```

Add `--repo` to also score test-automation maturity and API coverage:

```bash
npx @qulib/core confidence --url https://example.com --repo .
```

**Gate a release in CI** — turn the verdict into a pass/fail exit code:

```bash
# Block the deploy unless the verdict is better than `hold`, or score is ≥ 70.
npx @qulib/core confidence --url "$DEPLOY_URL" --repo . --fail-on hold --min-score 70
# exit 0 → ship;  exit 1 → gate failed (prints "GATE: FAIL — …").
```

- `--fail-on ` exits non-zero when the verdict is **at or worse than** the threshold (`caution` | `hold` | `block`).
- `--min-score ` exits non-zero when the 0–100 confidence score is below `n` (a `null` score — nothing evaluable — always fails).
- Combine with `--json` to capture the full report on stdout while the gate line goes to stderr. This is the "should we ship?" question wired straight into your pipeline.

**Analyze (full gap report):**

```bash
npx @qulib/core analyze --url https://example.com
```

**Scaffold a test suite:**

```bash
npx @qulib/core scaffold --url https://example.com --framework cypress-e2e
```

**Score automation maturity (repo only, no URL needed):**

```bash
npx @qulib/core score-automation --repo /path/to/repo
# or equivalently: npx @qulib/core automation-score --repo /path/to/repo
```

**Score pivotal-decision forks (LLM-judge tool — now in the CLI):**

```bash
npx @qulib/core score-decisions --forks agent-decisions.jsonl
# with a CI gate (exit non-zero when mean decision quality ` gate (0..1) exits non-zero when `aggregate.meanDecisionQuality `. In `--json` mode the gate line goes to stderr so stdout stays pure JSON.

**Score a learner bug report (LLM-judge tool — now in the CLI):**

```bash
npx @qulib/core score-bug-report --input bug-report.json
# emit full JSON result:
npx @qulib/core score-bug-report --input bug-report.json --json
```

The `--input` file is a JSON file with shape:

```json
{
  "report": {
    "title": "...",
    "description": "...",
    "steps": "...",
    "severity": "high"
  },
  "target": {
    "description": "...",
    "type": "validation",
    "severity": "high",
    "expectedBehavior": "..."
  }
}
```

`severity` must be one of `critical | high | medium | low`. Returns `matched`, `matchConfidence`, a 4-part rubric (`coverage / severity / repro / evidence`, each 0–25), and `feedback`. Falls back to deterministic scoring when `ANTHROPIC_API_KEY` is not set. Bad input prints a friendly one-line error — no raw stack.

From a clone (repo root):

```bash
npm run analyze -w @qulib/core -- --url https://example.com
```

**Smoke (no disk writes):**

```bash
npm run smoke
```

**Validate spec conformance — does the app do what the spec says?**

```bash
# Grade whether a deployed app matches a requirements file (offline, from an existing report):
npx @qulib/core validate --spec requirements.md --report output/report.json

# With live analysis from a URL (and the LLM judge enabled):
npx @qulib/core validate --spec requirements.md --url https://example.com --enable-llm-judge

# Gate CI — exit 1 when verdict is 'violates' or 'partial':
npx @qulib/core validate --spec requirements.md --report output/report.json --fail-on-violation
```

- `--spec `: a text or markdown file; each non-heading, non-empty line becomes a requirement (strips leading `- `, `* `, `N. `).
- `--report ` or `--url `: source for the observed app behavior summary.
- `--enable-llm-judge`: grade with the pinned LLM judge (requires `ANTHROPIC_API_KEY`). Without this, all requirements return `unknown` (honest — no fabricated verdicts).
- `--fail-on-violation`: exit 1 on `violates` or `partial` verdicts. `insufficient-evidence` does **not** trigger the gate.
- `--json`: emit the full `SpecConformanceResult` on stdout (stderr for the gate line).

**Cost doctor** (after a normal analyze that wrote `output/report.json`):

```bash
npx @qulib/core cost doctor
```

---

## Quick start (MCP)

```json
{
  "mcpServers": {
    "qulib": {
      "command": "npx",
      "args": ["-y", "@qulib/mcp"]
    }
  }
}
```

Ask your agent:

> Use Qulib to analyze https://example.com and tell me if it's ready to ship.

The agent will call **`qulib_score_confidence`** for the fused release verdict, or **`qulib_analyze_app`** for a detailed gap report. Default **`qulib_analyze_app`** responses are **summary-first** (top gaps, cost summary, next deterministic checks). Pass **`includeFullReport: true`** for the full `gapAnalysis` including all scenarios.

### MCP tool catalog

| Tool | What it tells you |
|------|-------------------|
| **`qulib_score_confidence`** | Fused **ship / caution / hold / block** verdict (0–100 score) from all evidence sources. Start here. |
| `qulib_analyze_app` | Live-app gaps — a11y, broken links, console errors, release confidence. |
| `qulib_score_automation` | Repo test-maturity from L1 (none) to L5 (advanced), across 6–7 scored dimensions. |
| `qulib_score_api` | API endpoint discovery + test coverage: are your routes exercised? |
| `qulib_scaffold_tests` | Ready-to-run Cypress spec + config, generated from a live crawl. |
| **`qulib_diff`** | Structured diff between two analyze outputs — added findings, resolved findings, severity changes, and a confidence delta. |
| **`qulib_detect_prompt_leakage`** | Scan a page surface for signals that AI system prompts or agent instructions are inadvertently exposed publicly. |
| **`qulib_score_bug_report`** | LLM-as-judge of a learner bug report against a planted-bug target — matched verdict, rubric (coverage/severity/repro/evidence), and feedback. Falls back to deterministic scoring without `ANTHROPIC_API_KEY`. |
| **`qulib_score_decisions`** | Pivotal-decision evaluation — scores senior-correctness at agent decision forks from a JSONL file. Deterministic by default; optional LLM refinement. |
| **`qulib_validate_spec`** | Spec-grounded conformance — grades whether the observed app behavior conforms to a supplied spec (PRD / requirements). Returns per-requirement verdicts (yes/no/unknown) + a fused verdict (conforms/partial/violates/insufficient-evidence). LLM judge optional; honest unknown without it. |
| `qulib_explore_auth` | All sign-in paths (OAuth, SSO, forms, magic link) and what to collect before scanning. |
| `qulib_detect_auth` | Single-pass auth pattern guess with a recommendation. Lighter than `explore_auth`. |

Legacy names (`analyze_app`, `explore_auth`, `detect_auth`) are kept as aliases through v1.0.

---

## CI integration (GitHub Actions)

Gate your deploys on Qulib's **honest agent-summary verdict** (`pass` / `warn` / `fail`) with a drop-in action. There are two surfaces — pick whichever fits your CI:

### Option A — composite action (drop into an existing job)

```yaml
# .github/workflows/qa.yml
jobs:
  qa:
    runs-on: ubuntu-latest
    steps:
      - name: Qulib analyze gate
        uses: TapeshN/qulib/.github/actions/qulib-analyze@v1
        with:
          url: https://your-app.example.com
          fail-on: fail        # fail (default) | warn | never
          qulib-version: 0.10.0 # pin a version for reproducible CI
```

### Option B — reusable workflow (whole job in one line)

```yaml
# .github/workflows/qa.yml
jobs:
  qa:
    uses: TapeshN/qulib/.github/workflows/qulib-analyze.yml@v1
    with:
      url: https://your-app.example.com
      fail-on: warn
    # secrets:
    #   auth-storage-state: ${{ secrets.QULIB_STORAGE_STATE }}
```

### How the gate decides

The action runs `@qulib/core analyze --url  --agent-summary`, which emits the stable agent-summary JSON. The CLI itself **always exits 0** — the verdict lives in the `gate` field — so the action reads `gate` and turns it into a CI exit code:

| `gate` | `fail-on: fail` (default) | `fail-on: warn` | `fail-on: never` |
|---|---|---|---|
| `pass` | ✅ pass | ✅ pass | ✅ pass |
| `warn` | ✅ pass | ❌ **fail** | ✅ pass |
| `fail` | ❌ **fail** | ❌ **fail** | ✅ pass |

A `fail` gate means a **critical gap**, a **blocked scan**, **null/too-low confidence**, or an **auth-required surface** that was never exercised — Qulib never silently passes an unevaluated deployment.

Every run **uploads the agent-summary JSON as an artifact** (`qulib-agent-summary`) and writes a **job summary** table with the gate, release confidence, top risks, and honesty notes.

### Inputs (composite action)

| Input | Default | Description |
|---|---|---|
| `url` | *(required)* | Base URL of the deployed app to analyze. |
| `fail-on` | `fail` | Gate policy: `fail`, `warn`, or `never`. |
| `qulib-version` | `latest` | npm version/dist-tag of `@qulib/core` run via `npx`. Pin for reproducibility. |
| `repo` | `""` | Optional path to the app repo for repo-aware analysis. |
| `config` | `""` | Path to a qulib config file relative to the working dir. |
| `auth-storage-state` | `""` | Path to a Playwright storage-state JSON for authenticated scans (write a secret to a file first — never inline it). |
| `extra-args` | `""` | Extra raw flags appended to `qulib analyze` (advanced). |
| `node-version` | `20` | Node.js version to set up. |
| `install-browser` | `true` | Install Playwright Chromium (Qulib crawls with Playwright). |
| `output-path` | `qulib-agent-summary.json` | Where to write the raw agent-summary JSON. |

### Outputs (composite action)

| Output | Description |
|---|---|
| `gate` | The verdict: `pass` \| `warn` \| `fail`. |
| `release-confidence` | Release confidence `0–100`, or `n/a` when null. |
| `coverage-status` | Coverage status enum (`ok` \| `thin` \| `blocked-by-auth` \| …). |
| `blocked` | `true` if the gate violated the `fail-on` policy (the job was failed). |
| `summary-path` | Path to the written agent-summary JSON artifact. |

> The composite action lives at [`.github/actions/qulib-analyze`](./.github/actions/qulib-analyze) and the reusable workflow at [`.github/workflows/qulib-analyze.yml`](./.github/workflows/qulib-analyze.yml). Reference them at a stable tag (e.g. `@v1`) for reproducible CI.

---

## Release confidence (short)

- Score starts from **100** and is reduced by **high / medium / low** gaps (see [`gaps.ts`](./packages/core/src/tools/scoring/gaps.ts)).
- If **fewer than `minPagesForConfidence`** pages were scanned, confidence is **capped at 40** and a **`low-coverage`** warning is set—thin coverage must not read as "ready".
- **`auth-required`** early exit sets confidence **0** with no gap inventory: the deployment was not actually exercised.

Details: [packages/core/README.md](./packages/core/README.md).

---

## Confidence Layer

> **Qulib turns delivery signals into release confidence.** The one question: *"Given everything we know right now, should we ship this?"*

The Confidence Layer fuses qulib's own evidence collectors into a single scored verdict:

| Verdict | Meaning |
|---|---|
| **ship** | Confidence ≥ 80, no blockers, all required sources evaluated |
| **caution** | Confidence 30–79, or an unknown signal on a required source |
| **hold** | Confidence ` to override.

### List saved baselines

```bash
qulib baseline list --url https://example.com
```

### Compare — detect drift

```bash
# Compare the two most-recent baselines for a URL:
qulib baseline compare --url https://example.com

# Compare two specific snapshots by id:
qulib baseline compare --from  --to 

# Machine-readable output for CI:
qulib baseline compare --url https://example.com --json
```

The comparison report shows:

- **new gaps** — problems present now that weren't before
- **resolved gaps** — problems that have since been fixed
- **severity changes** — same gap, escalated or de-escalated severity
- **confidence delta** — the numeric drift between the two runs, with a direction word (`improved` / `regressed` / `unchanged`)

Each changed item carries its `path`, `category`, and `severity` so you know exactly which dimension regressed or improved — no guessing from an

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [TapeshN](https://github.com/TapeshN)
- **Source:** [TapeshN/qulib](https://github.com/TapeshN/qulib)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-tapeshn-qulib
- Seller: https://agentstack.voostack.com/s/tapeshn
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.