# Lit Dedup

> >

- **Type:** Skill
- **Install:** `agentstack add skill-kennethkhoocy-legal-scholarship-skills-lit-dedup`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [kennethkhoocy](https://agentstack.voostack.com/s/kennethkhoocy)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [kennethkhoocy](https://github.com/kennethkhoocy)
- **Source:** https://github.com/kennethkhoocy/legal-scholarship-skills/tree/main/plugins/legal-scholarship/skills/lit-review-orchestrator/lit-dedup

## Install

```sh
agentstack add skill-kennethkhoocy-legal-scholarship-skills-lit-dedup
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Lit-Dedup (Stage 5 — Merge & Deduplicate)

Merge multiple pipeline stage outputs into one deduplicated master list
with provenance tracking.

**Input**: one or more JSON files from Stages 1–4
**Output**: `merged_results.json` + `merged_results.ris` + `dedup_log.json`

## Quick Start

```bash
# Merge all JSON files in a directory
python scripts/lit_dedup.py --input-dir ./results/ -o merged_results.json

# Merge specific files
python scripts/lit_dedup.py --inputs stage1.json stage2.json stage4.json -o merged.json

# DOI-only dedup (skip LLM pass)
python scripts/lit_dedup.py --inputs *.json --no-llm -o merged.json

# Non-interactive (skip confirmation)
python scripts/lit_dedup.py --inputs *.json -o merged.json --yes
```

## Prerequisites

```bash
pip install aiohttp
```

## API Keys

- `DEEPSEEK_API_KEY` — primary (DeepSeek chat API for fuzzy matching)
- `ANTHROPIC_API_KEY` — fallback (Claude Sonnet via Anthropic API)

## CLI Flags

| Flag | Default | Description |
|------|---------|-------------|
| `--inputs` | — | Input JSON files (nargs='+') |
| `--input-dir` | — | Directory to glob *.json from |
| `-o, --output` | `merged_results.json` | Output JSON path |
| `--model` | `deepseek-chat` | LLM model override |
| `--concurrency` | `100` | Max simultaneous LLM requests |
| `--no-llm` | off | Skip Pass 2 (DOI-only dedup) |
| `--yes` | off | Skip HITL confirmation checkpoint |
| `--emit-pairs PATH` | — | Agent-driven: write candidate pairs to PATH and stop before the LLM pass (no API) |
| `--ingest-verdicts EMIT VERDICTS` | — | Agent-driven: merge the subagent pair verdicts and write all outputs (no API) |

## Two-Pass Dedup

1. **Pass 1 — DOI match**: normalize DOIs, group by exact match, merge
   using priority (published > forthcoming > working paper)
2. **Pass 2 — LLM fuzzy match**: pre-filter candidate pairs by author
   surname overlap or title Jaccard > 0.4, then judge each pair for a
   same-paper verdict

## Model routing (agent-driven default)

In the orchestrator's agent-driven flow the Pass-2 same-paper judgment is made by a
**subagent**, not the API — a **Sonnet subagent by default**, promoted to **Opus**
when the GUI's *Use Opus for all tasks* (`all_opus`) is set. Run `--emit-pairs` to
dump the candidate pairs, judge each `{i, j, a, b}` pair with the subagent (yes/no
duplicate), then `--ingest-verdicts` to run the union-find merge and write every
output. The DeepSeek/Sonnet API path is the autonomous fallback for standalone runs.
See the orchestrator SKILL.md ("How it runs" → "Model routing").

## Output Schema

Same as pipeline stages, with added fields:

```json
{
  "title": "Paper Title",
  "authors": "Author1, Author2",
  "year": "2024",
  "doi": "10.1234/...",
  "abstract": "...",
  "journal": "Journal Name",
  "url": "https://...",
  "sources": ["undermind", "scholarlabs", "ssrn"],
  "alt_url": "https://ssrn.com/..."
}
```

## Output Files

- `merged_results.json` — deduplicated master list
- `merged_results.ris` — RIS for Zotero import
- `dedup_log.json` — log of all merge decisions and stats

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [kennethkhoocy](https://github.com/kennethkhoocy)
- **Source:** [kennethkhoocy/legal-scholarship-skills](https://github.com/kennethkhoocy/legal-scholarship-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-kennethkhoocy-legal-scholarship-skills-lit-dedup
- Seller: https://agentstack.voostack.com/s/kennethkhoocy
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.