Lit Dedup

Lit-Dedup (Stage 5 — Merge & Deduplicate)

Merge multiple pipeline stage outputs into one deduplicated master list with provenance tracking.

Input: one or more JSON files from Stages 1–4 Output: merged_results.json + merged_results.ris + dedup_log.json

Quick Start

# Merge all JSON files in a directory
python scripts/lit_dedup.py --input-dir ./results/ -o merged_results.json

# Merge specific files
python scripts/lit_dedup.py --inputs stage1.json stage2.json stage4.json -o merged.json

# DOI-only dedup (skip LLM pass)
python scripts/lit_dedup.py --inputs *.json --no-llm -o merged.json

# Non-interactive (skip confirmation)
python scripts/lit_dedup.py --inputs *.json -o merged.json --yes

Prerequisites

pip install aiohttp

API Keys

• DEEPSEEK_API_KEY — primary (DeepSeek chat API for fuzzy matching)
• ANTHROPIC_API_KEY — fallback (Claude Sonnet via Anthropic API)

CLI Flags

| Flag | Default | Description | |------|---------|-------------| | --inputs | — | Input JSON files (nargs='+') | | --input-dir | — | Directory to glob *.json from | | -o, --output | merged_results.json | Output JSON path | | --model | deepseek-chat | LLM model override | | --concurrency | 100 | Max simultaneous LLM requests | | --no-llm | off | Skip Pass 2 (DOI-only dedup) | | --yes | off | Skip HITL confirmation checkpoint | | --emit-pairs PATH | — | Agent-driven: write candidate pairs to PATH and stop before the LLM pass (no API) | | --ingest-verdicts EMIT VERDICTS | — | Agent-driven: merge the subagent pair verdicts and write all outputs (no API) |

Two-Pass Dedup

1. Pass 1 — DOI match: normalize DOIs, group by exact match, merge

using priority (published > forthcoming > working paper)

1. Pass 2 — LLM fuzzy match: pre-filter candidate pairs by author

surname overlap or title Jaccard > 0.4, then judge each pair for a same-paper verdict

Model routing (agent-driven default)

In the orchestrator's agent-driven flow the Pass-2 same-paper judgment is made by a subagent, not the API — a Sonnet subagent by default, promoted to Opus when the GUI's Use Opus for all tasks (all_opus) is set. Run --emit-pairs to dump the candidate pairs, judge each {i, j, a, b} pair with the subagent (yes/no duplicate), then --ingest-verdicts to run the union-find merge and write every output. The DeepSeek/Sonnet API path is the autonomous fallback for standalone runs. See the orchestrator SKILL.md ("How it runs" → "Model routing").

Output Schema

Same as pipeline stages, with added fields:

{
  "title": "Paper Title",
  "authors": "Author1, Author2",
  "year": "2024",
  "doi": "10.1234/...",
  "abstract": "...",
  "journal": "Journal Name",
  "url": "https://...",
  "sources": ["undermind", "scholarlabs", "ssrn"],
  "alt_url": "https://ssrn.com/..."
}

Output Files

• merged_results.json — deduplicated master list
• merged_results.ris — RIS for Zotero import
• dedup_log.json — log of all merge decisions and stats

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: kennethkhoocy
• Source: kennethkhoocy/legal-scholarship-skills
• License: MIT

Install and usage instructions live in the source repository linked above.