Contract Question Agent

contract-question-agent

contract-question-agent is a small open-source contract verification-question generator.

It is built for the moment when a contract clause feels risky, but the reviewer cannot yet say exactly why.

Instead of answering whether a clause is legal, enforceable, fair, risky, or safe to sign, it turns that vague concern into structured verification questions a human reviewer can raise before relying on the clause.

In that sense, it treats contract review as an information asymmetry problem: the drafter may understand the clause, edge cases, and business implications better than the counter-party reviewing it.

It also refuses to force generation when the input is outside the contract verification-question task.

Generated verification questions are prompts for further investigation. They must be reviewed with a qualified legal professional before anyone relies on them.

Thesis

Agent frameworks are useful, but they should not become the architecture.

The important part is not which framework owns the agent.

The important part is whether each responsibility has a visible boundary.

This repository is an experiment in separating orchestration, model execution, schema contracts, controlled context, skill constraints, reflection, and human-facing observability so that agent failures become debuggable state transitions instead of vague outputs.

In practice:

• LangGraph owns workflow orchestration and routing.
• Pydantic AI currently implements the structured model-execution boundary.
• Schemas define what can cross each boundary.
• MCP provides system-controlled candidate context, not autonomous tool use.
• Skill files constrain what the model is trying to do.
• Reflection checks whether the output follows that thesis.
• AG-UI and Streamlit expose workflow state to a human.

The goal is not to make the agent more autonomous.

The goal is to make it easier to inspect, constrain, and repair.

Status

This repository is at a v1.0.0 release-candidate stage.

It includes:

• CUAD data preparation
• CLI generation
• LangGraph workflow orchestration
• two-step scope checking
• Pydantic AI / OpenRouter structured model calls
• MCP candidate review lenses
• skill-thesis reflection
• FastAPI and AG-UI event stream
• Streamlit run viewer
• synthetic in-scope and out-of-scope examples

Design principles

The design keeps the economic framing intentionally narrow:

• Information asymmetry: the reviewer has less information than the

drafter, and targeted questions are a low-cost way to narrow that gap.

• Screening and signaling: questions help a reviewer screen for

terms that warrant deeper attention and give the drafter a channel to signal intent.

• Principal-agent problems: the reviewer's interests and the

drafter's interests are not always aligned, and explicit questions make that alignment testable.

• Loss aversion: reviewers tend to weight downside outcomes heavily,

so surfacing unknowns up front reduces the cost of acting on them later.

Those ideas map directly to runtime boundaries:

| Principle | Runtime behavior | Where to see it | |---|---|---| | Do not answer legal questions | Outputs verification questions, not legal conclusions | VerificationQuestionOutput, examples/in_scope_non_compete.md | | Do not force generation | Out-of-scope input stops before generation and reflection | PRECHECK_INPUT, CLASSIFY_SCOPE, examples/out_of_scope_non_contract_text.md | | Keep workflow state visible | AG-UI emits node-level status events | Streamlit node timeline | | Keep responsibilities separate | LangGraph handles workflow; Pydantic AI is used as the current structured model-execution boundary | workflows/workflow.py, model_client/openrouter.py | | Keep external knowledge bounded | MCP provides candidate review lenses, not autonomous tool use | mcp/, selected_review_lenses |

Architecture and observability

LangGraph:
  workflow orchestration / state transitions / routing

PRECHECK_INPUT:
  deterministic cheap input validation before any LLM call

CLASSIFY_SCOPE:
  LLM-backed structured task-scope classification

Pydantic AI:
  node-internal structured model calls / classification / generation / reflection

MCP:
  deterministic candidate review lenses

FastAPI:
  thin HTTP boundary

AG-UI SSE:
  human-facing run event stream

Streamlit:
  minimal run/output viewer

Langfuse:
  optional tracing / Agent Graph / business-node spans / generation usage

CI:
  GitHub Actions with uv-managed Python 3.13.13

Layout

data/cuad/
  raw/            # downloaded CUAD payload (gitignored)
  processed/      # contracts.jsonl, labels.jsonl, clause_spans.jsonl
docs/
  data.md         # data provenance, license, limitations
examples/
  in_scope_non_compete.md
  out_of_scope_non_contract_text.md
src/contract_question_agent/
  cuad_downloader.py  # optional downloader
  cuad_loader.py      # parser + JSONL writer
  cli_generate_questions.py
  api/                # thin FastAPI HTTP adapter
  model_client/        # Pydantic AI structured model-calling path
  prompts/             # model instructions
  skills/              # contract verification-question skill thesis
  mcp/                 # candidate review-lens server
  workflows/
    workflow.py        # LangGraph orchestration / state transitions
    tracing.py         # optional Langfuse tracing helpers
    nodes/             # framework-independent business node transitions
viewer/
  streamlit_app.py
  sse_client.py
tests/
  test_cuad_downloader.py
  test_cuad_loader.py
  test_generate_questions_cli.py
  test_workflow.py
  test_pydantic_ai_client.py
  test_safety.py
  test_schemas.py

Quick start

This project uses uv and a pinned .python-version to make local development reproducible with Python 3.13.13. uv sync installs the matching interpreter (no prior install needed), creates .venv/, and resolves all dependencies from the committed uv.lock.

# Set up the environment (Python 3.13.13 + locked dependencies).
uv sync

# Run the test suite.
uv run pytest

# 1. Download CUAD_v1.json (Hugging Face is the default source).
uv run cuad-downloader --source huggingface
# Writes data/cuad/raw/CUAD_v1.json by default.

# 2. Process it into JSONL filtered to the evaluation clause types.
uv run cuad-loader \
  --input data/cuad/raw/CUAD_v1.json \
  --output-dir data/cuad/processed

# 3. Run the minimal E2E generator without network access.
uv run contract-question-generate \
  --input data/cuad/processed/clause_spans.jsonl \
  --clause-type "Non-Compete" \
  --limit 3 \
  --dry-run

Zenodo is also supported as an alternative source:

uv run cuad-downloader --source zenodo
# Writes data/cuad/raw/CUAD_v1.zip by default. The loader reads .zip
# directly, so you can pass it straight to --input without unzipping.

The downloader is optional — if you already have CUAD_v1.json (or the zip archive) on disk, place it under data/cuad/raw/ and skip step 1.

See [docs/data.md](docs/data.md) for licensing and attribution requirements.

Minimal E2E generation

Workflow

The generator runs a deliberately linear LangGraph workflow:

LOAD_CLAUSE_SPANS
-> FILTER_RECORDS
-> PRECHECK_INPUT
-> CLASSIFY_SCOPE
-> GENERATE_MINIMAL_QUESTIONS
-> REFLECT_AGAINST_SKILL_THESIS
-> SAFETY_CHECK
-> WRITE_OUTPUT

PRECHECK_INPUT is deterministic. CLASSIFY_SCOPE is LLM-backed and returns a structured task-scope classification through the model-client boundary. Out-of-scope inputs stop before generation and reflection.

CLI usage

Record loading and filtering are deterministic and use only CLI arguments:

uv run contract-question-generate \
  --input data/cuad/processed/clause_spans.jsonl \
  --clause-type "Non-Compete" \
  --contract-id SOME_CONTRACT_ID \
  --limit 3 \
  --offset 0 \
  --dry-run

For real model calls, set OPENROUTER_API_KEY and omit --dry-run. OPENROUTER_MODEL is optional; --model overrides the environment and default.

Recommended local setup:

cp .env.example .env
# Edit .env and set OPENROUTER_API_KEY.
# .env is gitignored and must not be committed.

Alternative one-shell setup:

export OPENROUTER_API_KEY="..."
export OPENROUTER_MODEL="google/gemini-3-flash-preview"

Then run:

uv run contract-question-generate \
  --input data/cuad/processed/clause_spans.jsonl \
  --clause-type "Non-Compete" \
  --limit 3

Model runtime

The default OpenRouter model is configured in src/contract_question_agent/model_client/openrouter.py and can be overridden with OPENROUTER_MODEL or --model. OPENROUTER_API_KEY is required for real model calls; OPENROUTER_MODEL is optional. Tests use fake clients and do not call the network.

Pydantic AI is used as the current structured model-execution boundary for node-internal model calls and Pydantic output validation. LangGraph handles workflow orchestration.

MAF was removed because the project does not need a universal Agent framework. The workflow boundary is implemented with LangGraph, while Pydantic AI is used only as a structured model-call component inside classification, generation, and reflection nodes.

The workflow calls model_client.classify_scope() once per record that passes deterministic precheck, then calls model_client.generate() once per classified in-scope clause span. If --limit 1 produces one output row but OpenRouter or provider logs show additional upstream requests, the extra request is usually classification, reflection, or provider structured-output behavior, not a duplicate LangGraph generation node.

Scope classification

Scope checking is split into two visible workflow responsibilities:

• PRECHECK_INPUT is deterministic and cheap. It rejects clearly invalid input

such as empty text, missing clause_type, very short text, or obviously malformed text before any LLM call.

• CLASSIFY_SCOPE is an LLM-backed structured classifier behind the existing

model_client boundary. It decides only whether the text is a reasonable input for generating contract verification questions for the given clause_type.

The scope classifier does not judge legality, enforceability, fairness, risk, validity, or whether anyone should sign. Out-of-scope inputs stop before generation and reflection, then still write normal metadata and an empty output file.

Outputs and metadata

Each run creates a fresh directory under data/cuad/runs// by default. The timestamp run id uses local time in YYYYMMDD-HHMMSS format. Inside the run directory:

verification_questions.jsonl
run_metadata.json
run.log

verification_questions.jsonl contains the structured outputs. run_metadata.json records the run settings, scope counts, scope reasons, and row-count metrics: rows_read, rows_filtered, rows_in_scope, rows_out_of_scope, rows_generated, scope_status_counts, out_of_scope_reasons, scope_results, safety_failed_count, and rows_written. run.log records the same safe lifecycle events and row counts without logging API keys, clause text, or model output.

Use --output-dir to change the parent directory, --run-id for deterministic or manual run names, and --verbose for DEBUG logs. The command fails if the run directory already exists, so previous runs are not silently overwritten.

FastAPI adapter

The API is a minimal HTTP boundary over the same workflow. It does not add legal-advice behavior, autonomous tool calling, persistence, auth, background jobs, WebSockets, or run history.

POST /runs accepts a single clause payload, writes a temporary one-row JSONL input internally, executes the existing LangGraph workflow synchronously, and returns workflow observability fields plus generated verification questions.

FastAPI writes local artifacts under data/cuad/api-runs/{run_id}/.

This adapter is intended for local development and internal E2E validation. Do not expose it publicly without path restrictions, authentication, and deployment hardening.

uv run uvicorn contract_question_agent.api.app:app --reload --reload-dir src

Dry-run example:

curl -X POST http://127.0.0.1:8000/runs \
  -H "content-type: application/json" \
  -d '{
    "contract_id": "demo-contract",
    "clause_type": "Non-Compete",
    "evidence_text": "Employee will not compete for one year after termination.",
    "dry_run": true
  }'

AG-UI run event stream

The AG-UI endpoint exposes a minimal human-facing event stream for observing a single workflow run.

It is intentionally not a chat UI. The endpoint streams run lifecycle events over Server-Sent Events:

• RUN_STARTED
• STEP_STARTED
• STEP_FINISHED
• STATE_SNAPSHOT
• RUN_FINISHED
• RUN_ERROR

The AG-UI stream emits node-level workflow status events so the viewer can show which workflow node is currently running, such as PRECHECK_INPUT, CLASSIFY_SCOPE, GENERATE_MINIMAL_QUESTIONS, REFLECT_AGAINST_SKILL_THESIS, and SAFETY_CHECK.

This is observability only. It does not add checkpointing, interrupts, resume support, persistence, auth, WebSockets, or run history.

The event snapshot removes raw evidence_text from generated question outputs. Local artifacts are still written under data/cuad/api-runs/{run_id}/.

curl -N -X POST http://127.0.0.1:8000/ag-ui/runs \
  -H "content-type: application/json" \
  -d '{
    "contract_id": "demo-contract",
    "clause_type": "Non-Compete",
    "evidence_text": "Employee will not compete for one year after termination.",
    "dry_run": true
  }'

Streamlit AG-UI run viewer

The Streamlit viewer is a minimal human-facing observability UI.

It does not call LangGraph directly. It calls the FastAPI AG-UI SSE endpoint:

POST /ag-ui/runs

FastAPI remains the interface boundary, and LangGraph remains the workflow owner. Streamlit does not directly call workflow internals, model clients, or MCP. No upload support, checkpointing, resume support, persistence, auth, WebSockets, or run history is added.

Start the FastAPI backend:

uv run uvicorn contract_question_agent.api.app:app --reload --reload-dir src

Start the Streamlit viewer:

uv run streamlit run viewer/streamlit_app.py

Or start both together:

./scripts/dev.sh

Optional ports:

FASTAPI_PORT=8001 STREAMLIT_PORT=8502 ./scripts/dev.sh

Open the Streamlit URL shown in the terminal.

The viewer submits a single clause, reads AG-UI-compatible lifecycle events, shows the current workflow node and node timeline, and renders the final safe state snapshot. It removes raw evidence_text from rendered backend snapshots. It includes a few built-in synthetic sample clauses for demos and smoke tests, while keeping the manual input path editable. The samples are not drawn from CUAD or real contracts, and upload support is intentionally out of scope for v1.0.0.

This is not a chat UI. It is a run/output viewer.

Example outputs

This repository includes two synthetic example outputs:

• [examples/in_scope_non_compete.md](examples/inscopenon_compete.md)
• a successful in-scope run for a synthetic Non-Compete clause
• [examples/out_of_scope_non_contract_text.md](examples/outofscopenoncontract_text.md)
• an out-of-scope run where non-contract text stops before generation

Both examples are synthetic and are not drawn from CUAD or real contracts.

v1.0.0 scope

Included:

• CUAD data preparation
• CLI generation
• LangGraph workflow
• two-step scope classification
• MCP candidate review lenses
• skill-thesis reflection
• FastAPI / AG-UI / Streamlit viewer
• example outputs

Not included:

• upload support
• auth
• persistence
• run history
• checkpoint/resume
• production deployment hardening
• legal advice
• legal enforceability judgment
• signing recommendation

Non-goals

This project does not:

• answer legal

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mofuteq
• Source: mofuteq/contract-question-agent
• License: MIT

Install and usage instructions live in the source repository linked above.