Open Index

Open Index

[](https://github.com/DrDroidLab/open-index/actions/workflows/tests.yml) [](https://discord.gg/AQ3tusPtZn)

Open Index is a tool for building domain specific accurate, structured data that agents can actually operate on — and for keeping that data correct as things change.

You use Open Index to build a brain: a searchable, continuously-improving context graph of your domain. A brain is domain-agnostic — model a support org (product → "has common issue" → issue), a sales pipeline (customer → order), your infrastructure (service → runbook), or anything else. You define the concepts; Open Index stores them, searches them, and draws the map.

A brain is built from four primitives:

• doc_type — a concept you want to track and maintain (e.g. service, customer, issue).
• docschema — the fields stored for a given doctype.
• entity — one instance of a doc_type, stored per its schema. Every entity can link to others via related_to (the target) + relationship_edge_meaning (free-text edge semantics).
• connector — an optional source you extract entities from (e.g. an MCP server).

Quickstart

pip install -e '.[all]'          # core + UI (Streamlit) + MCP server

# Try the bundled example (support brain: products, issues, segments, comments)
open-index index --brain examples/support-brain
open-index ui    --brain examples/support-brain      # open the Map tab, pick an anchor

# Or start your own brain from scratch
open-index init my-brain
open-index add-doc-type customer --brain my-brain
# ...add entities under my-brain/entities/**/*.json...
open-index index --brain my-brain
open-index ui    --brain my-brain

Prefer containers, or need a brain several agents share? → [docs/deployment.mdx](./docs/deployment.mdx) (docker compose --profile sqlite up).

Commands

| Command | What it does | |---|---| | open-index init [dir] | Scaffold a new brain directory. | | open-index add-doc-type | Add a doc_type schema stub under doc_types/. | | open-index add-entity | Validate + store an entity JSON file. | | open-index import | Bulk-import entities from JSON / JSONL / CSV. | | open-index index | (Re)load entities/**/*.json into the search index. | | open-index validate | Validate brain.yaml, schemas, and every entity file (use in CI). | | open-index ingest | Run a connector now to pull entities from an MCP server. | | open-index run [--force] [--loop N] | Run every connector whose schedule is due (wire into cron/CI). | | open-index search [-t doc_type] | Search from the terminal. | | open-index ui | Launch the Streamlit explorer (Explore / Map / Analytics / Jobs). | | open-index mcp [--read-only] | Run the MCP context layer over stdio. Read+write by default; --read-only opts out of writes. | | open-index serve [--port --token --read-only] | Serve the MCP context layer over HTTP for remote agents (bearer-token auth). | | open-index serve --brains | Serve every brain under a directory from one process, each at //mcp. | | open-index mcp-config [--url --token] | Print the MCP connection block to paste into your agent. |

A context layer for domain-specialized agents

Open Index is designed to sit behind agents specialized for a domain—legal, marketing, customer support, sales, infrastructure, or a domain of your own. The MCP server gives those agents structured context and a validated way to keep that context current:

• agent prompt — dynamic domain navigation is published through MCP server

instructions so supporting hosts can inject it before the first turn

• read — navigation_guidelines() refreshes those instructions;

search_brain() and get_entity() retrieve domain context

• write — put_entity() (add/update an entity), create_doc_type() (define a concept)

Read and write is the default MCP mode so a domain agent can both use knowledge and maintain it. Add --read-only when the agent should consume context without mutating it. Claude Code is supported as one optional MCP client; open-index init scaffolds .mcp.json, CLAUDE.md, and an editing skill as conveniences for it.

Portable agent setup skill

[skills/setup-open-index/SKILL.md](./skills/setup-open-index/SKILL.md) follows the portable Agent Skills SKILL.md format used by agent runtimes including OpenClaw, Hermes, and Claude Code. Give or install this skill in the selected runtime when the agent should set up Open Index itself. It covers installation, domain-brain initialization, generic MCP wiring, default read/write verification, the --read-only opt-out, and production guardrails.

A brain on disk

my-brain/
  brain.yaml            # name + storage/search backend
  doc_types/*.yaml      # one schema per doc_type (fields, boosts, display color)
  entities/**/*.json    # entities, with related_to edges
  connectors/*.py       # optional ingestion scripts (MCP → entities)

Storage defaults to SQLite + FTS5 (zero external services). The backend sits behind a pluggable interface with two implementations: SQLite (default, local/dev) and OpenSearch (select with search.backend: opensearch — see [Using the brain from a cloud agent](#using-the-brain-from-a-cloud-agent-production)).

Where entities live — storage: file | index

Each doc_type declares its source of truth, so curated and machine-generated data don't fight over git:

• storage: index (default) — the search DB owns these entities; they are

not written to files. Right for connector-pulled, high-volume, or temporal data (hundreds of services, memories, alerts) that would otherwise churn the repo.

• storage: file — JSON files under entities// are the source of

truth, git-tracked and PR-reviewable. Right for curated, human/agent-authored entities.

open-index index reconciles file-backed types from disk on each run and leaves index-backed entities (written by connectors/agents) untouched. So brain.db is durable state for index-backed types — back it up or re-ingest; it's gitignored by default.

Creating a brain, step by step

open-index init scaffolds the directory below; then you author two kinds of file — doctypes (schemas) and entities (instances). Sample doctypes: infra (service, datastore, dashboard, runbook, alert), sales (lead, deal, account), lending (loan, borrower, application), or personal (goal, project, person, area, note). Three runnable examples ship in [examples/](./examples): support-brain, infra-brain, and personal-brain.

1. Define a doc_type

A doc_type is a concept plus its schema — one YAML file in doc_types/:

# doc_types/service.yaml
doc_type: service
description: A deployed service.
storage: file                 # file = git source of truth · index = DB-owned (default)
display:
  label_field: name
  color: "#7c3aed"
schema:
  fields:
    - { name: name,        type: string, search: syntactic, boost: 6 }   # weighted 6× in ranking
    - { name: description, type: text,   search: semantic }
    - { name: owner,       type: string, search: syntactic }
relationships:                # the correlations this type uses — optional but recommended
  - { name: "writes to",       target_doc_type: datastore }
  - { name: "is monitored by", target_doc_type: dashboard }

• boost sets per-field search weight — a hit in a boost: 6 title outranks a

boost: 1 description hit 6-to-1. Optional; defaults to 1.

• relationships declares the edge vocabulary so correlations are discoverable

(shown in the UI + navigation guide) and lightly validated (right target type). Optional — entities may still use undeclared meanings.

Create one with open-index add-doc-type service (writes a stub you edit), or ask your agent.

2. Add entities

An entity is one instance. For storage: file types, write one JSON per entity under entities//:

// entities/service/checkout.json
{
  "doc_type": "service",
  "id": "service:checkout",
  "name": "Checkout",
  "owner": "payments-team",
  "related_to": [
    { "target": "datastore:postgres-main",     "relationship_edge_meaning": "writes to" },
    { "target": "dashboard:checkout-latency",   "relationship_edge_meaning": "is monitored by" }
  ]
}

• id must be :.
• related_to is the reserved correlation field present on every entity — it

defines the graph edges (target + relationship_edge_meaning). This is how you say "this ticket is about that service" without any graph database.

Then open-index index (loads file-backed entities) and open-index validate.

3. Populate at scale (four ways, one validated store)

1. Manual / agent — write JSON, or open Claude Code in the folder and let it call

put_entity / create_doc_type over MCP.

1. Bulk — import a file directly, or let an agent write a batch in one call with

put_entities:

``bash open-index import issues.csv --doc-type issue --asserted-by import:jira open-index import export.jsonl --dry-run # validate first, write nothing ``

JSON arrays, JSONL, and CSV all work. Bare slugs are qualified (checkout → product:checkout), CSV scalars are coerced, and a related_to column takes target|meaning pairs separated by ;. A bad row is reported and skipped — the rest still land. --asserted-by / --confidence attribute the whole batch once instead of per row.

1. Connectors — connectors/*.py pull from an MCP server on a schedule; run with

open-index ingest or open-index run (cron/CI-friendly).

1. Agent write-back — a Stop hook that records learnings via put_entity (the

"continuously improving" loop).

See [Entity Management](./docs/guides/entity-management.mdx) for guidance on cadence and decay.

4. Explore

open-index ui opens a read-only explorer. The sidebar always shows every doc_type with its count and storage policy, so the structure is visible without navigating anywhere. Four tabs: Explore (search + browse + drill into an entity's relationships), Map (auto-anchored on the most-connected entities — click any node to expand it), Analytics (what context CLI/MCP/UI clients fetched, and how often — zero-result searches show what to model next), and Jobs (connectors and their schedules).

Using Open Index as your agent's context layer

open-index mcp runs an MCP server (stdio) exposing the brain to any MCP client — read and write by default:

• The server publishes dynamic, brain-specific instructions as part of the agent

prompt so supporting hosts can navigate the domain before the first tool call.

• navigation_guidelines() — refresh that guide after the index/schema changes.
• search_brain(query, doc_types, limit) · get_entity(id) — read.
• put_entity(...) · put_entities([...]) · create_doc_type(...) — write (validated,

honors the storage policy). put_entities writes a whole batch in one call and takes a shared provenance block.

Use open-index mcp --read-only (or open-index serve --read-only) to opt out when an agent should retrieve domain context but never maintain it.

Local context-fetch analytics

CLI and MCP searches, entity fetches, and navigation-guide reads are recorded in the user's local state directory (~/.local/state/open-index/), outside the brain checkout. The Analytics tab shows fetch counts by client/operation, frequently fetched queries or entity IDs, latency, failures, zero-result searches, and recent activity. This file stays local and is never sent to Open Index's creators.

open-index init also includes optional Claude Code conveniences: .mcp.json, a CLAUDE.md describing durable editing workflows (not runtime navigation), and an edit-brain skill. They are one client integration, not a requirement for building legal, marketing, support, or other specialized agents on Open Index.

Using the brain from a cloud agent (production)

📖 [Full deployment guide → docs/deployment.mdx](./docs/deployment.mdx) — local, remote (with and without Docker), TLS/proxying, and exactly what to paste into Claude Code, Claude Desktop, or Cursor.

Getting the connection details

Never hand-assemble the config. Ask for it:

open-index mcp-config --brain ./my-brain                        # local (stdio)
open-index mcp-config --url brain.acme.internal:8080 --token $OPEN_INDEX_TOKEN
open-index mcp-config --url https://brain.acme.com --token $TOKEN --cli   # `claude mcp add …`
open-index mcp-config --brain ./my-brain > .mcp.json            # pipes where it belongs

open-index serve prints the same details on startup — including the addresses a remote client can actually reach. (The bind address it listens on, 0.0.0.0, is not one of them.) Behind a proxy or tunnel, pass --public-url so what's printed is what agents should use.

With Docker (recommended for a shared brain)

cp .env.example .env          # set OPEN_INDEX_TOKEN + BRAIN_DIR

docker compose --profile sqlite     up --build    # single writer, no extra services
docker compose --profile opensearch up --build    # many writers, incl. the cluster

Both serve http://localhost:8080/mcp. Your brain.yaml is identical either way — the profile sets OPEN_INDEX_SEARCH_BACKEND, which overrides the file. Add --profile ui for the explorer on :8501. The brain directory is mounted, not baked into the image, so doc_types and entities stay in git.

Without Docker

pip install 'open-index[serve]'                  # add ,opensearch for that backend
open-index index --brain /srv/acme-brain         # load file-backed entities first
OPEN_INDEX_TOKEN=… open-index serve --brain /srv/acme-brain --port 8080

serve exposes the same read+write tools over streamable HTTP with bearer-token auth. Without a token the endpoint is unauthenticated — anyone who can reach the port can write to your brain. Use --read-only for a queryable endpoint that agents can't mutate.

Choosing a backend

SQLite is single-writer. That, not entity count, is the line: the moment a second agent needs to write, move to OpenSearch. It also gives native per-field boosting, fuzzy (typo-tolerant) search, and k-NN semantic search that scales past SQLite's ~10k-entity brute-force ceiling.

Select it per-environment without touching brain.yaml:

export OPEN_INDEX_SEARCH_BACKEND=opensearch
export OPEN_INDEX_OPENSEARCH_HOSTS=https://opensearch.internal:9200

…or commit it, with secrets as ${ENV} refs resolved at connect time:

search:
  backend: opensearch
  hosts: ["https://opensearch:9200"]
  index: open_index_acme          # optional; defaults to open_index_
  username: "${OPENSEARCH_USER}"
  password: "${OPENSEARCH_PASSWORD}"
  use_ssl: true
  verify_certs: true

The doc_type/file-entity part comes from git; index-backed data lives only in the cluster (or brain.db), so give it a persistent home and a backup. Rule of thumb: local/dev → SQLite; shared endpoint → OpenSearch + serve.

Controlling search

📖 [Full configuration reference → docs/guides/search-configuration.mdx](./docs/guides/search-configuration.mdx) — decision tables for storage: file | index, SQLite vs OpenSearch, and every search knob.

Schema (per field): data type (string/number/boolean/timestamp), processing (keyword/text/timestamp), and search kind (syntactic = keyword+prefix, semantic = vector-backed dense search, none = not indexed). Mark a field search: semantic and the backend automatically embeds it at index time.

Ranking — genuine per-field boosters: each field's boost weights how much a match there counts, so you tune "title matters more than description" with one number. For hybrid queries, keyword and semantic scores are blended with `search.sem

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: DrDroidLab
• Source: DrDroidLab/open-index
• License: MIT

Install and usage instructions live in the source repository linked above.