Ctxlayer

ctxlayer

[](LICENSE)

[](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)

Agent context layer — an MCP service on Cloudflare that:

• serves curated org docs (Markdown, with Vectorize-backed RAG search) as MCP

resources and a search_docs tool — and speaks the Open Knowledge Format (OKF) in and out, so an org's library round-trips with any OKF bundle;

• proxies other MCP servers (HTTP/SSE natively; a stdio MCP server is

supported via bring-your-own-bridge — run your own stdio↔HTTP bridge and register its URL as a streamable_http upstream), centralising per-user credentials sealed at rest;

• exposes a React + Vite SPA for self-onboarding, BlockNote + Yjs

collaborative markdown editing, admin upstream management, and usage analytics.

Screenshots

| | | | --- | --- | | [](docs/screenshots/admin-upstreams.png) | [](docs/screenshots/usage.png) | | Curate & gate upstream MCP servers — cache their tools, attach org playbooks + docs per tool | Usage analytics — calls, tokens, and top tools per user / upstream | | [](docs/screenshots/connect-upstreams.png) | [](docs/screenshots/skills.png) | | Self-service onboarding — connect upstreams via OAuth or a personal token, sealed at rest | Curated skills — procedural playbooks agents load on demand |

Per-upstream config — transport, auth strategy, resilience caps, and team/product visibility

New here? [CONTRIBUTING.md](CONTRIBUTING.md) is the human-contributor on-ramp (setup, the change loop, conventions); the [Quickstart](#quickstart-contributors-hacking-on-ctxlayer) below is the copy-pasteable version. The architecture & data-model reference is [docs/PLAN.md](docs/PLAN.md) (the milestone-driven plan that built ctxlayer is retired; PLAN.md is now a reference, not a roadmap). Briefing for AI agents working in this repo is [CLAUDE.md](CLAUDE.md) / [AGENTS.md](AGENTS.md). Architectural conventions and gotchas live in docs/plan/G-conventions.md.

> Integration surfaces. The supported, stable contract for external > clients is the MCP surface (/mcp, /sse) plus the OAuth provider. > The /api/* REST endpoints are an internal contract for the bundled > SPA — not versioned and subject to change between releases. Build agents > and scripts against MCP, not /api.

Current state

Everything described in [docs/PLAN.md](docs/PLAN.md) is shipped and deployed. The milestone framing below is kept as a feature inventory, not a roadmap (the milestone-driven plan is retired — see CLAUDE.md).

| Area | Status | What works | |---|---|---| | M1 — Skeleton + sign-in | ✅ done | GitHub / Google sign-in with allowlist, real /api/me | | M2 — Docs + RAG via MCP | ✅ done | BlockNote editor with revisions, R2 snapshots, Vectorize embedding pipeline, MCP server at /mcp with search_docs / get_doc / whoami / list_my_context / list_upstreams, doc resources, admin teams + products + tags | | M3 — Realtime collab (Yjs) | ✅ done | DocRoomDO over WS Hibernation, BlockNote Yjs extension, awareness-leader REST autosave, R2-backed snapshots | | M4 — Upstream proxy (HTTP/SSE + OAuth) | ✅ done | AES-GCM creds, MCP SDK Client for Streamable HTTP / SSE, namespaced tool aggregation, JSON-Schema → Zod schema preservation, full admin UI for upstreams, user /upstreams page with paste-bearer + OAuth. Validated end-to-end against Notion MCP via Claude Desktop — search, fetch, create-page. | | M5 — Admin polish | ✅ done | Admin Users (promote/demote + revoke creds), shared_bearer storage, admin Audit log viewer (/app/admin/audit), admin OAuth-clients viewer (/app/admin/oauth-clients), real /app/mcp-setup with per-client snippets. Bundled side features: folder organisation for docs, per-doc lock, modal-dialog system, doc-move UI | | M6 — Usage pipeline + dashboards | ✅ done | Per-user/upstream call + token charts, tiktoken consumer, daily rollups, admin + user usage pages | | Skills — curated playbooks | ✅ done | Published skills surface over MCP (list_skills / get_skill + mcp://ctxlayer/skills/{slug}), per-upstream/per-tool attachments, admin skill editor, CLI draft-skill | | Git sync — code docs from repos | ✅ done | Register a GitHub repo (PAT or OAuth), mirror Markdown into the doc store, product-link auto-tagging, scheduled cron sync | | Stdio upstreams — bring-your-own-bridge | ✅ supported | Run your own stdio↔HTTP bridge; register its URL as a streamable_http upstream | | OKF interop — Open Knowledge Format | ✅ done | Import/export/git-write-back of OKF docs; the editor rail is the YAML frontmatter editor; unknown keys preserved for round-trip. Deep-dive: [docs/plan/M-okf.md](docs/plan/M-okf.md) | | OKF bundles — directory up/download | ✅ done | Export a folder subtree as a tar.gz / zip OKF bundle (generated index.md/log.md); import an archive under a target folder. OKF-native path-based doc links with a consistency graph. Deep-dive: [docs/plan/N-okf-bundles.md](docs/plan/N-okf-bundles.md) |

Open Knowledge Format (OKF) — early adopter

ctxlayer natively speaks the Open Knowledge Format, Google Cloud's open convention for agent-friendly knowledge: a directory of Markdown files with YAML frontmatter. ctxlayer's doc library is, structurally, an OKF bundle — so an org's curated context is portable, not locked in.

• Import — register an OKF git repo as a source (or paste/upload a .md).

Frontmatter (type, description, resource, tags, title, …) is parsed onto the doc; unknown / extra producer keys (okf_version, etc.) are kept verbatim.

• Edit — the doc editor's right rail is the frontmatter editor. The OKF

fields carry an OKF badge with a tooltip pointing at the matching spec key.

• Export — "Export as OKF (.md)" in the rail, or `GET

/api/docs/:id/export`, emits a spec-compliant file: synthesised frontmatter (rail fields + preserved unknown keys) followed by the body.

• Git write-back — edits to a doc synced from an OKF repo propose a PR that

keeps and refreshes the frontmatter block.

OKF tags map to ctxlayer's free-form tags (not the team/product tags, which gate visibility). Full mapping, round-trip contract, and fidelity caveats: [docs/plan/M-okf.md](docs/plan/M-okf.md).

Quickstart (contributors hacking on ctxlayer)

These steps are for local development of this codebase. End users of a deployed ctxlayer and operators standing it up don't need any of this; see [Deploying ctxlayer](#deploying-ctxlayer-to-production) below.

brew install mkcert nss           # macOS contributors only; see docs/plan/G-conventions.md G11 for Linux/Windows
bun install
cp .dev.vars.example .dev.vars    # then edit it — see "Filling in .dev.vars" below
bun run migrate:local             # apply D1 migrations to the local (miniflare) DB
bun run seed:local                # load fixture teams / products / upstreams / docs
bun run dev                       # or split-terminals: dev:worker + dev:web (recommended)
bun run verify                    # typecheck + lint (Biome) + unit + integration tests (all offline)

Filling in .dev.vars

To sign in locally you need at least one IdP. The quickest is a GitHub OAuth app ( → "New OAuth App"):

• Authorization callback URL: https://localhost:8787/idp/github/callback
• Put its client id/secret in GITHUB_CLIENT_ID / GITHUB_CLIENT_SECRET.
• Set ALLOWED_GITHUB_USERS= — the allowlist gates who may sign in.
• Set ADMIN_EMAILS= so the admin pages are reachable.
• Generate ENCRYPTION_KEY and SESSION_COOKIE_SECRET with

openssl rand -base64 32 each.

• PUBLIC_BASE_URL is already set to https://localhost:8787 in the example.

Local dev needs no Cloudflare account — miniflare emulates D1, KV, R2, and Queues offline. Workers AI and Vectorize have no local emulator, so search_docs returns nothing locally (the reindex consumer soft-skips Vectorize in dev) — that's expected; exercise RAG end-to-end against a real deploy. bun run verify is fully offline; bun run verify:full additionally runs the smoke suite, which needs a running Worker (bun run dev:worker) or a preview URL.

The first dev run calls scripts/setup-dev-tls.mjs via the predev hook and generates a locally-trusted cert in .dev-tls/. Both Vite and Wrangler then serve HTTPS on localhost — required for the __Host- session cookie to work in dev. The cert never leaves your machine.

Backend + frontend in separate terminals (recommended)

When you're debugging the worker (especially OAuth, MCP, or anything where elided stack traces would bite), run each process in its own terminal so streams don't interleave and wrangler's interactive UI works correctly:

# Terminal 1 — worker only (wrangler on https://localhost:8787)
bun run dev:worker

# Terminal 2 — SPA only (vite on https://localhost:5173)
bun run dev:web

To defeat wrangler's log-elision (which folds long stack traces into [N lines elided]), pipe to a file:

bun run dev:worker 2>&1 | tee worker.log
# in another pane:
tail -f worker.log | grep -E '\[oauth\]|\[catalogue\]'

One-window combined runner

bun run dev                       # concurrently with worker,web prefix-coloured streams

Convenient for SPA-focused work; less ideal when chasing a backend bug because the two streams share one TTY and wrangler's elision is heavier.

Wiring Claude to a local ctxlayer

mcp-remote shims a remote MCP server into Claude Desktop's stdio MCP interface and handles the OAuth dance. Because localhost uses an mkcert root that Electron-bundled Node won't trust by default, point Node at the mkcert CA explicitly.

mkcert -CAROOT
# /Users//Library/Application Support/mkcert

~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "ctxlayer-local": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://localhost:8787/mcp"],
      "env": {
        "NODE_EXTRA_CA_CERTS": "/Users//Library/Application Support/mkcert/rootCA.pem"
      }
    }
  }
}

Fully quit Claude Desktop (⌘Q) and relaunch. On first run mcp-remote opens a browser tab for OAuth + GitHub sign-in; tokens persist in ~/.mcp-auth/.

Before connecting Claude, connect upstreams in the browser first at https://localhost:5173/upstreams — the proxy registry only registers proxied tools (notion__*, etc.) for users who have stored credentials at session-init time.

Deploying ctxlayer to production

If you're standing up an instance of ctxlayer for your org (no source edits), you don't need bun run dev or mkcert. Cloudflare's edge provides real HTTPS for the public hostname automatically.

The install has four phases: (1) provision Cloudflare resources, (2) configure at least one identity provider, (3) set deployment secrets, (4) deploy and pin to a custom domain. The step-by-step sequence is in the four numbered subsections below.

1. Provision Cloudflare resources

wrangler login                  # or set CLOUDFLARE_API_TOKEN + CLOUDFLARE_ACCOUNT_ID
bun install
bun run bootstrap               # provisions D1, KV, R2, Vectorize, both queues
                                # and patches the IDs into wrangler.toml
bun run migrate:remote          # applies migrations 0001..N to remote D1

bootstrap is idempotent — re-runnable any time. The manual fallback (if you're not using the script) is:

wrangler d1 create ctxlayer
wrangler kv namespace create OAUTH_KV
wrangler r2 bucket create ctxlayer-docs
wrangler vectorize create ctxlayer-docs --dimensions 768 --metric cosine
wrangler queues create ctxlayer-usage
wrangler queues create ctxlayer-reindex
wrangler queues create ctxlayer-git-sync
# then paste the printed IDs into the -marked slots in wrangler.toml

2. Identity provider configuration

ctxlayer ships GitHub and Google sign-in. Enabling an IdP is a two-sided contract: the OAuth app at the provider, and the matching client-id/secret + allowlist on the worker. At least one IdP must be enabled — the sign-in page hides any IdP whose *_CLIENT_ID secret is unset, and the allowlist is what decides who can sign in under the default open_domain policy (§2d covers the request/invite alternatives).

> Plan the hostname first. Every IdP callback URL bakes in > PUBLIC_BASE_URL. If you intend to use a custom domain (recommended > — see §4 below), decide it now so you register callbacks against the > real hostname once, instead of re-registering after a workers.dev > bootstrap.

2a. GitHub OAuth App

1. GitHub → Settings → Developer settings → OAuth Apps → New OAuth App.

(Org-owned apps live under the org's settings; personal apps under the user's. Org-owned is preferable for centralised control.)

1. Fill in:

• Application name: e.g. ctxlayer (acme)
• Homepage URL: https://ctxlayer.acme.com
• Authorization callback URL: https://ctxlayer.acme.com/idp/github/callback
• Enable Device Flow: leave off.

1. Click Register application, then Generate a new client secret.

Save the client ID and the secret — the secret is shown only once.

1. If you plan to require org membership (ALLOWED_GITHUB_ORG):

• The worker requests scope read:org and calls

GET /user/orgs to verify membership. Org owners may need to approve third-party access under Organization → Settings → Third-party access → OAuth app policy.

• Org-owned OAuth apps skip that approval flow.

1. Set the client creds (secrets) and the login allowlist (a non-secret

[vars] value, injected at deploy from .prod.vars): ``bash wrangler secret put GITHUB_CLIENT_ID wrangler secret put GITHUB_CLIENT_SECRET # ALLOWED_GITHUB_USERS is a [vars] value — do NOT wrangler secret put it # (that collides with the empty [vars] default: Cloudflare error 10053). # Put the logins in the gitignored .prod.vars; scripts/deploy.mjs injects # them at deploy time (same pattern as PUBLIC_BASE_URL): echo 'ALLOWED_GITHUB_USERS=alice,bob' >> .prod.vars # comma-separated logins # — or, instead of (or in addition to) ALLOWED_GITHUB_USERS — # ALLOWED_GITHUB_ORG is declared in wrangler.toml [vars]; edit the # value to your org slug ("acme-inc") and deploy. Leaving both empty # disables GitHub sign-in entirely. ` A user passes the allowlist when they're in ALLOWEDGITHUBORG **or** their login is in ALLOWEDGITHUBUSERS. ALLOWEDGITHUBUSERS` is the cheap path (no extra API call), checked first.

2b. Google Workspace / Google OAuth Client

1. Google Cloud Console → APIs & Services → OAuth consent screen.

• User type: Internal for a Workspace deployment scoped to

your domain (recommended); External if you'll allow personal Google accounts via per-email allowlist.

• Scopes: openid, email, profile.

1. Credentials → Create Credentials → OAuth Client ID.

• Application type: Web application
• Authorized JavaScript origins: https://ctxlayer.acme.com
• Authorized redirect URIs: https://ctxlayer.acme.com/idp/google/callback

1. Save the client ID and secret.
2. Set the secrets and allowlist:

``bash wrangler secret put GOOGLE_CLIENT_ID wrangler secret put GOOGLE_CLIENT_SECRET # ALLOWED_GOOGLE_HD lives in wrangler.toml [vars] — set it to your # Workspace hosted domain (e.g. "acme.com") so Google forces the # account chooser to that domain and the worker verifies the # hd` claim on the returned i

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: stevenn
• Source: stevenn/ctxlayer
• License: MIT

Install and usage instructions live in the source repository linked above.