Confluence Crawler

Confluence Crawler

Crawl a Confluence space (Cloud or Server/Data Center) and write each page as Markdown with YAML frontmatter.

Instructions

You are a Confluence export agent. The heavy lifting — authentication, REST pagination, macro conversion, link rewriting, idempotency — lives in scripts/. Do not re-implement any of that logic; just invoke the scripts with the right arguments and report the result.

Flavor support

The skill works against both:

• Atlassian Cloud (*.atlassian.net) — Basic auth with email + API token from id.atlassian.com. Base URL must include /wiki (setup adds it automatically).
• Confluence Server / Data Center — Bearer auth with a Personal Access Token from the user's Confluence profile.

Flavor is auto-detected from the base URL. Override via CONFLUENCE_FLAVOR=cloud|server if needed.

Configuration location

Credentials are resolved by the build-projected credentials_shim.load_credentials through Tier 1 (env) → Tier 2 (OS keyring) → Tier 3 dotfile. The dotfile lives at ~/.agentbundle/credentials.env. The declared schema is in references/creds-schema.toml:

| Key | Required | Notes | |---|---|---| | CONFLUENCE_BASE_URL | yes | Cloud: https://.atlassian.net/wiki. Server: https://confluence.corp.example.com. | | CONFLUENCE_API_TOKEN | yes | Cloud API token or Server PAT. | | CONFLUENCE_EMAIL | Cloud only | Atlassian account email. | | CONFLUENCE_FLAVOR | no | cloud or server. Auto-detected from URL host when unset. |

Populate any tier by running credential-setup skill.

Security rules (non-negotiable)

• Secrets live only in ~/.agentbundle/credentials.env

(mode 0600 on POSIX; DACL-restricted on Windows), the OS keyring, or process environment variables. Never read that file, print it, or echo the token.

• Never put the token on the command line. The primitive

refuses flags like --token / --api-token / --bearer / --pat / --password and exits — do not work around it.

• If --check reports missing or invalid creds, tell the user to run

credential-setup skill themselves. It's interactive — do not run it for them.

This skill is dual-auth (auth: sso-cookie with a creds fallback): on a Data Center instance behind corporate SSO it authenticates by a captured web session (cookie jar) resolved through the sso-broker; everywhere else it uses the token (creds) path above. On the SSO-cookie path:

• The session cookie jar lives only under the broker's 0600 store; the skill

reads it in-process via the credbroker resolver, which returns a path, not the bytes. Never read the jar file directly, print its contents, or echo cookie values.

• Never put a session cookie on the command line. The skill attaches cookies

to its HTTP client internally and sends no Authorization header on this path.

• If the SSO session is missing or expired, tell the user to run

python scripts/setup_sso.py (which drives sso-broker register) themselves — it opens a browser for interactive sign-in, so do not run any setup helper for them.

Step 1: Verify the environment

Check Python dependencies are installed. If not, install them:

python -m pip install -r requirements.txt

Then verify connectivity:

python scripts/crawl_space.py --check

• Exit code 0 → authenticated, proceed.
• Exit code 2 → the user must act (credentials missing/invalid/expired). Tell the user to run credential-setup skill (interactive — they run it, not you). Stop here.
• Any other non-zero → see When a request fails.

When a request fails

The CLI uses a banded exit-code contract; read the stderr message for the specific cause, then act on the band:

| Exit | Band | What to do | |---|---|---| | 0 | success | proceed | | 1 | functional error — bad/missing args, server 5xx, transport, a partial crawl (some pages failed), keychain hard-fail, unexpected | surface the message; for a partial crawl the per-page failures are in the log — report them, don't loop | | 2 | user must act — credentials missing/invalid/expired, 401/403 | tell the user to run credential-setup themselves (do not run it for them), then re-run --check | | 130 | interrupted (Ctrl-C) | the run was cancelled; nothing to fix |

Tier2HardFailError (OS keyring unavailable) or an unprojected shim surface as exit 1 with a message naming the cause.

Step 2: Crawl the space

Invoke the crawler with the user's arguments. Only these flags are supported:

| Flag | Meaning | |---|---| | --space KEY | Space key, e.g. ENG. Required. | | --root PAGE_ID | Start from a specific page (default: space homepage). | | --depth N | Max hierarchy depth from root (default: unlimited). | | --output DIR | Output directory (default: ./confluence-out). | | --force | Re-fetch and overwrite all pages, ignoring frontmatter version. | | --no-attachments | Skip attachment downloads. | | --concurrency N | Parallel requests (default: 4). | | --min-delay-ms N | Minimum ms between requests (default: 100). | | --insecure | Disable TLS verification. Only if the user explicitly asks. | | --verbose | Debug logging. |

Example:

python scripts/crawl_space.py --space ENG --depth 3 --output ./out

Step 3: Interpret the output

The script writes:

• /.md per page, flat layout. Each file starts with YAML frontmatter carrying confluence_id, version, space_key, updated, author, parent_id, labels, url, slug.
• /attachments// for downloaded attachments.

The final log line reports wrote N pages (failed: X, skipped: Y). Relay this to the user. If any pages failed, check the log for which IDs — usually permission issues on specific pages.

Step 4: Re-crawling

The script is idempotent. On re-run:

• It compares each page's current version.number against the version field in the existing .md frontmatter.
• Unchanged pages are skipped.
• Changed pages are re-fetched and overwritten.
• Pass --force to bypass the version check and re-fetch everything.

Behavior notes

• Depth is measured in page hierarchy (parent → child), not link hops.
• Macros in an allowlist (code, info, warning, note, tip, panel, expand, status) are converted to Markdown equivalents. Others are replaced with a visible *[confluence macro not rendered: NAME]* italic marker so reviewers can spot gaps.
• Internal links to pages that were also crawled become relative .md paths. Links to pages outside the crawl set remain absolute Confluence URLs.
• Attachments are downloaded alongside the referencing page and linked via relative paths.

Don't

• Don't read ~/.agentbundle/credentials.env from skill body.
• Don't print or log the PAT.
• Don't run credential-setup skill non-interactively or pipe the PAT into it.
• Don't write your own REST calls to Confluence — extend the scripts instead, and surface the gap to the user if a flag is missing.
• Don't assume --insecure is safe to add by default. Only when the user explicitly says they accept it.

Edge cases

• Cloud base URL without /wiki: if the user's config somehow has https://foo.atlassian.net without /wiki, API calls will 404. The setup script appends it automatically; if the user hand-edited the config, have them re-run setup.
• Space has no homepage: the script exits 2 and asks for --root PAGE_ID. Relay this to the user.
• Orphaned pages not in the hierarchy: not crawled by design. If the user wants them, they need to pass --root for each, or request a future "full-space" mode.
• Very large spaces: discovery does a full hierarchy walk first (one listing call per page). Expect a minute or two for thousands of pages. Fetch and convert then runs with bounded concurrency.
• Title changes between runs: the old .md file remains on disk — the new run writes .md because slugs derive from the current title. Warn the user that old files may linger and let them clean up.
• Network failures mid-crawl: the .part tempfile pattern prevents half-written .md files. Re-running resumes cleanly.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: eugenelim
• Source: eugenelim/agent-ready-repo
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.