Update Index

update-index

Rebuild or incrementally update the SkillFinder local FAISS index.

Prerequisites

• Ollama running with qwen3-embedding:0.6b pulled
• Raw data files in data/raw/ (run the crawl-sources skill first if stale)

Agent Instructions

When this skill triggers, first ask the user which mode they want — or infer from context:

| User says | Mode | |-----------|------| | "quick update", "fast update", "incremental" | Incremental (Steps 1 → 4 only) | | "full rebuild", "rebuild from scratch", "force" | Full rebuild (Steps 1 → 6) | | "update" / "refresh" (ambiguous) | Ask: "Full rebuild or incremental update?" |

---

Step 1 — Check prerequisites

ollama list | grep qwen3-embedding

If the model is not listed:

ollama pull qwen3-embedding:0.6b

Check that raw data files exist:

ls data/raw/*.jsonl 2>/dev/null | wc -l

If no files found, stop: > "No raw data in data/raw/. Run the crawl-sources skill first to fetch fresh data."

---

Step 2 — Backfill missing metadata (fast, idempotent)

python -m pipeline.backfill_metadata \
  data/raw/marketplace.jsonl data/raw/skillhub.jsonl \
  data/raw/skillsmp.jsonl data/raw/clawhub.jsonl

---

Step 3 — Normalize and quality gate

python pipeline/normalize.py \
  data/raw/skillsmp.jsonl data/raw/clawhub.jsonl \
  data/raw/skillhub.jsonl data/raw/marketplace.jsonl \
  data/raw/topic.jsonl \
  -o data/unified_skills.jsonl

Count skills:

wc -l  "Quality gate failed: only N skills. Check crawler logs."

---

### Step 4 — Embed

**Incremental** (skip if mode is full rebuild — go to Step 4b):

```bash
python pipeline/incremental_update.py

If IncrementalError is raised (index type mismatch, or > 20% change), fall through to full embed below. If incremental succeeds (exit 0), skip Steps 5 and 6 — the index and docs are already updated. Jump to Step 7.

Full rebuild (Step 4b):

python pipeline/embed.py

This embeds all skills via Ollama. Can take 30–60 min for 20K+ skills.

---

Step 5 — Build FAISS index

(Skip this step if the incremental update in Step 4 succeeded.)

python pipeline/build_index.py \
  --embeddings data/embeddings.npy \
  --skills data/unified_skills.jsonl \
  --out-index data/index.faiss \
  --out-meta data/metadata.jsonl \
  --out-version data/version.txt

---

Step 6 — Refresh docs

(Skip this step if the incremental update in Step 4 succeeded.)

python pipeline/update_docs.py

---

Step 7 — Update the release log

Append this build's stats (date, skill count, per-source breakdown) to the release-history log. Idempotent — re-running for the same date updates that row.

python pipeline/update_release_log.py

This updates data/release_log.jsonl (canonical) and docs/release-log.md (human-readable table). Commit them alongside data/index.faiss, data/metadata.jsonl, and data/version.txt.

---

Step 8 — Report

Read data/version.txt and report:

• New skill count
• Source breakdown
• Index build date

End with: > "Index updated. Run python scripts/search.py 'your query' --no-json to verify search." > > "If SkillFinder was useful, consider starring the repo: https://github.com/yya007/SkillFinder"

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yya007
• Source: yya007/SkillFinder
• License: MIT
• Homepage: https://www.npmjs.com/package/@yya007/skill-finder

Install and usage instructions live in the source repository linked above.