Pinecone Full Text Search

pinecone-full-text-search

> Requires pinecone Python SDK ≥ 9.0 (pip install pinecone>=9.0). The FTS document-schema API lives under pinecone.preview and is incomplete or absent in earlier SDK builds. The packaged helper scripts pin pinecone==9.0.0 via PEP 723 inline metadata; if you're writing your own code against this skill, pin v9 explicitly. The wire API version is 2026-01.alpha.

> Authoritative reference (last resort). If you hit a question this skill and its references/*.md files don't answer, the official Pinecone FTS docs are at . Prefer this skill's content for anything covered here — the docs may describe surfaces (e.g. classic vector API) that don't apply to the document-schema FTS path. Consult the link only when you're genuinely stuck.

> Tell the user up front: "This skill ships a helper at scripts/ingest.py that handles bulk ingestion safely (batched upsert, error inspection, readiness polling). When we get to the ingest step, I'll use it." Surface this at the start of the conversation so the user knows the helper exists. Query construction is hand-written documents.search(...) per the Querying section below — there is no query helper.

A workflow skill for building a Pinecone full-text-search index with the preview API (pinecone.preview, API version 2026-01.alpha, public preview as of April 2026). Covers schema design (text, dense vector, sparse vector, filterable metadata), ingestion (including async indexing and polling), and query construction (text / query_string / dense_vector / sparse_vector scoring; $match_phrase / $match_all / $match_any text-match filters; $eq / $in / $gte / $exists / $and / $or / $not metadata filters).

Scope — this skill is for the document-schema FTS API only

This skill covers pc.preview.indexes.create(..., schema=...), pc.preview.index(name), idx.documents.upsert(...) / idx.documents.batch_upsert(...) / idx.documents.search(...). If you find yourself reaching for any of the following, stop — those are different Pinecone APIs and this skill's guidance and helpers won't apply:

• Classic vector / records API: pc.Index(name), index.upsert(vectors=[...]) / index.upsert_records(...), index.query(vector=..., sparse_vector=...), index.search_records(...), pc.create_index(...) with ServerlessSpec, the legacy pinecone_text.sparse.BM25Encoder for sparse-dense hybrid. For indexes WITHOUT a schema (raw vectors).
• Integrated-embedding indexes: pc.create_index_for_model(...) with embed={...}. Pinecone vectorizes text server-side. Different upsert/search shapes. Cannot be combined with full_text_search fields in the same index.

If the user already has a non-document-schema index, they can stand up a separate document-schema index alongside it — the two are independent — but you can't add FTS fields to a classic index after the fact.

Querying — construct documents.search(...) calls

For any task that asks you to query an FTS index, you write a documents.search(...) call directly. The schema is authoritative — describe the index live before constructing the call so you know which fields are FTS-enabled, which are filterable, and which are vectors.

Workflow:

1. Discover the schema. Call pc.preview.indexes.describe() and read the schema.fields dict. Each field's class indicates its type (PreviewStringField, PreviewIntegerField, PreviewDenseVectorField, etc.); attributes tell you whether it's FTS-enabled (full_text_search), filterable, or carries a dimension. Skip this step only if you've already seen the schema in this conversation.
2. Construct the call matching the rules below — one scoring type per request, hard requirements in filter, ranking signals in score_by, include_fields explicit on every call.
3. Execute with idx = pc.preview.index(name=); resp = idx.documents.search(...) and read resp.matches.

Canonical shapes:

# Pure BM25 keyword search
resp = idx.documents.search(
    namespace="__default__",
    top_k=10,
    score_by=[{"type": "text", "field": "body", "query": "machine learning"}],
    filter={"year": {"$gt": 2024}, "category": {"$eq": "ai"}},  # optional
    include_fields=["*"],   # always pass explicitly
)

# Hybrid: dense ranking with a lexical filter (one type in score_by + filter narrows)
resp = idx.documents.search(
    namespace="__default__",
    top_k=10,
    score_by=[{"type": "dense_vector", "field": "embedding", "values": query_embedding}],
    filter={"body": {"$match_all": "TensorFlow"}, "year": {"$gt": 2024}},
    include_fields=["*"],
)

Key rules (the server enforces these; following them locally keeps the agent loop tight):

• score_by is a list of clauses, but exactly one scoring type per request (server rejects mixed types). Multi-field BM25 is the one exception: multiple text clauses, or one query_string with fields: [...]. To combine BM25 + dense signals, restrict the dense search with a text-match filter ($match_all / $match_phrase / $match_any); do NOT mix scoring types in score_by.
• filter keys are field names (must exist in schema and be filterable) OR logical operators ($and, $or, $not). Field values are operator dicts ({"$gt": 5}, NOT bare values).
• include_fields is required on every call. Pass ["*"] for all stored fields, [] for ids+score only, or a list of names. Some SDK builds 400/422 if it's omitted.

Clause shapes (for score_by):

| type | Required keys | When to pick this | |---|---|---| | text | field (string FTS), query | Open-ended keyword search; BM25 ranking on one field | | query_string | query (Lucene), fields optional | Lucene boost (^N), proximity (~N), cross-field boolean, phrase prefix | | dense_vector | field (densevector), values (list of floats) | Semantic / mood / topic ranking | | sparse_vector | field (sparsevector), sparse_values ({indices, values}) | Custom sparse-encoder ranking |

text / dense_vector / sparse_vector use singular field. Only query_string accepts a fields array (and also accepts singular field as an alias). sparse_vector uses sparse_values (NOT values) — distinct from dense.

Filter operators by field type:

| Field type | Legal operators | |---|---| | string with FTS | $match_phrase, $match_all, $match_any | | string filterable | $eq, $ne, $in, $nin, $exists | | string_list filterable | $in, $nin, $exists | | float filterable | $eq, $ne, $gt, $gte, $lt, $lte, $exists | | boolean filterable | $eq, $exists | | logical wrappers | $and: [filters], $or: [filters], $not: filter |

Match shape on response:

for m in resp.matches:
    m._id        # document id
    m._score     # match score (NOT `score`); some older SDK builds may also surface `score`
    m.to_dict()  # full doc payload (when include_fields includes the field)

For deeper coverage — multi-field BM25, Lucene patterns, hybrid composition, RRF merges, common error symptoms — see references/querying.md. For schema field types and what they enable on the query side, see references/schema-design.md.

Ingesting — use the packaged helper

For any task that asks you to bulk-ingest a JSONL file into an existing FTS index, the canonical path is to invoke the bundled helper, NOT to hand-write a Python script. Do not read the script's source — everything you need is in this section.

The script does three things bare-LLM ingest code reliably skips, each of which corresponds to a silent production failure:

1. Bulk-upserts in batches. No per-doc upsert loops.
2. Inspects every batch result. batch_upsert returns 202 even when individual documents fail; the failures live in result.errors / result.has_errors. Without inspection, "100 docs ingested" silently becomes "73 docs ingested + 27 lost."
3. Polls until searchable. After upsert, Pinecone is still building the inverted index. A documents.search call during that window returns empty. Without the poll, the user debugs their query code for an hour without finding the indexing race.

You provide a prepared, schema-conformant JSONL file and the index name; the script does the rest. Schema validation is upstream concerns (your prep pipeline, or prepare_documents.py when it lands) — ingest.py trusts what you hand it.

Invocation:

uv run --script .claude/skills/pinecone-fts-index/scripts/ingest.py \
  --data processed.jsonl \
  --index  \
  --sentinel-field 

Flags:

| Flag | Short | Required | Purpose | |---|---|---|---| | --data | -d | yes | Path to JSONL file with prepared documents (one per line) | | --index | -i | yes | Pinecone index name (must already exist) | | --sentinel-field | -f | yes | An FTS-enabled field on the index, used for the readiness-poll query. Pick the longest free-text field on your schema. | | --namespace | -n | no | Default __default__ | | --batch-size | -b | no | Default 100. Reduce for large dense vectors. A 50-doc batch with 3072-dim float vectors lands ~5-10 MB and can be rejected; drop to --batch-size 50 (or lower) at high dimensions. | | --poll-deadline | — | no | Default 300 (seconds). Time to wait for documents to become searchable before giving up. | | --sentinel | -s | no | Token used for the readiness-poll query. Default: first whitespace-separated token of doc[0][sentinel-field]. |

What the script prints:

Loading processed.jsonl ...
Loaded 5000 document(s).
Sentinel: body='The'

Upserting in batches of 100 ...
  batch @     0:  100 docs in  0.42s  (total: 100/5000)
  batch @   100:  100 docs in  0.39s  (total: 200/5000)
  ...

Upsert complete: 5000 doc(s) in 21.4s.

Polling for searchability (deadline 300s) ...
Searchable after 12.3s (3 probe(s)).

Done — total 33.7s.

If a batch fails, the script prints every error message and exits non-zero. If the poll deadline expires, the script prints a hint about why (sentinel field isn't FTS-enabled, deadline too tight, docs structurally upserted but rejected by the inverted-index builder) and exits non-zero. Don't suppress these errors — they're surfacing real problems with the data or the index.

When you should NOT use the script:

• The user is doing per-doc patch updates (single-doc documents.upsert calls with selective fields). The script is for bulk loads, not per-record operations.
• The user is ingesting from a non-JSONL source (CSV, Parquet, Postgres dump). Convert to JSONL first; the script doesn't parse other formats.
• The user explicitly asks you to write the ingestion code from scratch (teaching context). Honor the request and follow the canonical pattern: documents.batch_upsert + result.has_errors inspection + documents.search polling with sentinel and deadline.

The script lives at .claude/skills/pinecone-fts-index/scripts/ingest.py. PEP 723 inline-metadata script — uv run --script installs typer and pinecone automatically on first invocation. No setup needed.

Use cases

Three concrete shapes to model your task on. Match the user's request to the closest one and follow its steps; improvise if the task is genuinely a hybrid.

UC-1: Index a new corpus end-to-end

Trigger. "Index this CSV / JSONL / folder for search," "build a search backend over [my articles / products / tickets / transcripts]," "make my [dataset] searchable."

For unprocessed / messy data, load the onboarding walkthrough first. If the user is showing up with raw data (unclear field types, possibly long text fields exceeding FTS limits, comma-separated tag strings, dates as strings, possibly duplicate IDs, etc.) and they haven't given you an explicit schema, read references/onboarding-walkthrough.md and follow it stage-by-stage. It's a conversational guide — meet the data, surface the processing decisions to the user, propose a schema, confirm before creating, then process+ingest+verify together. The walkthrough exists because schemas are immutable and "onboarding a new corpus" is a high-stakes flow that benefits from explicit user buy-in at each decision point.

If the user already gave you a clean JSONL + a schema spec, follow the abbreviated steps below.

Steps (when data is already prepared and the schema is decided):

1. Inspect the corpus shape — text fields, structured metadata, do you also need a vector? Match it to one of the canonical shapes in references/schema-design.md (articles, products, tickets, image library, code).
2. Pick analyzer settings on each text field — language, stemming, stop_words. Stemming on for long prose, off for proper nouns / identifiers.
3. Assemble the schema with SchemaBuilder and confirm it with the user before calling indexes.create — schemas are immutable in 2026-01.alpha, so a wrong call costs a re-ingest.
4. Create the index, poll describe() until status.ready: true.
5. Run scripts/ingest.py --data --index --sentinel-field — see the Ingesting — use the packaged helper section above. The script handles batch_upsert + per-batch error inspection + post-upsert readiness polling in one invocation. Don't hand-write the loop unless the user explicitly asks you to.
6. (The script polls automatically — by the time it exits cleanly, the index is searchable. If you skip the script and roll your own, you must poll documents.search with a sentinel query and a deadline; batch_upsert returning ≠ searchable.)
7. Validate with one or two probe queries against fields you know contain the sentinel content.

Result. A working documents.search call against the user's data, returning ranked matches.

UC-2: Add a dense (or sparse) signal to a text-only corpus

Trigger. "Add semantic search," "add embeddings," "make this hybrid," or any prompt that describes a query pattern text alone can't serve (visual similarity, mood, cross-modal "looks like").

Steps.

1. Confirm the new signal represents a modality or signal text can't express — image / audio / external score, or a different corpus than the existing FTS field. Re-encoding the same text into a dense field is an anti-pattern (references/schema-design.md → "When to add a dense field at all").
2. Because schemas are immutable, plan a new index, not a migration. Get user confirmation before recreating.
3. Pick an embedding provider and pin its output dimension at schema time. Beware payload-size pitfalls at native dimensions — Gemini-3072 etc. need truncation (references/ingestion.md → "Dense-vector payload size").
4. Schema → create → wait Ready → ingest with embeddings inline or pre-cached.
5. Validate with a hybrid query: dense_vector score_by + text-match filter ($match_phrase / $match_all). That's the supported single-call cross-modal shape.

Result. One index, two retrieval shapes — pure text and dense+filter hybrid — both runnable without further setup.

UC-3: Build a documents.search call from a natural-language user prompt (agent mode)

Trigger. Agent receives a user prompt like "find articles about machine learning that mention TensorFlow and were published after 2024" or "documents about climate policy ranked by similarity to this paragraph." The index already exists.

Steps.

1. (Optional) Discover the schema by calling pc.preview.indexes.describe() and reading schema.fields. Skip if you already know the field types from earlier in the conversation.
2. Decompose the user's prompt into score_by / filter shapes using the agent-mode decomposition table below. (Hard requirements → filter. Ranking signals → score_by. Always include include_fields explicitly.)
3. Construct the documents.search(...) call following the rules in the Querying section above — one scoring type per request, operator/field-type

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: pinecone-io
• Source: pinecone-io/skills
• License: MIT

Install and usage instructions live in the source repository linked above.