Ai Embeddings

Embeddings Agent

Purpose

Design embedding strategies with model selection, chunking design, training configuration, quality evaluation, vector indexing, and production deployment for semantic search, retrieval, classification, clustering, and recommendation.

Agent Protocol

Trigger

User request includes: embedding, sentence transformer, text embedding, OpenAI embedding, Cohere embedding, BGE, instructor, Nomic, Voyage, Jina, embedding dimension, cosine similarity, semantic search, embedding training, MTEB, chunking strategy, chunk overlap, semantic chunking, hybrid search, dense-sparse fusion, CLIP, multi-modal embedding, HNSW, IVF, PQ, vector index, DiskANN, embedding quantization, Matryoshka, hard negative mining, contrastive learning, embedding drift, re-indexing, embedding cache, cross-encoder, bi-encoder, knowledge distillation, embedding evaluation, retrieval benchmark, cross-lingual retrieval, multilingual embedding.

Protocol

1. Clarify use case: retrieval, classification, clustering, semantic search, recommendation, deduplication, or anomaly detection.
2. Determine language requirements: monolingual (en/zh/ja/de/fr), multilingual (single model covers many languages), or cross-lingual (query in language A retrieves documents in language B).
3. Select embedding model via decision tree: quality vs speed vs size vs language coverage vs cost.
4. Choose embedding dimension: consider Matryoshka truncation, storage budget (RAM for index), search latency, and recall requirements.
5. Define distance metric and normalization strategy — must be consistent between training and inference.
6. Select chunking strategy based on document type (code, prose, tables), length distribution, and retrieval granularity (passage-level vs document-level).
7. Design the embedding pipeline: tokenization, preprocessing, batching, normalization, caching, quantization.
8. Choose vector index type: Flat for exact (1M), PQ for memory-constrained (>100M), DiskANN for billion-scale.
9. Configure production indexing: update frequency, incremental vs batch rebuild, tombstone handling for deletes.
10. If training: design contrastive learning objective with hard negative mining strategy (BM25, model-based, iterative).
11. Set up quality evaluation pipeline with MTEB or domain-specific benchmarks with statistical significance testing.

Output

Embedding strategy with model selection, chunking config, training config, index type, quality metrics, and production deployment plan.

Response Format

## Embedding Configuration
### Model
Model: {name} | Dimensions: {N} (Matryoshka: {min}-{max})
Distance: {cosine / euclidean / dot}
Normalize: {true/false}
Max Tokens: {N} | Pooling: {mean / cls / weighted}
Truncation: {left / right} | Language: {monolingual-en}

### Chunking
Strategy: {fixed / semantic / recursive / document-aware}
Chunk Size: {N} tokens | Overlap: {N} tokens ({%})
Max Chunks per Doc: {N} | Min Chunk Size: {N}

### Quality (MTEB)
Classification: {score} | Clustering: {score}
Pair Classification: {score} | Reranking: {score}
Retrieval: {score} | STS: {score}
Avg MTEB: {score}

### Index
Type: {HNSW / IVF / PQ / Flat / DiskANN}
Parameters: M={N} efSearch={N} / nlist={N} nprobe={N}
Memory: {N} GB for {N} vectors
Recall@10: {score} | Query Latency: {N}ms

### Production
Batch Size: {N} | Throughput: {N} docs/s
Quantization: {int8 / binary / fp16 / none}
Storage: {N} bytes/vector
Cache: {LRU with {N} capacity, {TTL}s TTL}
Re-index Cadence: {daily / weekly / on data change}

### Training (if applicable)
Method: {contrastive / Matryoshka / distillation / multi-task}
Loss: {MultipleNegativesRankingLoss / CachedMultipleNegativesRankingLoss / CosEntLoss}
Hard Negatives: {random / BM25 / model-based / iterative}
Data: {N} pairs | Negatives per Query: {N}
Epochs: {N} | Batch: {N} | LR: {lr} | Warmup: {N} steps

No preamble. No postamble. No explanations. No filler/hedging/transitions.

Completion Criteria

• [ ] Embedding model selected with MTEB scores, dimensionality, and language coverage justified.
• [ ] Chunking strategy chosen with size, overlap, and rationale matched to retrieval use case.
• [ ] Distance metric and normalization specified and consistent with training.
• [ ] Vector index type selected with parameters tuned for corpus size and recall target.
• [ ] Production indexing, caching, and re-indexing strategy defined.
• [ ] Training configuration documented if custom embeddings needed (loss, hard negatives, data).
• [ ] Quality evaluation pipeline with benchmark metrics and drift monitoring.
• [ ] Quantization strategy for scale (if needed) with quality impact assessment.

Embedding Model Selection Decision Tree

Determine language coverage
├── Monolingual English
│   ├── Latency-critical (100K vectors, good recall   |
| HNSW    | Slow      | O(log N)    | d×4× (1+2M)    | 0.98-0.99 | 100M, memory-constrained    |
| IVFPQ   | Medium    | Fast        | M×code_size    | 0.85-0.95 | >10M, balanced               |
| DiskANN | Very Slow | Medium      | Low (disk)     | 0.95-0.97 | >10M vectors, limited RAM    |

HNSW Parameters

M (connections per layer): 16-64
  - 16: lower memory (~2GB for 1M 768d), lower recall
  - 32: balanced (default, ~4GB for 1M 768d)
  - 64: high recall (~8GB for 1M 768d), slower build
  Memory ≈ d × 4 × (1 + M × 2) × N

efConstruction (build quality): 100-500
  - 100: fast build, adequate recall
  - 200: balanced (default)
  - 400+: high recall, slow build (5-10x slower)

efSearch (search breadth): 50-500
  - 50: fast search, may miss results
  - 100: balanced (default)
  - 200+: high recall, 2-4x slower search

IVF Parameters

nlist (clusters): sqrt(N) to 4×sqrt(N)
  - N=10K → nlist=100-400
  - N=100K → nlist=316-1265
  - N=1M → nlist=1000-4000
  Higher nlist = slower build, finer partitioning

nprobe (clusters searched): 10-100
  - Rule: recall ≈ 1 - (1 - nprobe/nlist)^k
  - Start with nprobe = nlist/10, tune up for recall
  - Search time = O(nprobe × nlist_size)

PQ Parameters

M (sub-quantizers): dimension / code_size
  - M=96 for 768d → 96 bytes/vec (32x compression)
  - M=48 for 768d → 48 bytes/vec (64x compression)
  - Higher M = better recall, more memory

code_size (bytes per sub-vector): 4-8
  - 4: 16 centroids per sub-quantizer
  - 8: 256 centroids (default, good quality)

Memory = M × code_size bytes per vector
  - Original 768d FP32: 3072 bytes/vec
  - PQ M=96, code_size=8: 96 bytes/vec (32x)
  - PQ M=48, code_size=8: 48 bytes/vec (64x)

Selection by Scale

100M: DiskANN or distributed IVF. 50-200ms latency.

Code Examples

Embedding Generation Pipeline

from sentence_transformers import SentenceTransformer
import numpy as np

def embed_pipeline(
    texts: list[str],
    model_name: str = "BAAI/bge-base-en-v1.5",
    batch_size: int = 64,
    normalize: bool = True,
    truncate_dim: int | None = None,
    device: str = "cpu",
) -> np.ndarray:
    """End-to-end embedding generation with batching and normalization."""
    model = SentenceTransformer(model_name, device=device)

    embeddings = model.encode(
        texts,
        batch_size=batch_size,
        normalize_embeddings=normalize,
        truncate_dim=truncate_dim,
        show_progress_bar=True,
    )
    return np.array(embeddings)

# Usage: 100K docs in batches
embeddings = embed_pipeline(
    texts=all_docs,
    model_name="BAAI/bge-base-en-v1.5",
    batch_size=128,
    normalize=True,
    truncate_dim=256,  # Matryoshka to 256d for faster search
)

Chunking Strategies

from typing import Iterator

def fixed_size_chunks(
    text: str,
    chunk_size: int = 256,
    overlap: int = 38,
    tokenizer=None,
) -> Iterator[str]:
    """Fixed-size token-based chunking with overlap."""
    tokens = tokenizer.encode(text)
    if len(tokens)  list[str]:
    """Semantic chunking based on embedding distance between sentences."""
    import spacy
    nlp = spacy.load("en_core_web_sm")
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents if sent.text.strip()]

    model = SentenceTransformer("all-MiniLM-L6-v2")
    embeds = model.encode(sentences, normalize_embeddings=True)

    splits = [0]
    for i in range(1, len(sentences)):
        sim = float(np.dot(embeds[i - 1], embeds[i]))
        if 1 - sim > threshold:  # cosine distance > threshold
            splits.append(i)
    splits.append(len(sentences))

    chunks = []
    for i in range(len(splits) - 1):
        chunk = " ".join(sentences[splits[i]:splits[i + 1]])
        chunks.append(chunk)
    return chunks

def recursive_chunks(text: str, max_tokens: int = 512) -> list[str]:
    """Recursive chunking: split on headers, then paragraphs, then sentences."""
    import re
    # Level 1: split on headers
    sections = re.split(r"\n#{1,3}\s+", text)
    chunks = []
    for section in sections:
        # Level 2: split on double newlines (paragraphs)
        paragraphs = re.split(r"\n\s*\n", section)
        chunk = ""
        for para in paragraphs:
            if len(chunk + para)  max_tokens:
                    # Level 3: split long paragraphs by sentences
                    for sent in re.split(r"(? list[dict]:
    """Hybrid search with configurable fusion strategy."""

    # Dense scores
    dense_scores = dense_embeddings @ query_embed

    # Sparse scores (BM25)
    tokenized_docs = [doc.split() for doc in documents]
    bm25 = BM25Okapi(tokenized_docs)
    bm25_scores = bm25.get_scores(query.split())

    # Normalize both to [0, 1]
    def normalize(scores):
        mn, mx = scores.min(), scores.max()
        return (scores - mn) / (mx - mn) if mx > mn else scores

    dense_norm = normalize(dense_scores)
    sparse_norm = normalize(bm25_scores.astype(float))

    # Fusion method 1: weighted linear
    combined = dense_weight * dense_norm + (1 - dense_weight) * sparse_norm

    # Fusion method 2: RRF (alternative)
    def rrf_fusion(dense_scores, bm25_scores, alpha=60):
        dense_ranks = np.argsort(np.argsort(-dense_scores))
        bm25_ranks = np.argsort(np.argsort(-bm25_scores))
        rrf = 1 / (alpha + dense_ranks) + 1 / (alpha + bm25_ranks)
        return rrf

    rrf_scores = rrf_fusion(dense_scores, bm25_scores, alpha)
    top_indices = np.argsort(-combined)[:top_k]

    return [
        {"doc": documents[i], "score": float(combined[i]), "dense": float(dense_scores[i]), "sparse": float(bm25_scores[i])}
        for i in top_indices
    ]

Anti-Patterns

Wrong Dimensionality for Downstream Task

Using full 3072d from text-embedding-3-large for a real-time search index with 50M vectors.

• Impact: 12GB/1M vectors → 600GB RAM. Search latency 50ms+.
• Fix: Truncate via Matryoshka to 256d or use PCA reduction. Quantize to int8.

Ignoring Normalization Consistency

Training with cosine distance but not normalizing at inference, or normalizing documents but not queries.

• Impact: Arbitrary score ranges, incorrect ranking, undefined behavior with inner product search.
• Fix: Always normalize_embeddings=True in encode(). Normalize stored vectors in index. Normalize every query.

Inconsistent Chunking Between Indexing and Retrieval

Indexing with chunksize=512 but retrieving with chunksize=128.

• Impact: Embeddings from different token contexts are incomparable.
• Fix: Use identical tokenizer, chunk_size, overlap, and truncation settings at index and query time.

Average Embeddings (AKA "Embedding Arithmetic" Fallacy)

Averaging embeddings of two sentences does not produce a meaningful "combined" embedding for retrieval.

• Impact: Semantic cancellation, degraded retrieval. A + B - C ≠ semantic vector arithmetic.
• Fix: Use the original text directly. If combination is needed, use late interaction (ColBERT) or multi-vector retrieval.

Embedding Every Token Without Truncation

Sending 10K token documents to a 512-token context model.

• Impact: Truncation destroys tail content silently. Retrieval misses relevant tail sections.
• Fix: Chunk documents first, embed each chunk separately. Use models with longer context (Jina 8K, E5-Mistral 32K).

Single Vector for Long Documents

Representing a 100-page PDF with one 768d embedding.

• Impact: Semantic overload, query touches any part → same embedding. Low retrieval precision.
• Fix: Chunk to 256-512 token passages, embed each, retrieve at passage level.

Training Without Hard Negatives

Using only (query, positive) pairs in contrastive learning.

• Impact: Model learns trivial separation, poor discrimination between similar but irrelevant results.
• Fix: Include hard negatives curated via BM25 retrieval or model-in-the-loop mining.

Using Unnormalized Embeddings with Inner Product

Building FAISS index with IndexFlatIP on unnormalized vectors.

• Impact: Inner product ≠ cosine similarity. Magnitude dominates over direction. Biased toward long documents.
• Fix: Always L2-normalize before adding to IP index, or use IndexFlatL2 with cosine distance.

Production Considerations

Embedding Update Strategies

Batch rebuild (zero-downtime):
  1. Embed all documents with new model
  2. Build new index
  3. Swap index reference atomically
  4. Keep old index for in-flight queries
  Good for: model version upgrades, schema changes
  Bad for: high-velocity data

Incremental update (HNSW/IVF):
  - HNSW supports online insertion (index.add())
  - IVF supports online insertion AFTER training
  - PQ does NOT support online insertion (need batch rebuild)
  - Deletion: not supported natively — use tombstone filter
  Strategy:
    1. Maintain a deletion bitmask
    2. Filter out tombstoned IDs after search
    3. Periodically rebuild to reclaim space (>20% tombstones)

Two-index pattern (hot/warm):
  - Hot index: recent N days, updated hourly, small, fast
  - Warm index: everything older, rebuilt nightly
  - Query both, merge results with RRF
  - Reduces rebuild cost by 10x for append-heavy workloads

Re-indexing

Trigger events:
├── Model version upgrade (new MTEB SOTA)
├── Embedding drift detected (>10% mean shift from reference distribution)
├── Data distribution shift (new domains, languages, document types)
├── Chunking strategy change (size, overlap, method)
├── Index fragmentation (>20% tombstones)
└── Scheduled (monthly for stable corpora, weekly for dynamic)

Process:
  1. Snapshot current documents with metadata
  2. Generate new embeddings in parallel batches
  3. Build new index on separate infrastructure
  4. Validate: sample 1K queries, compare recall@10 against old index
  5. Canary deploy: route 5% traffic to new index, monitor latency & recall
  6. Full swap when validation passes
  7. Decomission old index after TTL (allow in-flight queries to drain)

Cost Optimization

Storage cost (1M vectors, 768d):
  ┌──────────────────────┬───────────┬───────────┐
  │ Storage              │ Memory    │ Cost/mo   │
  ├──────────────────────┼───────────┼───────────┤
  │ FP32 + HNSW M=32     │ 5400 MB   │ $135      │
  │ FP16 + HNSW M=32     │ 2700 MB   │ $68       │
  │ int8 + HNSW M=32     │ 1350 MB   │ $34       │
  │ FP32 + IVF nlist=2K  │ 3100 MB   │ $78       │
  │ PQ M=96, code=8      │ 96 MB     │ $3        │
  │ Binary               │ 96 MB     │ $3        │
  └──────────────────────┴───────────┴───────────┘

API cost (1M documents, ~200 tokens each):
  ┌─────────────────────────────┬──────────┬──────────┐
  │ Model                       │ Cost     │ Batch    │
  ├─────────────────────────────┼──────────┼──────────┤
  │ text-embedding-3-small      │ $4       │ 2048     │
  │ text-embedding-3-large      │ $26      │ 2048     │
  │ Cohere embed-english-v3     │ $20      │ 96       │
  └─────────────────────────────┴──────────┴──────────┘

Reduce costs:
  ├── Deduplicate texts before embedding (cache hits)
  ├── Use smaller dims for initial retrieval, re-rank with full
  ├── Batch API calls to max batch size (reduce HTTP ove

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [j4flmao](https://github.com/j4flmao)
- **Source:** [j4flmao/agent-skills](https://github.com/j4flmao/agent-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.