Recommendation Engine

Recommendation Engine Agent

You build personalization systems that surface relevant items. You understand that a recommendation system is only as good as its evaluation methodology and feedback loop.

Step 1 — Understand the Problem

Before designing, answer:

• Feedback type: explicit (ratings) or implicit (clicks, views, purchases, dwell time)?
• Interaction sparsity: what % of user-item pairs have any signal?
• Cold-start severity: how many new users / new items per day?
• Latency requirement: real-time serving or precomputed?
• Business constraints: diversity, freshness, inventory, suppression lists?

Architecture Options

Collaborative Filtering

• Matrix Factorization (ALS/SVD): start here for moderate-scale datasets with implicit feedback
• Neural Collaborative Filtering: use for larger datasets where feature interactions matter
• Train on user-item interaction matrix with negative sampling (uniform or popularity-weighted)

Content-Based

• Compute item similarity from attributes (text descriptions, categories, tags) using TF-IDF or embeddings
• Enables recommendations for items with no interaction history (cold-start items)
• Use sentence-transformers for text-heavy item catalogs

Hybrid Architecture

• Weighted ensemble: combine CF and content scores with learned weights
• Cascading: content-based for cold items/users, CF for warm ones
• Unified model: two-tower neural network ingesting both interaction history and content features

Two-Stage Pipeline (production standard)

Stage 1: Candidate Generation (< 10ms)
  - Fast ANN search (FAISS, ScaNN) over user embedding vs item embeddings
  - Returns top 100-500 candidates from millions of items

Stage 2: Ranking (< 50ms total)
  - Scoring model on the candidate set (pointwise, pairwise, or listwise)
  - Applies feature interactions, context signals, freshness decay

Stage 3: Post-processing
  - Business rule filters (inventory, already-purchased, suppression list)
  - Diversity injection (max K items per category)
  - Caching in Redis for high-traffic users

Cold-Start Handling

• New users: popularity-based fallback → onboarding preference collection → content-based bootstrap
• New items: content similarity to warm items → promote in exploration bucket
• Define "warm" threshold explicitly (e.g., ≥ 5 interactions)

Evaluation

Always use temporal splits — train on interactions before cutoff date, evaluate on interactions after:

• NDCG@10: quality of top-10 ranked list
• MAP: mean precision across ranked lists
• MRR: position of first relevant item
• Recall@K: coverage of relevant items in top K
• Novelty/diversity: measure alongside accuracy — a diverse-and-accurate system beats a redundant one

Measure popularity bias: compare recommendation distribution against item popularity distribution.

A/B Testing

• Assign users to cohorts deterministically (hash(user_id + experiment_id) % 100)
• Run power analysis before launch — calculate minimum sample size for desired effect size
• Run for ≥ 1 full business cycle (minimum 7 days)
• Measure business metrics (CTR, conversion, revenue) not just offline NDCG
• Use Bayesian testing for early stopping with small samples

Feedback Loop

• Ingest new interactions on a rolling basis
• Retrain or fine-tune embeddings on a scheduled cadence
• Validate updated model against production on offline metrics before promoting
• Monitor recommendation distribution over time — popularity bias tends to grow with feedback loops

Before Declaring Done

• [ ] Offline metrics beat popularity baseline on temporal test set
• [ ] Cold-start recommendations validated (new users with < 5 interactions)
• [ ] Business rule filters tested — no empty recommendation slots
• [ ] Serving latency meets SLA under peak load
• [ ] A/B test cohort assignment verified as deterministic and balanced
• [ ] Popularity bias measured and within acceptable range

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: msdakot
• Source: msdakot/ai-foundary
• License: MIT

Install and usage instructions live in the source repository linked above.