Ai Rag Llm Engineer

AI/RAG/LLM Systems Engineer

Comprehensive guidance for designing and building AI systems powered by Large Language Models. Based on authoritative sources: AI-Powered Search (Grainger, Turnbull, Irwin), Super Study Guide: Transformers and LLMs (Amidi), Hands-On Large Language Models (Alammar, Grootendorst), and current research.

Supporting Documents:

• [transformers.md](transformers.md) - Transformer architecture deep dive
• [rag-systems.md](rag-systems.md) - RAG pipeline design and optimization
• [embeddings-vectors.md](embeddings-vectors.md) - Embeddings and vector search
• [fine-tuning.md](fine-tuning.md) - Fine-tuning and alignment techniques
• [agents.md](agents.md) - LLM agents and tool use
• [evaluation.md](evaluation.md) - Evaluation metrics and benchmarks

Core Concepts

The LLM Landscape

Large Language Models are deep neural networks trained on massive text corpora that can recognize, translate, summarize, predict, and generate text. Modern LLMs are based on the Transformer architecture (Vaswani et al., 2017), which applies "attention" to learn contextual representations of language.

Key Model Categories:

| Type | Architecture | Strengths | Examples | |------|-------------|-----------|----------| | Autoencoding | Encoder-only | Understanding, classification | BERT, RoBERTa | | Autoregressive | Decoder-only | Text generation | GPT-4, LLaMA, Claude | | Seq2Seq | Encoder-Decoder | Translation, summarization | T5, BART |

The AI-Powered Search Spectrum

Modern search operates on a personalization spectrum from traditional keyword search to fully personalized recommendations:

Keyword Search ←→ Personalized Search ←→ Semantic Search ←→ Recommendations
     ↓                    ↓                    ↓                   ↓
  Term matching    User signals +        Knowledge graphs    Collaborative
                   keyword search        + keyword search      filtering

Optimal search relevancy lies at the intersection of:

• Personalized Search: Traditional keyword search + collaborative recommendations
• Semantic Search: Traditional keyword search + knowledge graphs
• Multimodal Recommendations: Collaborative recommendations + knowledge graphs

Transformer Architecture

Self-Attention Mechanism

The core innovation enabling LLMs is self-attention, which allows each token to attend to all other tokens in a sequence, capturing long-range dependencies.

Key-Query-Value Computation:

For each token position:
1. Compute Query (Q), Key (K), Value (V) vectors from embeddings
2. Calculate attention scores: Attention(Q,K,V) = softmax(QK^T / √d_k) × V
3. Weighted sum produces context-aware representation

Properties of self-attention:

• ✅ Captures long-range dependencies (unlike RNNs)
• ✅ Parallelizable (unlike sequential processing)
• ✅ Position-agnostic (requires positional encoding)
• ❌ Quadratic complexity O(n²) with sequence length

Multi-Head Attention

Multiple attention heads allow the model to focus on different aspects simultaneously:

• Head 1: Syntactic relationships
• Head 2: Semantic similarity
• Head 3: Coreference resolution
• etc.

Positional Encoding

Since self-attention is position-agnostic, positional information is added:

• Sinusoidal: Fixed mathematical function (original Transformer)
• Learned: Trainable embeddings per position
• Rotary (RoPE): Rotation-based encoding (LLaMA, modern models)
• ALiBi: Attention with Linear Biases (no explicit encoding)

> 💡 Deep Dive Available: For detailed transformer internals including layer normalization, feed-forward networks, and architectural variants, see [transformers.md](transformers.md)

Embeddings and Vector Search

Word Embeddings

Embeddings are dense vector representations that capture semantic meaning:

Traditional (sparse):  [0, 0, 1, 0, 0, ..., 0]  # One-hot, high-dimensional
Embedding (dense):     [0.23, -0.45, 0.12, ...]  # Learned, lower-dimensional

Key insight: Similar concepts have similar vectors. Vector operations reveal relationships:

king - man + woman ≈ queen

Embedding Models

| Model | Dimensions | Use Case | |-------|------------|----------| | Word2Vec | 100-300 | Word-level similarity | | Sentence-BERT | 384-768 | Sentence similarity | | OpenAI text-embedding-3 | 256-3072 | General purpose, Matryoshka | | Cohere embed-v3 | 1024 | Multilingual | | BGE/E5 | 384-1024 | Open-source, fine-tunable | | ColBERT | 128 | Late interaction, high quality | | Jina-embeddings-v3 | 1024 | Long context, late chunking |

Vector Similarity

Cosine Similarity (most common):

similarity(A, B) = (A · B) / (||A|| × ||B||)

• Range: [-1, 1] (1 = identical direction)
• Normalized, ignores magnitude

Dot Product:

similarity(A, B) = A · B

• Faster computation
• Magnitude-sensitive

Euclidean Distance:

distance(A, B) = √(Σ(Aᵢ - Bᵢ)²)

• Lower = more similar

Vector Databases and ANN Search

For production systems, exact nearest neighbor search is too slow. Use Approximate Nearest Neighbor (ANN) algorithms:

HNSW (Hierarchical Navigable Small World):

• Graph-based index with hierarchical layers
• Excellent recall/speed tradeoff
• Memory-intensive
• Used by: Pinecone, Weaviate, Qdrant

IVF (Inverted File Index):

• Clusters vectors, searches relevant clusters
• Good for very large datasets
• Trade-off: cluster count vs accuracy
• Used by: Faiss, Milvus

Product Quantization (PQ):

• Compresses vectors for memory efficiency
• Some accuracy loss
• Often combined with IVF (IVF-PQ)

> 💡 Deep Dive Available: For vector database selection, indexing strategies, and optimization, see [embeddings-vectors.md](embeddings-vectors.md)

Retrieval-Augmented Generation (RAG)

RAG combines retrieval with generation to ground LLM outputs in factual, up-to-date information.

Why RAG?

LLMs have inherent limitations:

• Knowledge cutoff: Training data has a cutoff date
• Hallucination: May generate plausible but false information
• No source attribution: Can't cite where information came from
• Static knowledge: Can't access real-time or proprietary data

RAG addresses these by retrieving relevant context before generation.

RAG Pipeline Architecture

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Query     │───▶│  Retriever  │───▶│  Augment    │───▶│  Generator  │
│             │    │             │    │   Prompt    │    │    (LLM)    │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
                          │
                          ▼
                   ┌─────────────┐
                   │   Vector    │
                   │  Database   │
                   └─────────────┘

Document Processing Pipeline

1. Document Ingestion

Raw Documents → Text Extraction → Cleaning → Chunking → Embedding → Indexing

2. Chunking Strategies

| Strategy | Description | Best For | |----------|-------------|----------| | Fixed-size | Split by character/token count | Simple, predictable | | Sentence-based | Split at sentence boundaries | Preserves grammar | | Paragraph-based | Split at paragraph breaks | Preserves context | | Semantic | Split by topic/meaning changes | Best quality, slower | | Recursive | Hierarchical splitting | Long documents |

Chunking Best Practices:

• Chunk size: 256-512 tokens typical; balance context vs specificity
• Overlap: 10-20% overlap prevents information loss at boundaries
• Metadata: Preserve source, section headers, page numbers

3. Advanced Chunking Techniques

| Technique | Description | Best For | |-----------|-------------|----------| | Meta-Chunking | Preserve logical coherence (causal, transitional) | Complex documents | | Late Chunking | Embed full doc, chunk after transformer | Context preservation | | Contextual Retrieval | Prepend LLM-generated context to chunks | High-value collections |

Advanced RAG Architectures

| Architecture | Description | Use Case | |--------------|-------------|----------| | GraphRAG | Knowledge graph + vector retrieval | Multi-hop reasoning | | CRAG | Self-correcting retrieval with web fallback | Reducing hallucinations | | Self-RAG | LLM decides when/what to retrieve | Adaptive retrieval | | Agentic RAG | Autonomous agents manage retrieval | Complex queries |

Retrieval Strategies

Basic Retrieval:

1. Embed query using same model as documents
2. Find top-K nearest neighbors
3. Return documents above similarity threshold

Advanced Retrieval:

| Technique | Description | |-----------|-------------| | Hybrid Search | Combine dense (vector) + sparse (BM25) retrieval | | Query Expansion | Generate related queries, retrieve for all | | HyDE | Generate hypothetical answer, use as query | | Multi-Query | Break complex query into sub-queries | | Parent Document | Retrieve child chunks, return parent context | | Self-Query | LLM extracts metadata filters from query |

Re-ranking: After initial retrieval, re-rank results using:

• Cross-encoder models (more accurate, slower)
• LLM-based relevance scoring
• Diversity-aware re-ranking (MMR)

Prompt Augmentation

Structure the prompt to effectively use retrieved context:

System: You are a helpful assistant. Answer based on the provided context.
If the context doesn't contain the answer, say "I don't know."

Context:
{retrieved_documents}

User: {query}

Best Practices:

• Place context before the question
• Instruct model to cite sources
• Handle "no relevant context" cases
• Limit context to fit in context window

> 💡 Deep Dive Available: For advanced RAG patterns, evaluation, and production considerations, see [rag-systems.md](rag-systems.md)

Prompt Engineering

Prompt Structure

Effective prompts have clear structure:

[System Message]     - Role, constraints, output format
[Context/Examples]   - Few-shot examples, retrieved documents
[User Input]         - The actual query/task
[Output Format]      - Expected structure of response

Prompting Techniques

Zero-Shot: Direct instruction without examples

Classify the sentiment of this review as positive, negative, or neutral:
"The product arrived damaged but customer service was helpful."

Few-Shot: Provide examples to guide behavior

Classify sentiment:
"Great product!" → positive
"Terrible experience" → negative
"It was okay" → neutral
"The product arrived damaged but customer service was helpful." →

Chain-of-Thought (CoT): Encourage step-by-step reasoning

Let's solve this step by step:
1. First, identify...
2. Then, consider...
3. Finally, conclude...

ReAct (Reasoning + Acting): Interleave reasoning with actions

Thought: I need to find the current population of Tokyo
Action: search("Tokyo population 2024")
Observation: Tokyo's population is approximately 14 million
Thought: Now I can answer the question
Answer: Tokyo has approximately 14 million people

Prompt Engineering Best Practices

1. Be specific: Vague prompts yield vague results
2. Provide context: Include relevant background
3. Specify format: JSON, markdown, bullet points
4. Use delimiters: Clearly separate sections
5. Include constraints: Length limits, style requirements
6. Iterate: Test and refine prompts systematically

Fine-Tuning and Alignment

When to Fine-Tune

| Approach | When to Use | |----------|-------------| | Prompt Engineering | Quick iteration, no training data | | RAG | Need current/proprietary knowledge | | Fine-Tuning | Specific style, format, or domain expertise | | RLHF/DPO | Align with human preferences |

Fine-Tuning Methods

Full Fine-Tuning:

• Update all model parameters
• Requires significant compute
• Risk of catastrophic forgetting

Parameter-Efficient Fine-Tuning (PEFT):

| Method | Description | Parameters Updated | |--------|-------------|-------------------| | LoRA | Low-rank adaptation matrices | ~0.1-1% | | QLoRA | LoRA + quantization | ~0.1-1% | | Prefix Tuning | Learnable prefix tokens | ~0.1% | | Adapters | Small bottleneck layers | ~1-5% |

Alignment Techniques

Supervised Fine-Tuning (SFT):

• Train on high-quality instruction-response pairs
• Foundation for instruction-following

RLHF (Reinforcement Learning from Human Feedback):

1. Collect human preference data
2. Train reward model
3. Optimize policy with PPO

DPO (Direct Preference Optimization):

• Simpler alternative to RLHF
• Directly optimizes on preference pairs
• No separate reward model needed

> 💡 Deep Dive Available: For fine-tuning recipes, data preparation, and alignment details, see [fine-tuning.md](fine-tuning.md)

LLM Agents

Agent Architecture

LLM agents extend LLMs with:

• Memory: Short-term (context) and long-term (vector store)
• Tools: External capabilities (search, code execution, APIs)
• Planning: Task decomposition and execution

┌─────────────────────────────────────────────────────────┐
│                      LLM Agent                          │
├─────────────────────────────────────────────────────────┤
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐   │
│  │ Memory  │  │Planning │  │  Tools  │  │  LLM    │   │
│  │         │  │         │  │         │  │ (Brain) │   │
│  └─────────┘  └─────────┘  └─────────┘  └─────────┘   │
└─────────────────────────────────────────────────────────┘

Tool Use / Function Calling

Modern LLMs can call external functions:

{
  "name": "search_database",
  "description": "Search the product database",
  "parameters": {
    "query": {"type": "string", "description": "Search query"},
    "limit": {"type": "integer", "description": "Max results"}
  }
}

The LLM decides when to call tools and how to use results.

Agent Patterns

ReAct Pattern:

Thought → Action → Observation → Thought → ... → Answer

Plan-and-Execute:

1. Create plan with steps
2. Execute each step
3. Revise plan if needed
4. Return final result

Multi-Agent Systems:

• Coordinator agent delegates to specialists
• Agents can critique each other's work
• Enables complex workflows

> 💡 Deep Dive Available: For agent frameworks, memory systems, and production patterns, see [agents.md](agents.md)

Evaluation

LLM Evaluation Metrics

Perplexity: How confident is the model?

• Lower = more confident (but not necessarily correct)

Task-Specific Metrics:

| Task | Metrics | |------|---------| | Translation | BLEU, METEOR, chrF | | Summarization | ROUGE-1, ROUGE-2, ROUGE-L | | QA | Exact Match (EM), F1 | | Generation | Human evaluation, LLM-as-judge |

RAG Evaluation

Retrieval Quality:

• Recall@K: % of relevant docs in top K
• MRR: Mean Reciprocal Rank
• NDCG: Normalized Discounted Cumulative Gain

Generation Quality:

• Faithfulness: Does answer match retrieved context?
• Relevance: Does answer address the question?
• Groundedness: Is answer supported by sources?

Hallucination Detection

Hallucinations are outputs that don't align with facts or context:

• Factual hallucination: Contradicts known facts
• Faithfulness hallucination: Contradicts provided context
• Prompt misalignment: Doesn't follow instructions

Mitigation Strategies:

1. Use RAG to ground responses
2. Instruct model to say "I don't know"
3. Implement fact-checking pipelines
4. Use lower temperature for factual tasks

> 💡 Deep Dive Available: For evaluation frameworks, benchmarks, and testing strategies, see [evaluation.md](evaluation.md)

Design Checklist

Requirements Analysis

• [ ] Define use case (search, QA, generation, agents)
• [ ] Identify data

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: SpaceFrontiers
• Source: SpaceFrontiers/ai-overmind
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.