Llm Playground

LLM Playground

A comprehensive playground for experimenting with Model Context Protocol (MCP) servers, featuring web-based interfaces for testing multiple AI providers and MCP connectors.

⚠️ DISCLAIMER

This code is for reference and demonstration purposes only. Please read the following important disclaimers:

🔍 Code Accuracy & Reliability

• AI-Generated Content: This codebase is AI-generated and may contain inaccuracies, bugs, or incomplete implementations
• Reference Only: This code is provided as a learning resource and reference implementation, not for production use
• No Guarantees: The code may not work as expected and should be thoroughly tested before any use
• Educational Purpose: This is intended for educational and demonstration purposes only

💰 Token Calculator & Cost Estimation

• Indicative Data: All token counting, cost estimation, and pricing information provided is indicative only
• May Not Be Accurate: Token counts, costs, and pricing data may not reflect current or accurate values
• Provider Changes: AI providers frequently update their pricing, models, and tokenization methods
• Verify Independently: Always verify token counts and costs with official provider documentation
• No Financial Advice: Cost estimates should not be used for financial planning or budgeting

🎯 Demo Purpose Only

• Not Production Ready: This code is not intended for production environments
• Learning Tool: Use this as a learning tool to understand MCP concepts and implementations
• Experimental: This is experimental code that may break or change unexpectedly
• No Support: No guarantees of support, maintenance, or updates

🔒 Security & Privacy

• API Keys: Never use real API keys in this demo environment
• Test Data: Use only test data and dummy credentials
• No Sensitive Information: Do not process or store sensitive information with this code

By using this code, you acknowledge that you understand these limitations and will use it responsibly for educational purposes only.

---

🎯 Purpose & Overview

This repository provides a complete environment for working with MCP servers, designed to help developers and researchers:

• Learn MCP: Understand how Model Context Protocol works through hands-on examples
• Test AI Providers: Experiment with multiple AI providers (OpenAI, Anthropic, Google, Ollama) in a unified interface
• Integrate MCP Connectors: Connect to GitHub and PostgreSQL MCP servers for real-world data
• Build AI Applications: Create AI-powered applications that combine multiple data sources and AI models

🏗️ Architecture Overview

This repository includes three distinct playground implementations, each with different architectural approaches:

Basic Playground - Simple & Lightweight

• LLM Integration: Direct API calls using requests library
• MCP Integration: Custom fastmcp client implementation
• Async Support: Limited async support with manual asyncio.run()
• Type Safety: Basic type hints with dataclasses
• Use Case: Quick testing and learning MCP fundamentals

Extended Playground - Production-Ready Features

• LLM Integration: Direct API calls using requests library
• MCP Integration: Custom fastmcp client implementation
• Async Support: Limited async support with manual asyncio.run()
• Type Safety: Basic type hints with dataclasses
• Advanced Features: Session management, comprehensive logging, token calculator
• Use Case: Full-featured development with advanced capabilities

LangChain Playground - Modern & Scalable ⭐ Recommended

• LLM Integration: LangChain's standardized LLM interfaces
• MCP Integration: LangChain's MCP adapters for seamless integration
• Async Support: Full async/await support throughout
• Type Safety: Enhanced type hints with proper interfaces
• Architecture: Modular design following SOLID principles
• Performance: True async operations with better resource management
• Extensibility: Easy to add new providers and connectors
• Use Case: Production-ready applications with modern best practices

Key Architectural Differences

| Feature | Basic | Extended | LangChain | |---------|-------|----------|-----------| | LLM Integration | Direct API calls | Direct API calls | LangChain interfaces | | MCP Integration | Custom fastmcp | Custom fastmcp | LangChain MCP adapters | | Async Support | Limited | Limited | Full async/await | | Type Safety | Basic | Basic | Enhanced | | Modularity | Simple | Good | Excellent | | Testability | Basic | Good | Excellent | | Performance | Basic | Good | Optimized | | Maintainability | Simple | Good | Excellent | | Extensibility | Limited | Good | Excellent |

🧠 Context Strategies & MCP Integration

The playgrounds implement sophisticated Model Context Protocol (MCP) context management strategies that enable dynamic, real-time context updates when knowledge base content changes. This section documents the current implementation and roadmap for future enhancements.

Current Context Strategy Features

🔄 Real-Time Context Synchronization

• Immediate Context Propagation: Changes to knowledge bases are immediately reflected in connected MCP clients
• Session Persistence: MCP maintains context state across distributed systems during service interruptions
• Priority-Based Queuing: Critical context updates receive priority processing to ensure important changes aren't delayed

📊 Token-Efficient Context Management

• Intelligent Context Selection: Provides precisely the context needed for specific queries rather than overwhelming models
• Dynamic Context Sizing: Automatically adjusts context window sizes based on query complexity and available information
• Semantic Context Filtering: Uses semantic understanding to include only relevant context, reducing token waste
• Context Compression: Preserves meaning while reducing token count through intelligent summarization

🏗️ MCP-Specific Architecture

• Stateless Request Processing: Separates context operations from model inference for independent scaling
• Elastic Resource Allocation: Automatic provisioning of additional context servers during high-update periods
• Context-Aware Load Balancing: Intelligent routing ensures context updates are processed by optimized servers

🔧 Implementation Details

Prompt Template System

// Research-focused templates with MCP context integration
{
    id: "research_papers",
    name: "Research Paper Analysis", 
    text: `Analyze GitHub issues from {owner_repo} and match them with relevant AI research papers. Provide:
1. Key requirements extracted from GitHub issues
2. Relevant research papers that address these requirements  
3. Implementation recommendations based on research findings
4. Gap analysis and potential research opportunities`
}

Smart Prompt Optimization

# Token budget allocation with MCP context
context_budget_total = int(prompt_budget * 0.45)
issues_budget = max(150, context_budget_total // 2)
papers_budget = max(150, context_budget_total - issues_budget)

# Real-time context fetching
gh_result = await gh_connector.fetch_issues_and_comments(limit_issues=3, limit_comments=5)
pg_result = await pg_connector.fetch_research_papers(limit_rows=8)

Context Window Management

# Dynamic context sizing based on provider capabilities
reserve_reply = int(provider_cw_tokens * 0.25)  # 25% for response
reserve_system = 800  # Fixed system prompt reserve
available_context = int(provider_cw_tokens * max_context_usage)  # 80% max usage

🎯 Current Context Strategies

1. Dual-Context Integration

• GitHub Issues Context: Real-time project requirements and specifications
• Research Papers Context: AI research database with relevant papers
• Combined Analysis: Intelligent matching of requirements with research insights

2. Template-Based Context Injection

• 13 Specialized Templates: Covering research analysis, implementation guides, and literature reviews
• Dynamic Placeholder Substitution: {owner_repo} placeholders replaced with actual repository data
• Context-Aware Templates: Designed specifically for MCP-enhanced workflows

3. Real-Time Context Updates

• Fresh Data Fetching: Each request fetches fresh data from MCP servers
• Error Resilience: Graceful degradation when MCP servers are unavailable
• Parallel Processing: GitHub and PostgreSQL queries run simultaneously

4. Token Optimization

• Semantic Compression: Uses LLM to summarize context while preserving key facts
• Budget Enforcement: Prevents context overflow with intelligent trimming
• Provider Adaptation: Adjusts to different LLM providers' context windows

🚀 Roadmap: Advanced Context Strategies

Phase 1: Enhanced Context Persistence (Q2 2024)

• Long-Term Context Memory: Cross-session context persistence and evolution tracking
• Context Versioning: Track knowledge base changes and their impact on context
• Context Evolution Analytics: Monitor how context changes over time

Phase 2: Adaptive Context Selection (Q3 2024)

# Proposed: Adaptive Context Selection
class AdaptiveContextSelector:
    def select_context_strategy(self, query_type: str, available_tokens: int) -> ContextStrategy:
        if query_type == "research_analysis":
            return ResearchFocusedStrategy(available_tokens * 0.6)
        elif query_type == "implementation_guide":
            return ImplementationFocusedStrategy(available_tokens * 0.4)
        elif query_type == "literature_review":
            return LiteratureReviewStrategy(available_tokens * 0.7)

Phase 3: Real-Time Context Synchronization (Q4 2024)

# Proposed: WebSocket-based context updates
class RealTimeContextManager:
    async def subscribe_to_context_updates(self, repo: str):
        """Subscribe to real-time GitHub issue updates"""
        async for update in github_webhook_stream:
            await self.update_context_cache(update)
    
    async def broadcast_context_update(self, context_id: str, update: dict):
        """Notify all subscribed agents of context changes"""

Phase 4: Multi-Agent Context Coordination (Q1 2025)

# Proposed: Shared context store for multi-agent systems
class SharedContextStore:
    def __init__(self):
        self.context_cache = {}
        self.agent_subscriptions = {}
    
    async def coordinate_context_access(self, agent_id: str, context_request: dict):
        """Coordinate context access between multiple agents"""
    
    async def resolve_context_conflicts(self, conflicting_updates: list):
        """Resolve conflicts when multiple agents update context simultaneously"""

Phase 5: Enterprise Context Management (Q2 2025)

• Context Security Framework: OAuth 2.1 authorization with granular permissions
• Context Governance: Immutable audit trails and compliance tracking
• High-Throughput Processing: 50,000+ requests/second with 99.95% uptime
• Context Replication: Automated failover and context replication across regions

Phase 6: Advanced RAG Strategies & Hallucination Reduction (Q3 2025)

# Proposed: Advanced RAG Implementation with Hallucination Prevention
class AdvancedRAGManager:
    def __init__(self):
        self.vector_store = HybridVectorStore()
        self.retrieval_optimizer = RetrievalOptimizer()
        self.hallucination_detector = HallucinationDetector()
        self.context_verifier = ContextVerifier()
        self.knowledge_graph = KnowledgeGraphBuilder()
    
    async def enhanced_retrieval(self, query: str, context: dict) -> RetrievalResult:
        """Multi-modal retrieval with semantic and keyword search"""
        # Hybrid search combining dense and sparse retrievers
        dense_results = await self.vector_store.semantic_search(query, top_k=10)
        sparse_results = await self.vector_store.keyword_search(query, top_k=10)
        
        # Rerank using cross-encoder for better relevance
        reranked_results = await self.retrieval_optimizer.rerank(
            query, dense_results + sparse_results
        )
        
        return await self.context_verifier.validate_relevance(reranked_results, query)
    
    async def hallucination_prevention(self, response: str, context: dict) -> ValidationResult:
        """Multi-layer hallucination detection and prevention"""
        # Fact-checking against retrieved context
        fact_check = await self.hallucination_detector.fact_check(response, context)
        
        # Source attribution verification
        attribution_check = await self.context_verifier.verify_attributions(response, context)
        
        # Confidence scoring with uncertainty quantification
        confidence_score = await self.hallucination_detector.calculate_confidence(response)
        
        return ValidationResult(
            is_valid=fact_check.is_valid and attribution_check.is_valid,
            confidence=confidence_score,
            corrections=fact_check.corrections
        )
    
    async def knowledge_graph_enhancement(self, context: dict) -> KnowledgeGraph:
        """Build and maintain knowledge graph for better context understanding"""
        entities = await self.knowledge_graph.extract_entities(context)
        relationships = await self.knowledge_graph.extract_relationships(entities)
        
        return await self.knowledge_graph.build_graph(entities, relationships)

Advanced RAG Features:

• Hybrid Retrieval: Combines dense (semantic) and sparse (keyword) search for comprehensive results
• Multi-Modal RAG: Supports text, code, images, and structured data retrieval
• Dynamic Reranking: Uses cross-encoders to improve retrieval relevance
• Context-Aware Retrieval: Adapts retrieval strategy based on query type and domain
• Real-Time Knowledge Updates: Incremental updates to vector store and knowledge graph
• Query Expansion: Intelligent query reformulation for better retrieval
• Source Diversity: Ensures diverse source selection to reduce bias

Hallucination Prevention Strategies:

• Fact-Checking Pipeline: Automated verification against retrieved context
• Source Attribution: Mandatory citation and source linking for all claims
• Confidence Scoring: Uncertainty quantification for response reliability
• Contradiction Detection: Identifies and resolves conflicting information
• Context Consistency: Ensures response consistency with provided context
• Human-in-the-Loop: Fallback mechanisms for high-stakes decisions

Phase 7: AI Agents & Multi-Agent Systems (Q4 2025)

# Proposed: Multi-Agent System with Specialized AI Agents
class MultiAgentSystem:
    def __init__(self):
        self.agent_orchestrator = AgentOrchestrator()
        self.shared_memory = SharedMemoryStore()
        self.task_decomposer = TaskDecomposer()
        self.agent_registry = AgentRegistry()
        
    async def coordinate_agents(self, task: str, context: dict) -> AgentResponse:
        """Coordinate multiple specialized agents for complex tasks"""
        # Decompose complex task into subtasks
        subtasks = await self.task_decomposer.decompose(task)
        
        # Assign agents based on expertise
        agent_assignments = await self.agent_orchestrator.assign_agents(subtasks)
        
        # Execute tasks in parallel with shared memory
        results = await self.agent_orchestrator.execute_parallel(agent_assignments)
        
        # Synthesize results and resolve conflicts
        final_response = awa

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [amitpuri](https://github.com/amitpuri)
- **Source:** [amitpuri/llm-playground](https://github.com/amitpuri/llm-playground)
- **License:** MIT
- **Homepage:** https://openagi.news

Install and usage instructions live in the source repository linked above.