Langchainrust

langchainrust

[](https://www.rust-lang.org/) [](LICENSE) [](https://crates.io/crates/langchainrust) [](https://docs.rs/langchainrust) [](https://github.com/atliliw/langchainrust/actions/workflows/ci.yml) [](https://crates.io/crates/langchainrust)

A LangChain-inspired Rust framework for building LLM applications.

What it solves: Build Agents, RAG, BM25 keyword search, Hybrid retrieval, LangGraph workflows, MCP tools, A2A agent-to-agent protocols, Guardrails, multi-agent Handoffs — all in pure Rust.

---

Design Principles

The framework is engineered around a few hard rules that come out of its own design reviews. These are what make it feel different from hand-rolled LLM glue code:

| Principle | What it means in practice | |-----------|---------------------------| | Explicit over silent | No silent degradation. If an API promises X, it either delivers X or fails loudly. Batch-embedding alignment returns explicit errors instead of empty/shifted vectors; the keyword fallback for EmbeddingMatcher was removed; routing failures surface as errors, not swallowed Err(_) => {}. | | Consumed or deleted abstractions | Every trait either has real implementors or is removed. BaseChatMemory is now implemented by all four memories, all retrievers implement RetrieverTrait, and PairwiseJudge plugs into the unified Evaluator pipeline — so users write against one abstraction, not five ad-hoc names. | | Structured output over text parsing | Models that support tool_calls go through structured output (JSON schema / function calling). Regex-parsing model output is the last resort, not the default — it is the most common source of silent fragility. | | Production hardening first | Tool execution has timeouts, LLM calls retry with exponential backoff, agent loops are capped (max_iterations clamped to [1, 100]), parallel actions are concurrency-limited, and CancellationToken propagates through every runnable. Eliminate the deterministic failure paths before adding features. | | Type system enforces safety | Security properties live in types, not comments. Guardrails split InputGuardrailResult / OutputGuardrailResult so "Modify only applies to output" is enforced at compile time. | | Composition first | Everything can be piped. Since v0.16.0, prompts, memory, native providers, parsers and RAG are all Runnable — prompt.pipe(llm).pipe(parser) compiles and runs. | | Honest implementations | No fake backends. Empty-shell sandboxes (Wasm/E2B) were deleted; unsupported operations (e.g. Pinecone fetch-by-ID) return explicit StorageErrors instead of pretending to work. |

---

Core Features

LLM & Providers

| Component | Description | |-----------|-------------| | Unified LLM access | 11 providers behind one BaseChatModel trait: OpenAI, Ollama, Anthropic Claude, Gemini, Azure, Cohere, DeepSeek, Qwen, Moonshot, Zhipu, Mistral. LLMClient::from_env() auto-detects any of the 11 from environment variables. | | OpenAI-compatible endpoint (v0.22.4) | One generic OpenAICompatibleChat covers any base_url endpoint — keyless vLLM / LM Studio / SGLang / Ollama / internal gateways (no Authorization header is sent without a key), with GroqChat / OpenRouterChat / XaiChat presets, env constructors and extra_headers. | | OpenAI-compatible thin wrappers | DeepSeek / Qwen / Moonshot / Zhipu / Mistral reuse the OpenAI request path; each keeps its own error variant (ProviderError::DeepSeek, etc.) so you can tell which vendor failed. New vendors are cheap to add. | | Chat & Streaming | chat() (one full reply) and stream_chat() (first token in ~1s). Streaming chunks carry token usage (StreamChunk), so budget gates get real usage on the streaming path (v0.18). config.streaming = true makes chat() stream internally then aggregate. | | Function Calling | bind_tools() + result.tool_calls, the native path for tool-capable models. | | Multimodal Vision | Message::human_with_image / human_with_audio / human_with_file via schema ImageContent / AudioContent / FileContent. | | Thinking models | Reasoning is kept in LLMResult.thinking_content and never leaked into content (DeepSeek-R1, GLM-5.2, Claude Extended Thinking). | | OpenAI Assistants API | Stateful assistants with requires_action tool dispatch. | | OpenAI Responses API | Typed Responses-endpoint client (openai/responses/, ~1.4k lines): web_search / file_search / code-interpreter tooling. Chat Completions remains the portable default. | | Anthropic Extended Thinking | with_thinking for Claude reasoning. | | Model Routing | RouterLLM with 6 strategies — Fallback / RoundRobin / LeastLatency / LatencyWeighted(beta) (EMA-latency weighted draw, deterministic SplitMix64) / LowestCost (registry-priced, blended 0.75-in/0.25-out) / InputDirected — plus per-slot ModelRateLimit admission and a shared RouterBudget USD circuit breaker (v0.22.4). Remaining models always act as fallback. | | Cost & budgets (v0.22.4) | ModelPrice / PricingTable / shared CostTracker (run/session scopes, unknown models bill $0 and never break the loop), JSON-fetchable ModelRegistry of capabilities/prices, agent-level BudgetConfig.max_cost_usd hard gate on both invoke and stream paths, costs exported as ObsEvent::Cost. | | Batch API | BatchClient for OpenAI / Anthropic batch inference (~50% cost reduction). | | LLM Cache | LLMCache with TTL + true LRU eviction (hits refresh recency). | | Structured Output | with_structured_output + StructuredOutputExt trait, JsonOutputParser fallback, and streaming structured output via PartialJsonParser. | | Native JSON Schema (v0.21) | OpenAIChat::with_json_schema_output::() sends response_format: {type: "json_schema"} (strict mode) generated from schemars 1.0 — schema-constrained decoding on the provider side; make_strict_schema enforces additionalProperties: false + full required. with_structured_output remains the portable path. | | Token Counter | TiktokenCounter (precise) / CharRatioCounter (Chinese-friendly estimate) + TokenTrackingLLM usage stats + ModelPricing cost estimation. |

Embeddings

| Component | Description | |-----------|-------------| | Unified Embeddings trait | embed_query / embed_documents / dimension / model_name, with empty-input and batch-alignment checks enforced in the trait default path. | | Providers | OpenAI (ada-002 / 3-small / 3-large), DeepSeek, Qwen, Cohere (embed-v3.0, 4 input types), FastEmbed (local ONNX), Mock (deterministic, for tests), BagOfWords (local, always available). | | Qwen3-Embedding (v0.21) | qwen3-embedding-0.6b / 4b / 8b (dims 1024 / 2560 / 4096) + matryoshka output via QwenEmbeddingsConfig::with_dimensions(32..=4096); the configured dimensions is passed through to the DashScope request. | | Reliability | Exponential-backoff retries (429/5xx, max 3, 4xx not retried) with jitter and Retry-After header support (v0.21), concurrent batching (OpenAI: 2048 docs/batch, concurrency 8), and unified normalization so downstream similarity is provider-independent. | | Local ONNX | LocalEmbeddings via the local-embeddings feature (ort). | | Token-level embeddings (v0.21) | Optional TokenLevelEmbeddings capability trait (native async fn, statically dispatched): embed_tokens returns per-token TokenEmbedding { span, vector } with byte-offset spans. Local ONNX path shares the fastembed pipeline. | | Late chunking (v0.21, end-to-end in v0.23) | late_chunk(&embedder, text, &LateChunkConfig) embeds the whole text once at token level, then mean-pools token vectors per chunk range into L2-normalized chunk vectors — better context retention than chunk-then-embed for long documents. late_index_in(&vector_store, &embedder, parent_key, text, &config) (v0.23) closes the loop: one token-level pass → pooled chunks → straight into any VectorStore, with deterministic {parent}:{index} ids. Runnable offline demo: cargo run -p lc-rag --example late_chunking. | | Candle backend (v0.21) | CandleEmbeddings via the local-candle feature: pure-Rust CPU inference for BERT-family models (from_hf_hub("BAAI/bge-small-en-v1.5") or from_dir), masked mean-pooling, batch size 16. | | Vision embeddings (v0.22.4) | VisionEmbeddings trait maps (text/image) into one shared vector space: Cohere Embed v4 (CohereVisionEmbeddings, 1536-d) and DashScope multimodal-embedding-v1 (QwenVisionEmbeddings, 1024-d), plus MockVisionEmbeddings for tests. Feeds multimodal RAG (see below). |

Composition: Chains & LCEL

| Component | Description | |-----------|-------------| | LCEL | Runnable with four base actions — invoke / batch / stream / transform. Operators: pipe, RunnableLambda, RunnablePassthrough, RunnableParallel, RunnableBranch, RunnableBinding, RunnableWithFallbacks, RunnableAssign, with_retry, RunnableSequence. Type-erased with PhantomData — dynamic composition with compiler-checked type matches. | | Unified composition (v0.15) | Prompts, memory, native providers, parsers and RAG are all Runnable: prompt.pipe(llm).pipe(StrOutputParser) — no glue code. RunnableWithMessageHistory wraps "LLM + memory" as one runnable (auto read history → invoke → write back). RagRunnable makes retrieval-augmented generation one link of a chain. Native OpenAIChat/QwenChat/DeepSeekChat errors are unified into LcelError. | | Chains | BaseChain with 9 implementations: LLMChain, ConversationChain, SequentialChain, RouterChain, LLMRouterChain, RetrievalQA, ConversationRetrievalChain, plus the 4 document chains — Stuff / MapReduce / Refine / MapRerank. Chain streaming per token, ChainRunnable bridges chains into LCEL. | | Prompts | PromptTemplate (parsed once, cached segments), ChatPromptTemplate (Runnable, outputs Vec), FewShotPromptTemplate + ExampleSelectors (LengthBasedExampleSelector). {{/}} escapes, Chinese variable names, missing variables error loudly. v0.22.4 adds PromptRegistry — versioned, named prompt storage with render-by-name and fallback, so prompt text is managed centrally instead of scattered through call sites. | | Output Parsers | StrOutputParser, JsonOutputParser, CommaSeparatedListOutputParser, StructuredOutputParser, TypedOutputParser — all tolerant of dirty model output (markdown fences, trailing commas, trailing junk). | | Retrieval & Sessions in LCEL (v0.17) | RetrieverRunnable wraps any retriever as Runnable>; SessionManagerRunnable wraps persistent sessions as Runnable — both compose with pipe into a chain. | | Cancellation | CancellationToken threads through RunnableConfig into every execution. |

Agents & Multi-Agent

| Component | Description | |-----------|-------------| | BaseAgent / AgentExecutor | The "translator / butler" split: BaseAgent turns model output into a decision (Action / Actions / Finish), AgentExecutor is the one real loop — with tool timeouts, LLM retries, concurrency semaphore, and max_iterations clamped to [1, 100]. | | FunctionCallingAgent | Recommended path — reads native tool_calls (requires model support). | | ReActAgent | Text-regex thought/action loop, fallback for models without tool-calling. | | Plan-Execute | Planner → per-step executor → replan on failure; the executor factory is configurable (no longer hardcoded to function calling). | | DeepResearch | Multi-round research agent with sub-topic decomposition, parallel search, dedup, citation reporting. | | RAG Agents | CorrectiveRAGAgent (self-correcting grade/rewrite/detect), AdaptiveRAG (LLM-routed retrieval), as standalone graphs. | | Handoffs | Multi-agent handoff with max_handoff_depth (default 10) to stop A↔B ping-pong. | | Orchestrators | Supervisor (v0.24.0 — one router LLM delegates each round to a named sub-agent or FINISH, with scratchpad feedback and a bounded-rounds one-level recursion guard), FanOutFanIn (parallel fan-out + aggregate), SequentialPipeline (serial), OrchestratorRunnable for LCEL integration. | | Parallel tool calls (v0.24.0) | Multiple tool calls emitted in one model turn run concurrently, bounded by .with_max_concurrency(n) (default 8); observations zip back to actions in the model's call order on both invoke and stream paths. | | Agent Hooks | Approval (on_before_tool_call allow/reject/skip), PromptInjectionHook, TokenBudgetHook, ContentFilterHook, logging. | | Agent Gates (v0.16) | Async human-approval gate — .with_approval() (Allow / Deny / Modify; Deny feeds the reason back as an observation, Modify rewrites the arguments). Budget gate — .with_budget() with hard caps on tool calls / tokens / wall-clock duration / iterations, exceeding returns AgentError::BudgetExceeded. Both default off. | | Cross-process resume (v0.18) | FileResumeStore persists the pending human-approval / budget-gate state to disk (atomic write); a restarted executor loads the pending point and re-enters approval instead of restarting the agent loop. | | Context compaction (v0.21) | .with_compaction(CompactionConfig) — trigger on TurnCount / TokenCount / Any / All, compact with SlidingWindow or TokenBudget (turn-boundary truncation, no orphan tool results, min_recent_turns floor, default 2). Off by default; compaction count lands in AgentMetrics.compactions. | | Streaming | Token-level streaming via StreamingFunctionCallingAgent + AgentStreamEvent; tool-level events via AgentExecutor::stream. | | Web SSE (v0.22.4) | Optional axum SSE endpoint in lc-agents (sse-server feature) serving AgentStreamEvents to browsers — see the agent_sse_server example. | | Durable checkpoints (v0.22.4) | LangGraph persistence gains three production backends behind checkpoint-sqlite (rusqlite bundled, WAL) / checkpoint-postgres (tokio-postgres) / checkpoint-redis (Lua CAS) features. Optimistic concurrency: stale writes return GraphError::CheckpointVersionConflict instead of last-write-wins. | | Two-layer semantic memory (v0.22.4) | Episodic layer (raw per-turn observations, vector-retrieved) + semantic layer (LLM-consolidated, deduped facts) with an async background extractor — the agent accumulates durable knowledge across sessions instead of only replaying recent chat. | | Dynamic interrupt/resume + time travel (v0.24.0) | InterruptibleNode suspends inside a node — its closure runs with resume: None, raises an InterruptRequest payload into the checkpoint, then re-enters with resume: Some(decision) via CompiledGraph::resume_with_value (even from a fresh process over a durable checkpointer; NodeInterrupt/Resumed stream events). Agent approvals converge on this one path: ApprovalGate + ApprovalDecision (Allow / Deny / Modify), so the gated tool executes exactly once and Deny performs no side effect; the non-graph ResumeStore remains for the plain executor. get_state_history() lists CheckpointInfos and fork_from(checkpoint, node, override_state) branches a new forward-only lineage off any past state. | | Tool Policies | ToolPolicy / ToolRisk risk classification for tool access control. |

Retrieval & RAG

| Component | Description | |-----------|-------------| | Unified RetrieverTrait | All retrievers implement it: SimilarityRetriever, BM25Retriever / ChunkedBM25Retriever, UnifiedHybridIndex, ParentDocumentRetriever — so any retrieval strategy plugs into the RAG pipeline. | | RAGPipeline | RAGPipelineBuilder (llm + embeddings + vector store + retriever) → index_documents / query / query_with_sources (citation tracing). | | Document Loaders | Text / JSON / Markdown / PDF / CSV / HTML + WebScraper / Sitemap / D

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: atliliw
• Source: atliliw/langchainrust
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.