Llama Cpp

llama.cpp -- C/C++ LLM Inference Framework Guide

Official Documentation

| Source | URL | |--------|-----| | GitHub Repository | https://github.com/ggml-org/llama.cpp | | C API Header (llama.h) | https://github.com/ggml-org/llama.cpp/blob/master/include/llama.h | | C++ RAII Wrappers | https://github.com/ggml-org/llama.cpp/blob/master/include/llama-cpp.h | | Build Instructions | https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md | | Server Documentation | https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md | | Quantization Tool | https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md | | GGUF Specification | https://github.com/ggml-org/ggml/blob/master/docs/gguf.md | | Function Calling Docs | https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md | | Multimodal Docs | https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md | | Examples Directory | https://github.com/ggml-org/llama.cpp/tree/master/examples | | HuggingFace GGUF Hub | https://huggingface.co/docs/hub/gguf-llamacpp | | Llama-Unreal Plugin | https://github.com/getnamo/Llama-Unreal |

What is llama.cpp?

llama.cpp is a pure C/C++ LLM inference engine with minimal dependencies, designed for high-performance local inference across CPUs and GPUs. Key properties:

• MIT licensed, extremely active development (~daily releases, currently b8766+)
• Widest hardware support: NVIDIA (CUDA), AMD (ROCm/Vulkan), Apple (Metal), Intel (SYCL/Vulkan), Qualcomm (OpenCL), ARM, WebGPU
• GGUF model format: single-file, mmap-compatible, 40+ quantization types from 1.5-bit to 16-bit
• Built-in HTTP server: OpenAI-compatible API, Anthropic Messages API, streaming, function calling, multimodal
• Language bindings: Python (llama-cpp-python), Go, Rust, C#, Node.js, Java, Swift, and more

Quick Start

Run a model via server (fastest path)

# Install
brew install llama.cpp    # macOS/Linux
winget install llama.cpp  # Windows

# Start server with a HuggingFace model
llama-server -hf bartowski/Llama-3.3-70B-Instruct-GGUF:Q4_K_M -ngl 99

# Query via curl
curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello"}],"temperature":0.8}'

Embed in a C++ project (library usage)

#include "llama.h"

// 1. Init backends and load model
ggml_backend_load_all();
auto params = llama_model_default_params();
params.n_gpu_layers = 99;
llama_model * model = llama_model_load_from_file("model.gguf", params);

// 2. Create context
auto ctx_params = llama_context_default_params();
ctx_params.n_ctx = 4096;
llama_context * ctx = llama_init_from_model(model, ctx_params);

// 3. Tokenize, decode, sample (see C API reference for full pattern)

Build from source with GPU

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON      # or GGML_VULKAN=ON, GGML_METAL=ON
cmake --build build --config Release -j

Core Architecture

llama.cpp/
  include/
    llama.h          # C API (primary interface)
    llama-cpp.h      # C++ RAII wrappers (unique_ptr aliases)
    ggml.h           # Tensor computation library
  common/
    common.h         # High-level convenience layer
  tools/
    server/          # HTTP server (llama-server)
    quantize/        # Model quantization tool
  examples/
    simple/          # Minimal inference example
    simple-chat/     # Multi-turn chat example

Inference pipeline:

Model (GGUF file)
  -> llama_model_load_from_file()  [load + mmap weights]
  -> llama_init_from_model()       [create context with KV cache]
  -> llama_tokenize()              [text -> tokens]
  -> llama_decode()                [run transformer, fill KV cache]
  -> llama_get_logits()            [get output probabilities]
  -> llama_sampler_sample()        [select next token]
  -> llama_token_to_piece()        [token -> text]
  -> repeat decode/sample loop until EOS

Quantization Quick Reference

| Type | Bits | Quality | Recommended For | |------|------|---------|-----------------| | Q4KM | ~4.5 | Good | Default choice -- best quality/size balance | | Q5KM | ~5.5 | Very good | When ~20% more space is acceptable | | Q80 | 8 | Near-lossless | Validation, quality-critical tasks | | IQ4XS | ~4.25 | Best at 4-bit | With imatrix, slightly smaller than Q4KM | | Q3KM | ~3.5 | Acceptable | RAM-constrained scenarios | | IQ2_XS | ~2.3 | Reduced | Extreme compression (needs imatrix) | | F16 | 16 | Reference | Full precision baseline |

# Quantize a model
llama-quantize model-f16.gguf model-q4km.gguf Q4_K_M

# Convert from HuggingFace
python convert_hf_to_gguf.py /path/to/hf_model --outfile model.gguf

GPU Backends

| Backend | Flag | Hardware | |---------|------|----------| | CUDA | GGML_CUDA=ON | NVIDIA GPUs | | Metal | auto on macOS | Apple Silicon | | Vulkan | GGML_VULKAN=ON | Cross-platform (NVIDIA/AMD/Intel) | | HIP | GGML_HIP=ON | AMD GPUs (ROCm) | | SYCL | GGML_SYCL=ON | Intel GPUs |

GPU offloading (-ngl 99) is the single most impactful performance setting.

Server API

The built-in server provides OpenAI-compatible endpoints:

| Endpoint | Purpose | |----------|---------| | POST /v1/chat/completions | Chat (streaming supported) | | POST /v1/completions | Text completion | | POST /v1/embeddings | Embeddings | | POST /v1/messages | Anthropic Messages API | | POST /completion | Native API with full parameter control | | GET /health | Health check | | GET /metrics | Prometheus metrics |

Features: function calling (--jinja), grammar constraints (--grammar/--json-schema), multimodal (vision+audio), parallel decoding, speculative decoding, router mode, LoRA hot-swap, built-in web UI.

CMake Integration

# Method 1: Subdirectory (embedding)
add_subdirectory(vendor/llama.cpp)
target_link_libraries(myapp PRIVATE llama ggml)

# Method 2: Installed package
find_package(llama REQUIRED)
target_link_libraries(myapp PRIVATE llama)

Detailed Reference Documents

• C/C++ API reference: See [references/c-api-reference.md](references/c-api-reference.md) for complete llama.h function signatures, types, enums, structs, sampling chain API, and full working examples
• Build system & integration: See [references/build-and-integration.md](references/build-and-integration.md) for all CMake options, GPU backend builds, library linking methods, Docker, and package managers
• Server REST API: See [references/server-api.md](references/server-api.md) for all HTTP endpoints, CLI flags, environment variables, curl examples, function calling, grammar constraints, and Python client usage
• GGUF & quantization: See [references/quantization-guide.md](references/quantization-guide.md) for GGUF format spec, all 40+ quantization types, imatrix generation, model conversion, hardware requirements, and supported architectures
• Performance & GPU backends: See [references/performance-and-backends.md](references/performance-and-backends.md) for backend comparison, CUDA/Vulkan/Metal optimization, memory management, speculative decoding, and hardware recommendations
• Unreal Engine integration: See [references/unreal-engine-integration.md](references/unreal-engine-integration.md) for Llama-Unreal plugin, custom C++ integration with Build.cs, HTTP server approach, performance in-game, and common UE pitfalls

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: maystudios
• Source: maystudios/claude-skills
• License: MIT

Install and usage instructions live in the source repository linked above.