Local Inference Optimizer

Local Inference Optimizer

Use this skill when the user asks an agent to:

• determine the best inference engine for their hardware;
• set up a Python project with uv + .venv;
• choose kernels / quantization / model format;
• tune flags, batching, KV cache, context length, parallelism, and launch commands;
• optimize a model server for local use, LAN serving, or production-style OpenAI-compatible serving.

The core principle: do not pick an inference engine first. Pick from:

1. hardware strategy;
2. workload shape;
3. model architecture and format;
4. latency/throughput/cost/privacy target;
5. operational maturity required.

Then the engine and flags follow.

Required agent behavior

Do not give generic advice only. Produce a concrete, runnable setup backed by live checks.

1. Inspect hardware and environment first.
2. Ask only for missing non-discoverable inputs: model repo/path, target use case, desired context/concurrency, production vs local. If the user gave enough info, proceed.
3. Create or update a project using uv and .venv.
4. Select one primary engine and one fallback engine.
5. Install only the packages needed for the selected path.
6. Write launch scripts/config files.
7. Run a real smoke test: health endpoint and at least one completion/chat request.
8. Report exact commands, measured output, and next tuning knobs.

Never claim optimization succeeded without a real launch or a clearly stated blocker.

Inputs to collect

If not already provided, determine or ask for:

• Hardware: OS, CPU, RAM, GPU vendor/model/count, VRAM/unified memory, interconnect, driver/CUDA/ROCm/Metal availability.
• Model: Hugging Face repo or local path, model size, architecture, dense/MoE/multimodal, quantization format (GGUF, EXL2/EXL3, AWQ, GPTQ, FP8, BF16, MLX).
• Workload: short/long prompts, expected output length, context length, concurrency, shared system prompts/prefixes, streaming, JSON/structured output, LoRA use.
• Serving target: local CLI, desktop app, OpenAI-compatible API, LAN server, production service, multi-node/fleet.
• Optimization target: lowest latency, highest throughput, fits-in-memory, cheapest, privacy/offline, developer convenience.

Hardware discovery commands

Run the relevant commands and include outputs in your reasoning.

Cross-platform basics

uname -a || true
python3 --version || python --version || true
uv --version || true
free -h || vm_stat || true
df -h .

macOS / Apple Silicon

sw_vers
sysctl -n machdep.cpu.brand_string
sysctl -n hw.memsize | awk '{printf "RAM: %.1f GB\n", $1/1024/1024/1024}'
system_profiler SPDisplaysDataType SPHardwareDataType | sed -n '1,160p'

NVIDIA CUDA

nvidia-smi
nvidia-smi --query-gpu=name,memory.total,memory.free,driver_version,compute_cap --format=csv
python3 - .sh` — launch script.
- `scripts/smoke_test.py` — OpenAI-compatible test request.
- `benchmarks/bench_.sh` — minimal benchmark with realistic prompt/output lengths.
- `README.md` — exact usage and tuning notes.

Use `.env` only for secrets. Do not hardcode tokens in scripts.

## Engine-specific setup and tuning

### MLX / MLX-LM on Apple Silicon

Best when: MacBook/Mac Studio, unified memory, MLX-format model or easy conversion.

Install:

```bash
uv pip install mlx mlx-lm huggingface_hub

Serve:

python -m mlx_lm.server \
  --model "$MODEL" \
  --host 127.0.0.1 \
  --port 8000 \
  --max-tokens "$MAX_TOKENS" \
  --temp "$TEMPERATURE"

Tuning knobs:

• Keep context realistic; unified memory is large but bandwidth still matters.
• Prefer MLX-native quantized models where available.
• Use llama.cpp fallback when only GGUF exists or MLX serving lacks needed API behavior.

llama.cpp / llama-server

Best when: GGUF, CPU/Metal/CUDA/ROCm portability, laptops, edge, odd hardware.

Install/build options:

• macOS Homebrew: brew install llama.cpp
• Python package route: uv pip install llama-cpp-python only if embedding in Python.
• Source build if specific Metal/CUDA/ROCm flags are needed.

Typical server:

llama-server \
  -m "$MODEL_GGUF" \
  --host 127.0.0.1 \
  --port 8000 \
  -c "$CTX_SIZE" \
  -ngl "$GPU_LAYERS" \
  -b "$BATCH_SIZE" \
  -ub "$UBATCH_SIZE" \
  --parallel "$PARALLEL" \
  --cache-type-k "$CACHE_TYPE_K" \
  --cache-type-v "$CACHE_TYPE_V" \
  --flash-attn

Initial flags:

• Apple Silicon: -ngl 999, --flash-attn if supported, moderate -b/-ub.
• CPU-only: lower -c, smaller quant (Q4_K_M/Q5_K_M), tune threads with -t.
• CUDA/ROCm: -ngl 999, verify GPU offload in logs.
• KV cache: start --cache-type-k q8_0 --cache-type-v q8_0 for memory savings; compare quality/speed.
• Batch: increase -b for prefill throughput; reduce if memory pressure or latency spikes.

vLLM

Best when: production-style OpenAI API, continuous batching, PagedAttention, NVIDIA/AMD serving.

Install examples:

# CUDA wheels depend on platform; check current vLLM docs if install fails.
uv pip install vllm

Server:

python -m vllm.entrypoints.openai.api_server \
  --model "$MODEL" \
  --host 127.0.0.1 \
  --port 8000 \
  --dtype auto \
  --tensor-parallel-size "$TP" \
  --gpu-memory-utilization "$GPU_MEM_UTIL" \
  --max-model-len "$MAX_MODEL_LEN" \
  --max-num-seqs "$MAX_NUM_SEQS" \
  --max-num-batched-tokens "$MAX_BATCHED_TOKENS" \
  --enable-prefix-caching

Tuning knobs:

• --gpu-memory-utilization: start 0.85–0.92; lower if OOM/fragmentation.
• --max-model-len: do not set higher than needed; KV cache grows with context.
• --max-num-seqs: concurrency cap; raise for throughput, lower for latency/memory.
• --max-num-batched-tokens: higher helps prefill throughput; too high can hurt TTFT.
• --enable-prefix-caching: enable when system prompts/RAG prefixes repeat.
• Quantization: choose only formats with optimized kernels for your GPU (AWQ, GPTQ, FP8, etc. depending on model/GPU/vLLM support).
• Multi-GPU: set tensor parallel to GPU count only if model/activation/KV require it or throughput improves; PCIe-only systems may underperform.

SGLang

Best when: long context, MoE, structured output, routing, advanced scheduling, prefill/decode split.

Install:

uv pip install sglang[all]

Server:

python -m sglang.launch_server \
  --model-path "$MODEL" \
  --host 127.0.0.1 \
  --port 8000 \
  --tp "$TP" \
  --mem-fraction-static "$MEM_FRACTION_STATIC" \
  --context-length "$MAX_MODEL_LEN" \
  --enable-torch-compile

Tuning knobs:

• Long context/MoE: start here before vLLM if workload is complex.
• Use radix/prefix cache features when prompts share prefixes.
• Tune static memory fraction to avoid starving runtime memory.
• Verify structured-output requirements with a real JSON-constrained test.

ExLlamaV2 / ExLlamaV3

Best when: consumer NVIDIA local speed with EXL2/EXL3 quantized models.

Use if the user has compatible NVIDIA GPUs and a model already available in EXL format, or if conversion is part of the task.

Tuning knobs:

• Prefer EXL2/EXL3 quants matched to VRAM budget.
• Single RTX: ExLlamaV2 often wins for local low-latency quantized inference.
• 2–4 RTX: ExLlamaV3 if mature/available in the environment; verify multi-GPU support and server API.
• Do not choose ExLlama if the user needs broad production serving features over raw local speed.

TensorRT-LLM

Best when: NVIDIA datacenter hardware and peak performance matter more than portability.

Use only after confirming:

• supported NVIDIA GPU/driver/CUDA stack;
• model architecture is supported;
• user accepts build/engine compilation complexity;
• benchmark target justifies setup time.

Always keep vLLM or SGLang as fallback unless TensorRT-LLM is explicitly required.

Quantization and kernel selection rules

Choose quantization by optimized kernels, not just file size.

• Apple Silicon: MLX quantized models or GGUF with Metal via llama.cpp.
• llama.cpp: GGUF Q4_K_M for fit/speed baseline, Q5_K_M/Q6_K if memory allows, Q8_0 for quality near fp16 but larger.
• vLLM/SGLang NVIDIA: prefer BF16/FP16 on datacenter GPUs when memory allows; AWQ/GPTQ/FP8 only when supported by optimized kernels for the model/GPU.
• Consumer RTX: EXL2/EXL3 for ExLlama; AWQ/GPTQ or GGUF fallback depending engine.
• KV cache quantization: use when context/concurrency is KV-bound; benchmark quality and latency, not just memory.

Initial sizing heuristics

These are starting points, not guarantees. Always verify with real runs.

• Leave 5–15% GPU memory headroom.
• Lower context length first if OOM happens before lowering model quality.
• For local single-user chat, optimize for TTFT and TPOT, not maximum throughput.
• For API serving, optimize p95/p99 latency under realistic concurrency.
• Raise batch/prefill tokens until TTFT or memory becomes unacceptable.
• Prefix caching helps when system prompts/tools/RAG prefixes repeat; it does not help random unique prompts much.
• Multi-GPU over PCIe can be slower than one GPU with a smaller quantized model.

Smoke test

For OpenAI-compatible servers, create scripts/smoke_test.py:

import os
from openai import OpenAI

base_url = os.getenv('OPENAI_BASE_URL', 'http://127.0.0.1:8000/v1')
model = os.getenv('MODEL_NAME') or os.getenv('MODEL')
client = OpenAI(base_url=base_url, api_key=os.getenv('OPENAI_API_KEY', 'not-needed'))

resp = client.chat.completions.create(
    model=model,
    messages=[
        {'role': 'system', 'content': 'You are a concise assistant.'},
        {'role': 'user', 'content': 'Reply with exactly: inference smoke test ok'},
    ],
    temperature=0,
    max_tokens=16,
)
print(resp.choices[0].message.content)

Install client:

uv pip install openai httpx

Run:

OPENAI_BASE_URL=http://127.0.0.1:8000/v1 MODEL_NAME="$SERVED_MODEL_NAME" python scripts/smoke_test.py

Benchmark checklist

Benchmark beyond single-user tokens/sec:

• TTFT: time to first token.
• TPOT: time per output token.
• p50/p95/p99 latency.
• Prefill throughput and decode throughput separately.
• Throughput under realistic concurrency.
• GPU memory usage and KV cache utilization.
• Prefix cache hit rate when available.
• Cost per million tokens when relevant.

Suggested tools: engine-native benchmark scripts first; otherwise use a small Python/httpx load script with fixed prompt/output distributions.

Final answer format

When finished, report:

## Итог
- Primary engine:  — why
- Fallback engine:  — why
- Model: 
- Hardware detected: 
- Workload assumption: 

## Что создано
- 

## Команды запуска
```bash

Проверка

• Health:
• Smoke completion:
• Memory/latency notes:

Тюнинг дальше

If blocked, report the blocker, exact error, and the next viable alternative.

## Common pitfalls

- Picking vLLM just because it is popular, even on Mac or GGUF-only setups.
- Setting context length to the model maximum when the workload does not need it.
- Ignoring KV cache memory when estimating fit.
- Using quantization formats without optimized kernels on the chosen engine.
- Overusing tensor parallelism on PCIe consumer GPUs.
- Optimizing single-user tokens/sec while the real workload is concurrent serving.
- Treating Ollama convenience as production optimization.
- Skipping smoke tests and reporting unverified launch commands.

## Source inspiration

Based on Ahmad Osman's inference-engine decision guide thread: https://x.com/TheAhmadOsman/status/2057183854444843202
Key idea: inference engines are memory managers, schedulers, kernel dispatchers, cache accountants, parallelism planners, API surfaces, and sometimes deployment frameworks. Choose them from hardware + workload + operational needs, not from hype.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [ForgetMeAI](https://github.com/ForgetMeAI)
- **Source:** [ForgetMeAI/local-inference-optimizer-skill](https://github.com/ForgetMeAI/local-inference-optimizer-skill)
- **License:** MIT

Install and usage instructions live in the source repository linked above.