Mlx Serving

Apple Silicon LLM Serving (MLX / oMLX)

Operational guidance for running and tuning local LLM inference on Apple Silicon (M-series). Covers two common backends: Apple's mlx-lm and the oMLX server built on top of MLX.

Both expose OpenAI-compatible HTTP APIs, so client code is portable. The differences are in features, packaging, and the failure modes you hit when pushing models near the limits of unified memory.

---

Pre-Flight: Verify Environment

Before debugging any serving issue, confirm the basics:

# Hardware: confirm Apple Silicon and unified-memory size
system_profiler SPHardwareDataType | grep -E "Model|Chip|Memory"

# Activate the venv where mlx-lm is installed before the next two probes,
# otherwise they hit system Python and may report nothing useful.
python -c "import mlx.core as mx; print(mx.__version__, mx.default_device())"
python -c "import mlx_lm; print(mlx_lm.__version__)"

# oMLX (DMG install): inspect available subcommands and run health checks
omlx --help
omlx diagnose menubar   # checks Tahoe ControlCenter visibility (oMLX-specific)

# Confirm a process is actually listening
lsof -nP -iTCP -sTCP:LISTEN | grep -E "omlx|mlx_lm"

Verify which backend is bound before debugging: both can serve OpenAI- compatible endpoints on user-chosen ports, and a client request hitting the wrong backend silently produces correct-looking but unrelated output.

---

Backend Decision: mlx-lm vs oMLX

| Concern | mlx-lm | oMLX | |---|---|---| | Install | pip install mlx-lm | DMG (macOS app bundle) | | Process model | One model per mlx_lm.server process | Built-in registry; multi-model | | Config surface | Minimal CLI flags | Rich per-model JSON (model_settings.json) | | Feature flags | --temp, --top-p, basic | turboquant_kv, dflash, MTP, specprefill, thinking budget | | Tool calling | Format depends on chat template | Per-model parser (configurable) | | Models > RAM | Not supported (OOM on load) | OptiQ proxy build (sensitivity-driven quant) | | GUI | None | Native macOS app + menu bar | | Default port | 8080 | 8000 | | Best for | Library-first scripts, embedding in Python | Long-running daemon, multiple models, GUI ops |

> Skill bias: the depth here leans oMLX — feature flags, cache controls, > upstream bug patterns. mlx-lm coverage is lighter and focuses on the > shared substrate.

Pick mlx-lm when: you control the Python process, you want minimal surface area, you serve one model at a time, and the model fits comfortably in RAM.

Pick oMLX when: you need a daemon that survives terminal sessions, you serve multiple models behind one endpoint, you want feature flags (turboquant, dflash, MTP) without writing them yourself, or your model exceeds RAM and you need OptiQ proxy.

Both backends share the same fundamental constraints (unified memory, KV cache pressure, model architecture quirks), so most troubleshooting in this skill applies to either.

---

Symptom → Cause Triage

| Symptom | Likely cause | First move | |---|---|---| | OOM during model load | Model + KV scratch > available RAM | Smaller quant; OptiQ proxy (oMLX); free other apps | | OOM during long-context inference | KV cache growth | Enable turboquant_kv; set --paged-ssd-cache-dir for spill-to-SSD (CLI flag is the operative control; dflash_* keys are version-dependent overrides) | | Crash / hang at batch > 1 | Architecture cache shape mismatch | Pin --max-concurrent-requests 1 (oMLX); see upstream-bug-patterns | | Tool calls return as plaintext content | Parser doesn't recognize the model's tool-call JSON shape | See upstream-bug-patterns; check tool_calls field in response | | Throughput regression after config change | Feature flag interaction | Bisect via bench, one flag at a time | | Server hangs on startup | Large checkpoint loading from cold disk | Wait; tail server.log; check disk I/O via iostat | | Wrong / garbled outputs | Quantization too aggressive, or wrong chat template | Reduce turboquant_kv_bits or disable; verify chat template matches model card | | First request slow, rest fast | Cold prefix cache | Enable specprefill; warm cache with a dummy request | | Output cut off mid-sentence | max_tokens too low; thinking budget exhausted | Raise max_tokens; check thinking_budget_enabled |

For deeper triage on each symptom: references/symptoms.md.

---

Core Principles

1. Lower --max-concurrent-requests to 1 for batch-fragile architectures.

oMLX defaults to --max-concurrent-requests 8. Higher concurrency exposes architecture-specific batch handling bugs (notably in cache implementations like ChunkedKVCache used by Llama-4). For affected architectures, pin concurrency to 1 at launch and bench upward only after confirming clean output at batch>1.

1. One flag at a time. Feature flags interact (e.g., dflash +

turboquant_kv both touch the KV cache path). Changing two flags simultaneously and observing a regression makes attribution impossible. Bench between each change.

1. Bench, don't guess. "It feels faster" is not a tuning signal.

Maintain a small bench suite (chat / coding / tool-calling correctness + throughput) and re-run it after every config change. See references/bench-methodology.md.

1. Match chat templates to model cards. Tool-calling and reasoning

behavior depend on the chat template applied at request time. The model's HuggingFace card is authoritative.

1. Quantization is a curve, not a switch. turboquant_kv_bits=4.0

keeps most quality; lower bits trade quality for memory. Always include a quality cell in the bench suite — a config that passes throughput but fails coding can be worse than slower-but-correct.

1. OptiQ proxy is for the "model > RAM" case only. It builds a

sensitivity-driven proxy of the model so per-layer quant decisions can be made without holding the full model in memory. It's not a general speedup; for in-RAM models it adds startup cost without runtime benefit.

1. Server-side bugs masquerade as config problems. When a symptom

reproduces across multiple config combinations, suspect upstream rather than chasing more flags. See references/upstream-bug-patterns.md.

---

References

• [references/symptoms.md](references/symptoms.md) — symptom triage with

diagnostics and fixes (OOM, batch>1 crashes, tool-call-as-text, throughput regression, log triage)

• [references/omlx-feature-flags.md](references/omlx-feature-flags.md) —

per-flag reference (turboquantkv, dflash, MTP, specprefill, thinking budget, max-concurrent-requests, forcesampling) with interactions

• [references/bench-methodology.md](references/bench-methodology.md) —

bench-it-don't-guess: suite design, backend-agnostic harness shape, scoring, change attribution

• [references/upstream-bug-patterns.md](references/upstream-bug-patterns.md)

— when to suspect upstream vs config; two recurring bug patterns (cache-shape mismatch under batched scheduling; model tool-call format not parsed) with diagnostics and reporting guidance

---

What This Skill Does Not Cover

• Cloud LLM hosting (Bedrock, OpenAI API, Anthropic API) — different

surface entirely

• Non-MLX local backends (llama.cpp, Ollama, vLLM) — overlapping problem

space but different tooling

• Model training, fine-tuning, or LoRA — see MLX training tutorials
• General Python / HTTP debugging unrelated to MLX
• Model card creation or HuggingFace upload workflows

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: AeyeOps
• Source: AeyeOps/aeo-skill-marketplace
• License: MIT

Install and usage instructions live in the source repository linked above.