Vmlx

Chat with any MLX model -- thinking mode, streaming, and syntax highlighting Agentic chat with full coding capabilities -- tool use and structured output

Quickstart

Install from PyPI

Published on PyPI as vmlx -- install and run in one command:

# Recommended: uv (fast, no venv hassle)
brew install uv
uv tool install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

# Or: pipx (isolates from system Python)
brew install pipx
pipx install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

# Or: pip in a virtual environment
python3 -m venv ~/.vmlx-env && source ~/.vmlx-env/bin/activate
pip install vmlx
vmlx serve mlx-community/Qwen3-8B-4bit

> Note: On macOS 14+, bare pip install fails with "externally-managed-environment". Use uv, pipx, or a venv.

The vMLX inference server is now running at http://0.0.0.0:8000 with an OpenAI + Anthropic compatible API. Works with any model from mlx-community -- thousands of models ready to go.

Or download the desktop app

Get MLX Studio -- a native macOS app with chat UI, model management, image generation, and developer tools. No terminal required. Just download the DMG and drag to Applications.

Use with OpenAI SDK

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
response = client.chat.completions.create(
    model="local",
    messages=[{"role": "user", "content": "Hello!"}],
    stream=True,
)
for chunk in response:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Use with Anthropic SDK

import anthropic

client = anthropic.Anthropic(base_url="http://localhost:8000/v1", api_key="not-needed")
message = client.messages.create(
    model="local",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}],
)
print(message.content[0].text)

Use with curl

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": true
  }'

Model Support

vMLX runs any MLX model. Point it at a HuggingFace repo or local path and go.

| Type | Models | |------|--------| | Text LLMs | Qwen 2/2.5/3/3.5/3.6, Llama 3/3.1/3.2/3.3/4, Mistral/Mixtral, Mistral-Medium-3.5 (ministral3), Mistral-Small-4, Gemma 3/4, Phi-4, DeepSeek V2/V3/V4, GLM-4/5, MiniMax M2.5/M2.7, Nemotron, Laguna (poolside), ZAYA (CCA + MoE), Kimi K2.5/K2.6, StepFun, and any mlx-lm model | | Vision LLMs | Qwen-VL, Qwen3.5-VL / Qwen3.6-VL, Pixtral, InternVL, LLaVA, Gemma 3n / 4-VL, Mistral-Medium-3.5 (PIXTRAL) | | Multimodal Omni | Nemotron-3-Nano-Omni (text + image + audio + video) — Parakeet audio encoder + RADIO ViT vision tower; routed via OmniMultimodalDispatcher across /v1/chat/completions, /v1/messages, /v1/responses, /api/chat | | MoE Models | Qwen 3.5/3.6 MoE (A3B/A10B), Mixtral, DeepSeek V2/V3/V4, MiniMax M2.5/M2.7, Llama 4, Laguna (256 routed experts top-8) | | Hybrid SSM | Nemotron-H, Jamba, GatedDeltaNet (Mamba + Attention), Qwen3.5-A3B hybrid, Granite MoE Hybrid, LFM2 | | Image Gen | Flux Schnell/Dev, Z-Image Turbo (via mflux) | | Image Edit | Qwen Image Edit (via mflux) | | Embeddings | Any mlx-lm compatible embedding model | | Reranking | Cross-encoder reranking models | | Audio | Kokoro TTS, Whisper STT (via mlx-audio) |

Features

Inference Engine

| Feature | Description | |---------|-------------| | Continuous Batching | Handle multiple concurrent requests efficiently | | Prefix Cache | Reuse KV states for repeated prompts -- makes follow-up messages instant | | Paged KV Cache | Block-based caching with content-addressable deduplication | | KV Cache Quantization | Compress cached states to q4/q8 for 2-4x memory savings | | Disk Cache (L2) | Persist prompt caches to SSD -- survives server restarts | | Block Disk Cache | Per-block persistent cache paired with paged KV cache | | Speculative Decoding | Small draft model proposes tokens for 20-90% speedup | | Prompt Lookup Decoding | No draft model needed — reuses n-gram matches from the prompt/context. Best for structured or repetitive output (code, JSON, schemas). Enable with --enable-pld. | | JIT Compilation | mx.compile Metal kernel fusion (experimental) | | Hybrid SSM Support | Mamba/GatedDeltaNet layers handled correctly alongside attention | | Distributed Compute | Pipeline parallelism across multiple Macs via Thunderbolt 5 / Ethernet / WiFi |

Distributed Inference (Multi-Mac)

Run models too large for a single Mac across 2+ machines. Each Mac loads a subset of transformer layers and they communicate hidden states over the network.

# On worker Macs:
pip install vmlx
vmlx-worker --secret mysecret

# On coordinator Mac (runs the server):
vmlx serve JANGQ-AI/Qwen3.5-Coder-Rerank-397B-A27B-JANG_2L --distributed --cluster-secret mysecret

| Feature | Description | |---------|-------------| | Pipeline Parallelism | Split layers across nodes -- hidden state (~8KB/step) flows sequentially | | Auto-Discovery | Bonjour mDNS, UDP broadcast, HTTP probes, Tailscale, cached peers, manual IP | | Capability-Scored Election | Most powerful Mac becomes coordinator automatically | | Any Network Works | TB5 (120 Gbps), 10GbE, 1GbE, WiFi, Tailscale -- PP is not bandwidth-bound | | JANG Support | Each worker loads its layer range from JANG safetensors (mmap) | | Live Node List | Desktop app shows discovered nodes, link type, latency, layer assignments | | Cluster API | /v1/cluster/status, /v1/cluster/nodes, /v1/cluster/scan REST endpoints |

5-Layer Cache Architecture

Request -> Tokens
    |
L1: Memory-Aware Prefix Cache (or Paged Cache)
    | miss
L2: Disk Cache (or Block Disk Store)
    | miss
Inference -> float16 KV states
    |
KV Quantization -> q4/q8 for storage
    |
Store back into L1 + L2

Tool Calling

Auto-detected parsers for every major model family:

qwen - llama - mistral - hermes - deepseek - glm47 - minimax - nemotron - granite - functionary - xlam - kimi - step3p5

Reasoning / Thinking Mode

Auto-detected reasoning parsers that extract `` blocks:

qwen3 (Qwen3, QwQ, StepFun) - minimax_m2 (MiniMax M2/M2.5/M2.7) - deepseek_r1 (DeepSeek R1, Gemma 3, GLM, Phi-4) - openai_gptoss (GLM Flash, GPT-OSS)

Audio

| Feature | Description | |---------|-------------| | Text-to-Speech | Kokoro TTS via mlx-audio -- multiple voices, streaming output | | Speech-to-Text | Whisper STT via mlx-audio -- transcription and translation |

Image Generation & Editing

Generate and edit images locally with Flux models via mflux.

pip install vmlx[image]

# Image generation
vmlx serve schnell                    # or dev, z-image-turbo
vmlx serve ~/.mlxstudio/models/image/flux1-schnell-4bit

# Image editing
vmlx serve qwen-image-edit            # instruction-based editing

Generation API

curl http://localhost:8000/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "schnell",
    "prompt": "A cat astronaut floating in space with Earth in the background",
    "size": "1024x1024",
    "n": 1
  }'

# Python (OpenAI SDK)
response = client.images.generate(
    model="schnell",
    prompt="A cat astronaut floating in space",
    size="1024x1024",
    n=1,
)

Editing API

# Edit an image with a text prompt (Flux Kontext / Qwen Image Edit)
curl http://localhost:8000/v1/images/edits \
  -H "Content-Type: application/json" \
  -d '{
    "model": "flux-kontext",
    "prompt": "Change the background to a sunset",
    "image": "",
    "size": "1024x1024",
    "strength": 0.8
  }'

# Python
import base64
with open("source.png", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = requests.post("http://localhost:8000/v1/images/edits", json={
    "model": "flux-kontext",
    "prompt": "Make the sky purple",
    "image": image_b64,
    "size": "1024x1024",
    "strength": 0.8,
})

Supported Image Models

Generation Models:

| Model | Steps | Speed | Memory | |-------|-------|-------|--------| | Flux Schnell | 4 | Fastest | ~6-24 GB | | Z-Image Turbo | 4 | Fast | ~6-24 GB | | Flux Dev | 20 | Slow | ~6-24 GB |

Editing Models:

| Model | Steps | Type | Memory | |-------|-------|------|--------| | Qwen Image Edit | 28 | Instruction-based editing | ~54 GB |

API Reference

API Gateway

The desktop app runs an API Gateway on a single port (default 8080) that routes requests to all loaded models by name. Run multiple models simultaneously and access them all through one URL.

# All models accessible through the gateway
curl http://localhost:8080/v1/chat/completions \
  -d '{"model": "Qwen3.5-122B", "messages": [{"role": "user", "content": "Hi"}]}'

# Works with Ollama CLI too
OLLAMA_HOST=http://localhost:8080 ollama run Qwen3.5-122B

The gateway supports OpenAI, Anthropic, and Ollama wire formats. Configure the port in the API tab.

Endpoints

OpenAI / Anthropic

| Method | Path | Description | |--------|------|-------------| | POST | /v1/chat/completions | OpenAI Chat Completions API (streaming + non-streaming) | | POST | /v1/messages | Anthropic Messages API | | POST | /v1/responses | OpenAI Responses API | | POST | /v1/completions | Text completions | | POST | /v1/images/generations | Image generation | | POST | /v1/images/edits | Image editing (Qwen Image Edit) | | POST | /v1/embeddings | Text embeddings | | POST | /v1/rerank | Document reranking | | POST | /v1/audio/transcriptions | Speech-to-text (Whisper) | | POST | /v1/audio/speech | Text-to-speech (Kokoro) | | GET | /v1/models | List loaded models | | GET | /v1/cache/stats | Cache statistics | | GET | /health | Server health check |

Ollama

| Method | Path | Description | |--------|------|-------------| | POST | /api/chat | Chat completion (NDJSON streaming) | | POST | /api/generate | Text generation (NDJSON streaming) | | GET | /api/tags | List loaded models | | POST | /api/show | Model details | | POST | /api/embeddings | Generate embeddings |

curl Examples

Chat completion (streaming)

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Explain quantum computing in 3 sentences."}],
    "stream": true,
    "temperature": 0.7
  }'

Chat completion with thinking mode

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "Solve: what is 23 * 47?"}],
    "enable_thinking": true,
    "stream": true
  }'

Tool calling

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "messages": [{"role": "user", "content": "What is the weather in Tokyo?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {"type": "string", "description": "City name"}
          },
          "required": ["location"]
        }
      }
    }]
  }'

Anthropic Messages API

curl http://localhost:8000/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: not-needed" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "local",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Embeddings

curl http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "input": "The quick brown fox jumps over the lazy dog"
  }'

Text-to-speech

curl http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{
    "model": "kokoro",
    "input": "Hello, welcome to vMLX!",
    "voice": "af_heart"
  }' --output speech.wav

Speech-to-text

curl http://localhost:8000/v1/audio/transcriptions \
  -F file=@audio.wav \
  -F model=whisper

Image generation

curl http://localhost:8000/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "schnell",
    "prompt": "A mountain landscape at sunset",
    "size": "1024x1024"
  }'

Reranking

curl http://localhost:8000/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local",
    "query": "What is machine learning?",
    "documents": [
      "ML is a subset of AI",
      "The weather is sunny today",
      "Neural networks learn from data"
    ]
  }'

Cache stats

curl http://localhost:8000/v1/cache/stats

Health check

curl http://localhost:8000/health

Desktop App

vMLX includes a native macOS desktop app (MLX Studio) with 5 modes:

| Mode | Description | |------|-------------| | Chat | Conversation interface with chat history, thinking mode, tool calling, agentic coding | | Server | Manage model sessions -- start, stop, configure, monitor | | Image | Text-to-image generation and image editing with Flux, Kontext, Qwen, and Fill models | | Tools | Model converter, GGUF-to-MLX, inspector, diagnostics | | API | Live endpoint reference with copy-pasteable code snippets |

Image generation and editing with Flux models Developer tools -- model conversion and diagnostics

Anthropic Messages API endpoint -- full compatibility GGUF to MLX conversion -- bring your own models

Download

Get the latest DMG from MLX Studio Releases, or build from source:

git clone https://github.com/jjang-ai/vmlx.git
cd vmlx/panel
npm install && npm run build
npx electron-builder --mac dmg

Menu Bar

vMLX lives in your menu bar showing all running models, GPU memory usage, and quick controls.

Advanced Quantization

vMLX supports standard MLX quantization (4-bit, 8-bit uniform) out of the box. For users who want to push further, JANG adaptive mixed-precision assigns different bit widths to different layer types -- attention gets more bits, MLP layers get fewer -- achieving better quality at the same model size.

###

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: jjang-ai
• Source: jjang-ai/vmlx
• License: Apache-2.0
• Homepage: https://vmlx.net

Install and usage instructions live in the source repository linked above.