Fastdeploy Deploy Simple

FastDeploy Simple Deployment

A skill to install FastDeploy, start an OpenAI-compatible server, and validate the API — all in a few commands.

What this skill does

• Detect hardware backend (NVIDIA CUDA SM80/90, NVIDIA CUDA SM86/89, Kunlunxin XPU, Hygon DCU, or CPU)
• Install PaddlePaddle and FastDeploy with the appropriate backend packages from PaddlePaddle's package index
• Start the FastDeploy server via python -m fastdeploy.entrypoints.openai.api_server
• Test the OpenAI-compatible API (/v1/chat/completions)
• Support virtual environment isolation
• Support streaming and non-streaming responses

Prerequisites

• OS: Linux (X86_64)
• Python: 3.10 – 3.12
• GPU Driver ≥ 535, CUDA ≥ 12.3, cuDNN ≥ 9.5 (NVIDIA GPU path)
• pip package manager (uv also supported)
• curl (for API testing)

> Important: FastDeploy packages are NOT available on PyPI. You must install from PaddlePaddle's dedicated package index as shown below.

Installation

NVIDIA GPU (SM80/90 — A30/A100/H100/A800/H800)

# Step 1: Install PaddlePaddle GPU
python -m pip install paddlepaddle-gpu==3.3.0 \
    -i https://www.paddlepaddle.org.cn/packages/stable/cu126/

# Step 2: Install FastDeploy GPU (SM80/90 index)
python -m pip install fastdeploy-gpu==2.4.0 \
    -i https://www.paddlepaddle.org.cn/packages/stable/fastdeploy-gpu-80_90/ \
    --extra-index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

NVIDIA GPU (SM86/89 — A10/L20/L40/RTX 4090)

# Step 1: Install PaddlePaddle GPU
python -m pip install paddlepaddle-gpu==3.3.0 \
    -i https://www.paddlepaddle.org.cn/packages/stable/cu126/

# Step 2: Install FastDeploy GPU (SM86/89 index)
python -m pip install fastdeploy-gpu==2.4.0 \
    -i https://www.paddlepaddle.org.cn/packages/stable/fastdeploy-gpu-86_89/ \
    --extra-index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

> Note: The pre-built Docker image only supports SM80/90 GPUs. If deploying on SM86/89 GPUs (L40/4090/L20), reinstall fastdeploy-gpu after creating the container using the SM86/89 index above.

Kunlunxin XPU (P800)

# Step 1: Install PaddlePaddle XPU
python -m pip install paddlepaddle-xpu==3.3.0 \
    -i https://www.paddlepaddle.org.cn/packages/stable/xpu-p800/

# Step 2: Install FastDeploy XPU
python -m pip install fastdeploy-xpu==2.4.0 \
    -i https://www.paddlepaddle.org.cn/packages/stable/fastdeploy-xpu-p800/ \
    --extra-index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple

Verify installation

python -c "import fastdeploy; print(fastdeploy.__version__)"

Usage

Create a virtual environment (optional but recommended)

# Using uv (recommended)
uv venv .venv --python 3.12
source .venv/bin/activate

# Or using standard venv
python3 -m venv .venv
source .venv/bin/activate

Run the complete workflow (suggested)

Use scripts/quickstart.sh with default options (model: baidu/ERNIE-4.5-0.3B-Paddle, port: 8180):

scripts/quickstart.sh

Or with custom options:

# Specify model and port
scripts/quickstart.sh --model "baidu/ERNIE-4.5-0.3B-Paddle" --port 8180

# With GPU memory utilization
scripts/quickstart.sh --model "baidu/ERNIE-4.5-21B-A3B-Paddle" --port 8180 --gpu_memory_utilization 0.8

# With virtual environment
scripts/quickstart.sh --venv /path/to/venv --model "baidu/ERNIE-4.5-0.3B-Paddle" --port 8180

Start server manually

export ENABLE_V1_KVCACHE_SCHEDULER=1
python -m fastdeploy.entrypoints.openai.api_server \
    --model baidu/ERNIE-4.5-0.3B-Paddle \
    --port 8180 \
    --metrics-port 8181 \
    --engine-worker-queue-port 8182 \
    --max-model-len 32768 \
    --max-num-seqs 32

Key parameters:

• --model: Model name or local path. If not found locally, auto-downloads from AIStudio.
• --port: HTTP service port (default 8000; quick start guide uses 8180)
• --metrics-port: Prometheus metrics endpoint port
• --max-model-len: Maximum context length in tokens (default: 2048)
• --max-num-seqs: Maximum concurrent requests in Decode phase (default: 8)
• --tensor-parallel-size: Number of GPUs for tensor parallelism (default: 1)
• --gpu-memory-utilization: GPU memory utilization ratio (default: 0.9)
• --enable-prefix-caching: Enable prefix caching to speed up repeated prompts
• --quantization: Online quantization: wint4, wint8, block_wise_fp8, wfp8afp8

> Note: --max-model-len and --max-num-seqs should be tuned based on your GPU memory. See Parameter Documentation for guidance.

Run individual script commands (step-by-step)

# Install only
scripts/quickstart.sh install

# Start server
scripts/quickstart.sh start --model "baidu/ERNIE-4.5-0.3B-Paddle" --port 8180

# Test API
scripts/quickstart.sh test --port 8180

# Check status
scripts/quickstart.sh status

# Stop server
scripts/quickstart.sh stop

# Restart server
scripts/quickstart.sh restart --port 8180

Configuration

scripts/quickstart.sh [command] [OPTIONS]

Commands:
  install  - Install PaddlePaddle + FastDeploy for detected hardware
  start    - Start the FastDeploy OpenAI-compatible server
  stop     - Stop the FastDeploy server
  test     - Test the OpenAI-compatible API
  status   - Show server status and recent logs
  restart  - Stop then start the server
  all      - Run complete workflow (default)

Options:
  --model MODEL                  Model name or local path
                                 (default: baidu/ERNIE-4.5-0.3B-Paddle)
  --port PORT                    Server port (default: 8180)
  --venv VENV_PATH               Virtual environment path (optional)
  --gpu_memory_utilization VRAM  GPU memory utilization ratio (default: 0.8)
  --max_model_len N              Max context length in tokens (default: 32768)
  --max_num_seqs N               Max concurrent requests (default: 32)

Hardware Backend Detection

The script automatically detects your hardware:

| Hardware | Detection Method | Package | |---|---|---| | NVIDIA CUDA SM80/90 | nvidia-smi, computecap ≤ 8.0 or ≥ 9.0 | fastdeploy-gpu (8090 index) | | NVIDIA CUDA SM86/89 | nvidia-smi, computecap 8.6–8.9 | fastdeploy-gpu (8689 index) | | Kunlunxin XPU | xpu_smi or KUNLUN_VISIBLE_DEVICES | fastdeploy-xpu | | Hygon DCU | /dev/kfd + /dev/dri devices | ROCm-compatible build | | CPU | Fallback | fastdeploy |

API Testing

After the server starts successfully, you should see:

api_server.py[line:91] Launching metrics service at http://0.0.0.0:8181/metrics
api_server.py[line:94] Launching chat completion service at http://0.0.0.0:8180/v1/chat/completions
INFO:     Uvicorn running on http://0.0.0.0:8180 (Press CTRL+C to quit)

Health check

curl -i http://localhost:8180/health

Chat completion (curl)

curl -X POST "http://localhost:8180/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Say hello!"}],
    "max_tokens": 50,
    "stream": false
  }'

Streaming response

curl -X POST "http://localhost:8180/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Write a short poem."}],
    "stream": true
  }'

Python client (OpenAI-compatible)

import openai

client = openai.Client(
    base_url="http://localhost:8180/v1",
    api_key="null"
)

response = client.chat.completions.create(
    model="null",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Write me a poem about large language models."},
    ],
    stream=True,
)
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")
print()

Supported Models

FastDeploy supports auto-downloading models from AIStudio (default), ModelScope, or HuggingFace:

# Set download source (AISTUDIO / MODELSCOPE / HUGGINGFACE)
export FD_MODEL_SOURCE=AISTUDIO
# Set download cache directory
export FD_MODEL_CACHE=/ssd1/models

Large Language Models (LLM)

| Model | Precisions | Example Model ID | |---|---|---| | ERNIE-4.5-0.3B | BF16 | baidu/ERNIE-4.5-0.3B-Paddle | | ERNIE-4.5-21B-A3B | BF16, WINT4, WINT8 | baidu/ERNIE-4.5-21B-A3B-Paddle | | ERNIE-4.5-21B-A3B-Thinking | BF16 | baidu/ERNIE-4.5-21B-A3B-Thinking | | ERNIE-4.5-300B-A47B | BF16, WINT4, WINT8, FP8 | baidu/ERNIE-4.5-300B-A47B-Paddle | | Qwen3 | BF16, WINT8, FP8 | Qwen/qwen3-8B | | Qwen3-MoE | BF16, WINT4, FP8 | Qwen/Qwen3-30B-A3B | | Qwen2.5 | BF16, WINT8, FP8 | Qwen/qwen2.5-7B | | DeepSeek-V3/R1 | BF16, WINT4 | unsloth/DeepSeek-V3-0324-BF16 | | GLM-4.5/4.6 | BF16, wfp8afp8 | zai-org/GLM-4.5-Air |

Multimodal Language Models (VLM)

| Model | Precisions | Example Model ID | |---|---|---| | ERNIE-4.5-VL-28B-A3B | BF16, WINT4, WINT8 | baidu/ERNIE-4.5-VL-28B-A3B-Paddle | | ERNIE-4.5-VL-28B-A3B-Thinking | BF16 | baidu/ERNIE-4.5-VL-28B-A3B-Thinking | | ERNIE-4.5-VL-424B-A47B | BF16, WINT4, WINT8 | baidu/ERNIE-4.5-VL-424B-A47B-Paddle | | PaddleOCR-VL | BF16, WINT4, WINT8 | PaddlePaddle/PaddleOCR-VL | | Qwen2.5-VL | BF16, WINT4, FP8 | Qwen/Qwen2.5-VL-7B-Instruct |

See full supported models list.

Troubleshooting

fastdeploy not found / import error:

• Ensure you installed from the PaddlePaddle index, NOT PyPI
• Check: python -c "import fastdeploy; print(fastdeploy.__version__)"

Server won't start:

• Check port conflicts: lsof -i :8180
• Verify GPU: nvidia-smi (NVIDIA) or xpu_smi (Kunlunxin)
• Check logs: cat ./tmp/fastdeploy-server.log

Model download fails:

• Set FD_MODEL_SOURCE to MODELSCOPE or HUGGINGFACE if AIStudio is unreachable
• For HuggingFace private models: set HF_TOKEN environment variable

Out of memory (OOM):

• Use a smaller model (e.g., baidu/ERNIE-4.5-0.3B-Paddle)
• Reduce --gpu-memory-utilization (e.g., 0.7)
• Reduce --max-num-seqs or --max-model-len

Wrong backend detected:

• NVIDIA: ensure nvidia-smi is in PATH
• Kunlunxin: set KUNLUN_VISIBLE_DEVICES or ensure xpu_smi is in PATH

Notes

• Server logs: ./tmp/fastdeploy-server.log (or $VENV_PATH/tmp/fastdeploy-server.log)
• Server PID: ./tmp/fastdeploy-server.pid
• First run downloads model weights automatically; subsequent runs use cached weights
• ENABLE_V1_KVCACHE_SCHEDULER=1 enables the V1 KV cache scheduler (recommended)
• Add --enable-prefix-caching and --swap-space 50 (GB) to enable prefix caching

References

• FastDeploy Quick Start (10-Min Deploy)
• NVIDIA GPU Installation Guide
• Kunlunxin XPU Installation Guide
• Online Serving Documentation
• Full Parameter Reference
• Supported Models List
• FastDeploy Documentation Site

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: yuanlehome
• Source: yuanlehome/fastdeploy-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.