# Modal Llm Serving

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-jamesrobmccall-modal-skills-modal-llm-serving`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [jamesrobmccall](https://agentstack.voostack.com/s/jamesrobmccall)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [jamesrobmccall](https://github.com/jamesrobmccall)
- **Source:** https://github.com/jamesrobmccall/modal-skills/tree/main/skills/modal-llm-serving

## Install

```sh
agentstack add skill-jamesrobmccall-modal-skills-modal-llm-serving
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Modal LLM Serving

## Quick Start

1. Verify the actual local Modal environment before writing code.

```bash
modal --version
python -c "import modal,sys; print(modal.__version__); print(sys.executable)"
modal profile current
```

- Do not assume the default `python` interpreter matches the environment behind the `modal` CLI.

2. Confirm that the request is really about self-hosted open-weight text generation on Modal.

- Standard online API
- Cold-start-sensitive vLLM deployment
- Low-latency interactive serving
- High-throughput or offline batch text inference

3. Read [references/performance-playbook.md](references/performance-playbook.md) and then exactly one primary reference.

- Standard online API: [references/vllm-online-serving.md](references/vllm-online-serving.md)
- Cold starts: [references/vllm-cold-starts.md](references/vllm-cold-starts.md)
- Low latency: [references/sglang-low-latency.md](references/sglang-low-latency.md)
- Throughput or batch inference: [references/vllm-throughput.md](references/vllm-throughput.md)

4. Default to vLLM plus `@modal.web_server` unless the user explicitly optimizes for lowest latency or offline throughput.
5. Ground every implementation in the actual workload: target latency or throughput, model size and precision, GPU type and count, region, concurrency target, and cold-start tolerance.

## Choose the Workflow

- Use vLLM with `@modal.web_server` for the default OpenAI-compatible serving path. Read [references/vllm-online-serving.md](references/vllm-online-serving.md).
- Use vLLM with memory snapshots and a sleep or wake flow only when cold-start latency is a first-class requirement. Read [references/vllm-cold-starts.md](references/vllm-cold-starts.md).
- Use SGLang with `modal.experimental.http_server`, explicit region selection, and sticky routing when the user cares most about latency. Read [references/sglang-low-latency.md](references/sglang-low-latency.md).
- Use the vLLM Python `LLM` interface inside `@app.cls` or another batch worker when the task is about tokens per second or tokens per dollar rather than per-request HTTP behavior. Read [references/vllm-throughput.md](references/vllm-throughput.md).

## Default Rules

- Pin model revisions when pulling from Hugging Face or another mutable registry.
- Cache model weights in one Modal Volume and engine compilation artifacts in another when the runtime produces them.
- Set `HF_XET_HIGH_PERFORMANCE=1` for Hub downloads unless the environment has a specific reason not to.
- Include a readiness check before reporting success. Add a smoke-test path such as a `local_entrypoint` or a small client that hits `/health` and one OpenAI-compatible request.
- Treat `max_inputs`, `target_inputs`, tensor parallelism, and related knobs as workload-specific. Start conservative and benchmark before increasing them.
- Expose only the ports and endpoints the task actually needs.
- Keep one serving pattern per file unless the user explicitly asks for a comparison artifact or benchmark harness.
- Use SGLang only when lowest latency is the explicit objective and the extra setup is justified.
- Use the vLLM Python `LLM` interface only for offline or batch inference that does not need per-request HTTP behavior.
- Use snapshot-based cold-start reduction only when startup latency matters enough to justify extra operational complexity.
- Keep the scope on self-hosted text generation engines. Do not stretch this skill to cover embeddings, generic `transformers` pipelines, diffusion inference, or purely hosted-API usage.
- If the task is really about training or post-training, stop and use `modal-finetuning`.
- If the task is really about detached job orchestration, retries, or `.map` and `.spawn`, stop and use `modal-batch-processing`.
- If the task is really about isolated interactive execution or sandbox lifecycle, stop and use `modal-sandbox`.

## Validate

- Run `npx skills add . --list` after editing the package metadata or skill descriptions.
- Keep `evals/evals.json` and `evals/trigger-evals.json` aligned with the actual workflow boundary of the skill.
- Run `python3 -m py_compile skills/modal-llm-serving/scripts/qwen3_throughput.py` when changing the throughput artifact.

## References

- Read [references/performance-playbook.md](references/performance-playbook.md) first for objective-setting, engine selection, and tuning priorities.
- Read [references/vllm-online-serving.md](references/vllm-online-serving.md) for the default HTTP serving path.
- Read [references/vllm-cold-starts.md](references/vllm-cold-starts.md) only when cold-start reduction is worth snapshot complexity.
- Read [references/sglang-low-latency.md](references/sglang-low-latency.md) only when the task explicitly optimizes for low latency.
- Read [references/vllm-throughput.md](references/vllm-throughput.md) only when the workload is throughput or batch oriented.
- Read [references/example-patterns.md](references/example-patterns.md) for compact serving templates and adaptation paths.
- Read [references/troubleshooting.md](references/troubleshooting.md) for common serving failures and recovery steps.
- Adapt [scripts/qwen3_throughput.py](scripts/qwen3_throughput.py) for throughput-oriented batch workloads with a pinned model and local benchmark entrypoint.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [jamesrobmccall](https://github.com/jamesrobmccall)
- **Source:** [jamesrobmccall/modal-skills](https://github.com/jamesrobmccall/modal-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-jamesrobmccall-modal-skills-modal-llm-serving
- Seller: https://agentstack.voostack.com/s/jamesrobmccall
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.