Vllm Test Generator

vLLM Test Generator

Generate well-structured tests for vllm-project/vllm, following the conventions of the existing test suite.

Step 1 — Fetch Context

Before writing, fetch the relevant source and existing tests:

# Browse existing tests for the target area
gh api repos/vllm-project/vllm/contents/tests/ --jq '[.[] | {name}]'

# Read a representative existing test file
gh api repos/vllm-project/vllm/contents/tests/.py --jq '.content' | base64 -d

# Read the source under test if needed
gh api repos/vllm-project/vllm/contents/vllm/.py --jq '.content' | base64 -d

Step 2 — Classify the Test

| What is being tested | Type | Directory | |---|---|---| | Single function / class / method | Unit | tests/ or matching subdir | | Config parsing, data structures, utils | Unit | tests/ root | | CUDA kernels | Unit | tests/kernels/ | | Attention backends | Unit/Integration | tests/v1/attention/ or tests/kernels/ | | Full model inference with LLM class | Integration | tests/entrypoints/llm/ | | OpenAI-compatible API | Integration | tests/entrypoints/openai/ | | Model correctness (HF vs vLLM) | Integration | tests/basic_correctness/ or tests/models/language/ | | Quantization end-to-end | Integration | tests/quantization/ | | LoRA end-to-end | Integration | tests/lora/ | | Distributed / multi-GPU | Integration | tests/distributed/ | | v1 engine internals | Unit/Integration | tests/v1/ matching subdir | | v1 e2e scenarios | Integration | tests/v1/e2e/general/ |

If the user specifies a directory, use it.

Step 3 — Write the Test

License header (required on every file)

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

Unit test pattern

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

from unittest.mock import MagicMock, patch
import pytest

from vllm. import 

def test_():
    # Arrange
    obj = (...)
    # Act
    result = obj.(...)
    # Assert
    assert result == expected

@patch("vllm..")
def test_(mock_dep):
    mock_dep.return_value = ...
    result = (...)
    mock_dep.assert_called_once_with(...)
    assert result == expected

class Test:

    def test_(self):
        ...

Key rules for unit tests:

• Never load a real model
• Mock external deps (torch.cuda.*, network calls, file I/O)
• Use pytest.raises(ExceptionType) to test error paths
• Prefer plain pytest functions; use classes only when grouping related tests

Integration test pattern (with LLM class)

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

import weakref
import pytest
from vllm import LLM, SamplingParams
from vllm.distributed import cleanup_dist_env_and_memory

MODEL = "distilbert/distilgpt2"  # use a tiny model

PROMPTS = ["Hello, my name is", "The capital of France is"]

@pytest.fixture(scope="module")
def llm():
    llm = LLM(model=MODEL, max_num_batched_tokens=4096,
               gpu_memory_utilization=0.10, enforce_eager=True)
    yield weakref.proxy(llm)
    del llm
    cleanup_dist_env_and_memory()

@pytest.mark.skip_global_cleanup
def test_(llm: LLM):
    outputs = llm.generate(PROMPTS, sampling_params=SamplingParams(max_tokens=10))
    assert len(outputs) == len(PROMPTS)

Integration test pattern (HF vs vLLM correctness)

# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project

import pytest
from ..conftest import HfRunner, VllmRunner
from ..models.utils import check_outputs_equal

MODELS = ["hmellor/tiny-random-LlamaForCausalLM"]

@pytest.mark.parametrize("model", MODELS)
def test_(hf_runner, vllm_runner, model, example_prompts):
    with hf_runner(model) as hf_model:
        hf_outputs = hf_model.generate_greedy(example_prompts, max_new_tokens=5)
    with vllm_runner(model) as vllm_model:
        vllm_outputs = vllm_model.generate_greedy(example_prompts, max_tokens=5)
    check_outputs_equal(outputs_0_lst=hf_outputs, outputs_1_lst=vllm_outputs,
                        name_0="hf", name_1="vllm")

Step 4 — Choose a Small Model

Always use the smallest suitable model. Prefer these:

| Use case | Model | Size | |---|---|---| | General text / any architecture | distilbert/distilgpt2 | ~117M | | Tiny random weights (no GPU quality) | hmellor/tiny-random-LlamaForCausalLM | .py`

• If user didn't specify a directory, use the mapping in Step 2
• Ensure __init__.py exists in the target directory (check with gh api)

Step 6 — Update CI (Integration tests only)

vLLM uses Buildkite (not GitHub Actions) for its main CI. Configuration lives in .buildkite/test_areas/*.yaml.

Find the matching yaml for the test area:

gh api repos/vllm-project/vllm/contents/.buildkite/test_areas --jq '[.[] | {name}]'
gh api repos/vllm-project/vllm/contents/.buildkite/test_areas/.yaml --jq '.content' | base64 -d

Add the new test file to source_file_dependencies and commands of the relevant step:

steps:
- label: Basic Correctness
  source_file_dependencies:
  - vllm/
  - tests/basic_correctness/test_basic_correctness
  - tests/basic_correctness/test_my_new_test.py   # ← add here
  commands:
  - pytest -v -s basic_correctness/test_basic_correctness.py
  - pytest -v -s basic_correctness/test_my_new_test.py          # ← add here

If no existing yaml fits, create a new one following the same structure (group, depends_on, steps).

Unit tests (no model loading) do NOT need CI changes — they run as part of existing catch-all steps.

Reference

For detailed annotated examples from the real test suite, see:

• [references/test-patterns.md](references/test-patterns.md) — read when you need a specific pattern (conftest fixtures, parametrize, mock usage, distributed markers)

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: shen-shanshan
• Source: shen-shanshan/vllm-dev-skills
• License: Apache-2.0
• Homepage: https://zhuanlan.zhihu.com/p/2031696581678866733

Install and usage instructions live in the source repository linked above.