Vibe Adversarial Test Generation

vibe-adversarial-test-generation

Happy-path tests prove your code works. Adversarial tests prove it doesn't break. Spec-linked tests prove it does what it's supposed to.

When to Use This Skill

• After writing happy-path tests for a feature
• Before claiming test coverage is complete
• When spec requirements lack corresponding tests
• When preparing for a security review
• When vibe-spec-sync or vibe-coverage-enforcer reports uncovered requirements

When NOT to Use This Skill

• Before happy-path tests exist (write those first)
• For throwaway/prototype code
• When the function is trivially simple (e.g., getters/setters)

Modes

Mode 1: Adversarial (Edge Cases)

Generate tests that break assumptions across 6 categories.

1. Boundary Values

• Zero, one, max, max+1 for all numeric inputs
• Empty string, single char, max-length string
• Empty array, single element, very large array
• Exactly at threshold values

2. Nil/Null/Undefined

• nil pointer as receiver
• nil arguments to every parameter
• nil nested fields in structs
• Returning nil where non-nil expected

3. Type Edge Cases

• Unicode: emoji, RTL text, zero-width chars, combining chars
• Strings: newlines, tabs, null bytes, control characters
• Numbers: NaN, Infinity, -0, very large, very small
• Dates: leap year, DST transitions, timezone boundaries, epoch

4. Concurrency

• Two goroutines/threads calling the same function
• Read during write
• Close during use
• Cancel during operation

5. Resource Exhaustion

• Very large inputs (10MB string, 1M element slice)
• Disk full simulation
• Network timeout simulation
• Memory pressure

6. Malformed Input

• Invalid JSON/YAML/XML
• Truncated input (cut off mid-field)
• Wrong types (string where int expected)
• Extra fields, missing required fields
• SQL injection patterns, XSS payloads (for external inputs)

Mode 2: Spec-Driven (Requirement Coverage)

Generate tests that trace back to specific spec requirements.

1. Read the spec — Find the specification document for the feature under test

1. Extract requirements — Parse each requirement into an atomic, testable statement:

• "Users can reset passwords via email" → testable
• "The system should be fast" → not testable (flag it)

1. Map existing tests to requirements — Scan test files for:

• Comment markers: # req:[REQ-ID] or // req:[REQ-ID]
• Function name patterns: test_req_[REQ-ID]_*
• If no markers exist, use semantic matching (test name/body vs requirement text)

1. Identify uncovered requirements — Requirements with no mapped tests

1. Generate tests for uncovered requirements:

• One test function per requirement minimum
• Name format: test_req_[REQ-ID]_[description] (e.g., test_req_AUTH003_password_reset_sends_email)
• First line of test body MUST include traceability marker:

``python # req:AUTH-003 ` `go // req:AUTH-003 ` `typescript // req:AUTH-003 ``

• Test must assert actual behavior against the requirement, not just call the function
• No skip(), no TODO, no empty bodies

1. Report coverage delta:

• Requirements covered before: X/N
• Requirements covered after: Y/N
• Remaining uncovered (with reasons — e.g., "requires external service mock")

Steps (Adversarial Mode)

1. Read the function under test — understand inputs, outputs, side effects
2. For each input parameter, generate adversarial values from each category
3. For each adversarial input, determine expected behavior:

• Should it return an error? (most common)
• Should it handle gracefully? (fallback behavior)
• Should it panic? (almost never the right answer)

1. Write the tests as table-driven test cases
2. Run them and fix any unexpected panics or wrong error handling

Steps (Spec-Driven Mode)

1. Read the spec and extract atomic requirements with IDs
2. Scan existing tests for requirement markers and semantic matches
3. Generate a coverage map: requirement → test(s) or UNCOVERED
4. Write tests for uncovered requirements with traceability markers
5. Run tests and verify they pass against current implementation
6. Report the before/after coverage delta

Output Format

Adversarial Tests: [Function Name]

Tests Generated: X Categories Covered: Y/6

| # | Category | Input | Expected | Actual | |---|----------|-------|----------|--------| | 1 | Nil input | nil | ErrNilInput | PASS | | 2 | Empty string | "" | ErrEmpty | PANIC! |

Issues Found

1. [Function] panics on nil input (should return error)

---

Spec-Driven Tests: [Feature/Spec Name]

Spec: [path/to/spec.md] Requirements Found: N Previously Covered: X/N (Y%) Now Covered: Z/N (W%)

| REQ ID | Requirement | Test Status | Test Function | |--------|-------------|-------------|---------------| | AUTH-001 | Login requires email + password | Covered | testreqAUTH001loginrequirescredentials | | AUTH-002 | Failed login locks after 5 attempts | NEW | testreqAUTH002lockoutafterfive_failures | | AUTH-003 | Password reset sends email | UNCOVERED | (requires email service mock) |

Remaining Gaps

1. AUTH-003: Requires email service mock — suggest adding mock in conftest.py

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ash1794
• Source: ash1794/vibe-engineering
• License: MIT

Install and usage instructions live in the source repository linked above.