Testing Strategy

Test Strategy: ROI-Based Selection

Core Principle: Maximum Coverage, Minimum Tests

Philosophy: 10 reliable tests > 100 unmaintained tests

Quality over quantity - focus resources on high-value tests that provide maximum coverage with minimum maintenance burden.

ROI Calculation Framework

ROI Formula

ROI Score = (Business Value × User Frequency + Legal Requirement × 10 + Defect Detection)
            / (Creation Cost + Execution Cost + Maintenance Cost)

Value Components

Business Value (0-10 scale):

• 10: Revenue-critical (payment processing, checkout)
• 8-9: Core business features (user registration, data persistence)
• 5-7: Important secondary features (search, filtering)
• 2-4: Nice-to-have features (UI enhancements)
• 0-1: Cosmetic features

User Frequency (0-10 scale):

• 10: Every user, every session (authentication)
• 8-9: >80% of users regularly
• 5-7: 50-80% of users occasionally
• 2-4: 80% of users)
• Core CRUD operations for primary entities
• Critical business workflows
• Data integrity for primary data models

Budget Exception: Critical User Journeys may exceed standard budget limits with explicit justification.

Test Selection Guidelines

Selection Thresholds

Integration Tests:

• ROI > 3.0: Strong candidate
• ROI 1.5-3.0: Consider based on available budget
• ROI 2.0: Strong candidate
• ROI 1.0-2.0: Consider if Critical User Journey
• ROI 3.0 for integration, >1.5 for E2E

1. Budget Utilization: Actual tests / Budget limit

• Target: 80-100% (full utilization of valuable test slots)

1. Defect Detection Rate: Bugs caught / Total tests

• Track over time to validate ROI predictions

Calibration

Periodically review:

• Are high-ROI tests actually catching bugs?
• Are cost estimates accurate?
• Do business value ratings align with stakeholder priorities?

Adjust formula weights based on empirical data

Anti-Patterns to Avoid

❌ Gaming the System:

• Inflating business value scores to justify favorite tests
• Ignoring ROI when it contradicts intuition
• Cherry-picking ROI calculation only for preferred tests

✅ Proper Usage:

• Apply ROI calculation consistently to all candidates
• Document justification when overriding ROI decisions
• Use empirical data to calibrate scores over time

❌ Analysis Paralysis:

• Spending excessive time on precise ROI calculations
• Debating single-point differences in scores
• Treating ROI as exact science rather than decision aid

✅ Practical Application:

• Use ROI for relative prioritization, not absolute precision
• Focus on order-of-magnitude differences (8.9 vs 0.34)
• Make quick decisions for obvious high/low ROI cases

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: shinpr
• Source: shinpr/agentic-code
• License: MIT

Install and usage instructions live in the source repository linked above.