Vendor Evaluation

Vendor evaluation

Choosing a vendor is choosing a dependency you will live with for years and pay to leave. The evaluation goes wrong when it is run on demos and vibes: the polished sales walkthrough decides it, criteria get invented to justify the favorite, and no one asks what it costs to get back out. A disciplined evaluation tests the vendor on your work and prices the exit before it prices the entry.

Method

1. Set weighted criteria before you see a single demo. List what matters:

functionality, reliability, security posture, support, total cost, and how hard it is to leave. Assign weights that sum to a fixed total, and write them down first. Criteria invented after the demo just rationalize the vendor who demoed best.

1. Score against requirements, not against each other. Rate every vendor on

each criterion against your bar, so a weak field cannot make a mediocre option look strong by comparison. Separate must-haves, which are pass or fail, from nice-to-haves that earn weighted points.

1. Design the proof of concept around your real workload. Feed it your actual

data shapes, your peak volume, your awkward edge cases, and your integration points. A POC on the vendor's sample data proves their demo works. Define the success metrics and the time box before you start so the POC ends in a verdict, not an open-ended trial.

1. Cost the exit, not just the entry. Estimate what it takes to migrate off:

data export formats, proprietary interfaces you would rewrite, retraining, and contract lock-in. High switching cost is a real price even if it never appears on an invoice. Prefer vendors whose data you can get back in a usable form.

1. Verify security and compliance with evidence, not the questionnaire alone.

Ask for the SOC 2 or ISO report, the data processing terms, subprocessor list, breach history, and where data is stored. A vendor who cannot produce these for a serious deal is telling you something.

1. Reference-check past the list they gave you. Talk to the references, then

find a customer they did not hand you and ask what breaks at month six: support response times, hidden costs, the feature that was "on the roadmap." Sales sells the roadmap; a real customer tells you the product.

Checks

• Were the weighted criteria fixed before demos, or shaped to fit a favorite?
• Did the POC run on your workload and data, or on the vendor's canned scenario?
• Can you state, in dollars and weeks, what leaving this vendor would cost?

Boundaries

This picks the vendor; it does not negotiate the contract terms or run procurement and legal review, which have their own owners and thresholds. Approval limits and required security sign-offs are company policy: route spend and data agreements through the process your organization requires rather than deciding them here.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Amey-Thakur
• Source: Amey-Thakur/AI-SKILLS
• License: MIT
• Homepage: https://amey-thakur.github.io/AI-SKILLS/

Install and usage instructions live in the source repository linked above.