AI Vendor Evaluation Checklist (Downloadable PDF)

The 6-point filter (and why each question matters)

1. "Walk me through your eval methodology"

Real answer: "Your domain experts label 150-300 test cases. We score every prompt change against that set." Hand-wave answer: "We have a comprehensive testing framework." The hand-wave is the single highest correlate with buyer regret in our data — 18 of 23 failed-engagement post-mortems had no real eval methodology.

2. "Show me a SOC 2 Type II report under NDA before we sign"

Real answer: vendor shares the report inside 48 hours. Red flag: vendor says they are "SOC 2 compliant" but cannot produce the report, or has only a Type I (point-in-time) not Type II (12-month operating effectiveness). For US regulated workloads, this is non-negotiable.

3. "Who owns the code, the prompts, and the eval set at the end?"

Real answer: "You do. We provide the source code, the prompts, and the eval set in a Git repository you control on day one." Red flag: "Our proprietary IP" or "you license our framework." If the answer is anything other than full transferable ownership, your exit cost is hidden in their contract.

4. "What is the exit clause if we cancel mid-engagement?"

Real answer: pro-rata refund on unused fees + delivery of all work-product to date. Red flag: 12-month minimum commits, non-refundable fees, or IP that "reverts" to the vendor if cancelled. We have seen $180K engagements where the buyer walked away with nothing because the contract favored the vendor on cancellation.

5. "How many of your last 5 engagements went live on the committed date?"

Real answer: specific number (e.g. "4 of 5; one slipped by 3 weeks because the client's data layer needed cleanup first"). Red flag: "Most" or "the majority" or refusal to give a specific count. Specificity here strongly predicts whether your engagement will ship on time.

6. "Can you give me 3 client references in my industry from the last 12 months?"

Real answer: 3 names + intros within a week. Red flag: only one reference, only old references, or references all from one tier (only enterprise or only small business when your scope is different). Industry-specific recent references catch 70% of the "they cannot actually do my use case" risk.

The 4 failure patterns this checklist catches

1. Vague scope language — 31% of regretted engagements. "AI to streamline operations" instead of "AI agent to deflect customer support tickets in three categories: order status, returns, subscription changes."
2. No eval methodology — 24% of regretted engagements. Vibes-based testing means every prompt change is a guess. Quality drifts. The buyer cannot tell if the vendor is making things better or worse.
3. Weak IP ownership — 18% of regretted engagements. Buyer cannot leave without losing the work. Becomes hostage to vendor pricing on renewal.
4. Missing audit-evidence — 12% of regretted engagements. Year-2 SOC 2 or HIPAA audit fails because nobody captured the logs / decision trails the auditor needs.

The remaining 15% of failures spread across pricing surprises, communication breakdowns, and personnel changes on the vendor side. The 6-point filter does not catch every failure but it catches the 85% that matter most.

FAQ

Methodology

The 67% buyer-regret figure and $92K average wasted-spend reference IDC's 2026 Worldwide AI Spending Guide and the Forrester Total Economic Impact study on AI vendor selection. The 4-pattern failure breakdown comes from 23 documented Braincuber project post-mortems plus 18 buyer-side audit conversations we ran between 2024-2026. Checklist refined through 40+ buyer engagements where we either won or lost the deal — both outcomes shaped the questions.