How to evaluate an AI vendor: a checklist
A practical checklist for working out whether the AI agency, consultant, or vendor across the table from you is worth engaging. Twelve questions and what their answers should sound like.
If you’re a business owner who’s been pitched by an AI agency in the last twelve months, you’ll recognise the shape of the meeting.
Confident slide deck. Generic case studies. A demo that looks impressive but doesn’t quite map to your business. A proposal a few days later that’s higher than expected and vaguer than you’d hoped.
Most of these vendors are not bad people. Some are excellent. The problem is that AI as a category is moving fast enough that the vendors haven’t all caught up with each other, and the buyers haven’t caught up with how to tell them apart.
Here’s a checklist of twelve questions to ask before you engage anyone for AI work. The right answers won’t always be the same, but you’ll know a problematic answer when you hear one.
1. Have you worked with a business like ours before?
What you want: specifics. A named industry, a similar revenue range, a similar staff count, ideally a similar tooling situation.
What’s a problem: “Yes, we’ve worked with lots of businesses like yours” with no specifics.
If they can’t or won’t share who they’ve worked with at the size you operate at, you’re either a guinea pig or talking to someone who’s overstating their experience. Both are fine sometimes. Just don’t pay senior rates for either.
2. Can I talk to a reference at the same scale as me?
What you want: a name, a number, and a willingness to make the introduction inside a week.
What’s a problem: vague offers to share references later, references at much larger or much smaller businesses than yours, or references who turn out to be trade press contacts rather than actual clients.
A vendor who can put you on a call with someone running a similar business who’s used them is doing the most efficient version of selling there is. Most vendors won’t because either the references won’t take the call or the work didn’t go well.
3. What’s the deliverable?
What you want: a written description of what physically arrives, when, and what state it’ll be in.
What’s a problem: “We’ll work iteratively with your team,” “We’ll run a discovery sprint,” “We’ll deliver a strategy.”
If the vendor can’t tell you what arrives in your inbox or on your shared drive at each milestone, you’re going to spend the engagement chasing them for clarity.
4. What happens if we want to stop?
What you want: a clear answer about contract length, exit, and what’s portable.
What’s a problem: 12 month contracts with no exit clause, custom code or models that you can’t take elsewhere, integrations built into proprietary platforms that lock you in.
Vendor lock-in is one of the bigger hidden costs in AI engagements. The good ones build with portable tools and document things so you can take them to another provider. The bad ones build moats around themselves.
5. Who actually does the work?
What you want: a clear understanding of whether you’re getting the senior people in the meeting, junior people, or offshore contractors.
What’s a problem: a senior partner at the pitch, junior delivery you never met, and a budget that assumes you got the senior person.
This isn’t necessarily disqualifying. It’s just expensive if you don’t know it’s happening.
6. How do you handle our data?
What you want: clarity on what they store, where, for how long, and whether your data is used to train any models.
What’s a problem: hand-waving, “everything’s secure”, or no clear answer on whether your customer information is used to improve their tools.
If they don’t give you specific answers here, they’re either being lazy or they don’t know. Neither is good.
7. What tools will you actually use?
What you want: specific tool names. Make.com, Claude, ChatGPT, Microsoft 365, Brevo, Stripe, named CRM platforms.
What’s a problem: “Our proprietary AI platform”, “We use a range of best-in-class tools”, or “We can talk about that after the contract is signed.”
A vendor who’ll tell you upfront what they’re building with respects you and isn’t trying to obscure the work. A vendor who hides the toolset is usually charging margin on commodities.
8. What’s the cost of the tools, separate from your fees?
What you want: a line-item view of monthly subscription costs, set up fees, and expected scaling costs.
What’s a problem: bundled pricing where you can’t see the underlying tool costs, or surprise tool fees that turn up after the engagement starts.
Some vendors fold tool fees into their pricing as margin. That’s fine if you know it’s happening. It’s a problem if it’s hidden.
9. Show me a deliverable from a previous client
What you want: an actual document, sanitised if needed. A real workflow diagram. A real audit. A real implementation plan.
What’s a problem: case study slides, testimonials in marketing collateral, or “we can’t share client work.”
If they’ve never produced a deliverable they can put in front of you, they may have done less actual work than they’re claiming.
10. What’s the success metric?
What you want: a measurable outcome tied to revenue, time saved, or another concrete number, agreed before the work starts.
What’s a problem: “Improved efficiency”, “better customer experience”, or any answer that doesn’t translate into a number you’d recognise three months later.
If the success metric is fuzzy, the engagement will be too. Six months later you’ll be wondering whether the money worked.
11. What are you not going to recommend?
What you want: a real answer about what they would skip, what they think is overrated, or what they’d actively warn you against.
What’s a problem: an enthusiast who thinks AI fits everywhere, or a vendor who can’t name something they’d avoid.
Anyone who’s done the work has a list of things they’ve seen fail. If they don’t, they probably haven’t done the work.
12. What happens if AI changes underneath us?
What you want: a thoughtful answer about how they keep current, how they handle deprecations, and what their migration story is when a model or tool goes away.
What’s a problem: “AI is changing fast, so we’ll keep adapting” with no specifics on how.
The AI tooling landscape changes monthly. A vendor who’s locked into one platform or one model will be in trouble in 18 months. A vendor with a clear, tool-agnostic approach will keep working.
Final thought
If a vendor passes most of these and pushes back politely on a couple, that’s usually a good sign. If they pass all twelve perfectly, ask harder questions.
If they fail more than three or four, they may be fine for a small project, but I wouldn’t put six figures behind them.
The audit I run includes a vendor evaluation section where relevant, because part of what I do is help business owners avoid spending money in the wrong places.