Choosing an AI partner is mostly about vetting execution, not ideas. The ideas are cheap and the demos are easy. What separates a partner who ships from one who leaves you with a stalled pilot is the unglamorous work: integration, data, seniority, hand-off, and honest pricing.
You will hear that 95 percent of enterprise AI pilots fail. That figure comes from one MIT report, rests on 52 interviews and a narrow definition of return, and we take it apart properly here. The precise rate is arguable. What is not arguable, and what every estimate agrees on, is where the failures happen: at execution, not at idea selection. Which is what these ten questions are for.
So the questions below are aimed squarely at execution. Use them on the first call. A good partner will answer all ten without flinching, and a few of the answers will disqualify vendors faster than any reference check.
Why choosing wrong is so expensive
An AI project that stalls does not just waste its own budget. It burns months, it teaches your team that AI does not work here, and it makes the next attempt harder to fund. The direct cost of a failed build is the smallest part of the loss.
That is why it pays to vet slowly and start small. The right partner will usually suggest a small, cheap first step rather than a big commitment, precisely because they are confident enough to let the first phase earn the second. Be suspicious of anyone pushing a large scope before they have shipped you anything.
The ten questions
1. Will they tell you when AI is the wrong answer?
The most important signal is whether a vendor will talk you out of a bad-fit project. A partner whose only product is AI has every incentive to recommend AI for everything, including workflows better solved by fixing a process or making a hire. Ask directly: what would you refuse to build for us, and why? If they cannot name anything they would turn down, they are selling, not advising. We end about one in six discovery calls by recommending the buyer not use AI for the thing they asked about, and that honesty is usually why they trust us on the projects that are a real fit.
2. Are they software engineers, or prompt people?
The hard part of an AI project is rarely the model. It is the data, the integrations, the infrastructure, the testing, and the maintenance. Ask how long they have built production software, not how long they have done AI. A shop that has shipped real systems for years and layered AI on top will handle the parts that break projects. A shop founded in 2023 that has only ever wired up model calls will discover those parts on your budget.
3. How do they price, and what happens if it runs over?
Pricing tells you who carries the risk. Open-ended time and materials with no ceiling puts all the risk on you. A firm fixed price on genuinely new work usually means they have padded the number 30 to 40 percent to cover their own unknowns. The model that protects you is a hard budget cap: a ceiling set before work starts, billed against actual hours, never exceeded without your written sign-off. Ask what happens if the work runs over. The right answer is “we stop, show you the new scope, and wait for you to approve it,” not “we bill the extra.” Then ask the follow-up almost nobody asks: how often do your projects come in under cap, and can you show me the numbers? A cap that has never been beaten is a fixed price wearing a friendlier name.
It is also worth pricing the alternative properly before you judge any quote. A senior AI engineer’s fully loaded cost, salary plus benefits, equity and overhead, runs into six figures a year before they have shipped anything and before you have paid a recruiter or spent months hiring. Real systems rarely need only one skill set, so an in-house capability is usually several senior salaries running continuously whether or not there is a full year of work to fill. Against that, a scoped build with a ceiling is often the cheaper way to get a working system, and you carry none of the hiring risk. In-house wins once the work is continuous enough to keep a team busy every week. Our cost guide has the full comparison.
4. Who actually does the work?
Agencies often sell with senior people and staff with junior ones. Ask who specifically will be on your project and what their role is. On a healthy engagement, a senior engineer, a senior AI specialist, and a delivery lead are on the actual work, and junior people only handle the parts that genuinely do not need a senior, like a dashboard or a UI edit. Get the names. A vendor that will not tell you who does the work is hiding something about seniority.
5. Do they handle the integration and data, or just the model?
This is where most pilots die. Ask how they plan to connect the AI to your existing systems and what they will do about messy or scattered data. If integration is described as “phase two” or data cleanup is not mentioned at all, they are quoting you a demo. A partner who has shipped before will bring up your legacy systems and your ugly data early, because they know that is the real work.
6. Do they plan the hand-off?
A project is not done when it works in a demo. It is done when your team can run it without the vendor. Ask whether documentation, a runbook, and training are in the scope, or an upsell at the end. If knowledge transfer is not written into the contract, expect the system to rot the moment the vendor leaves or the one person who understood it moves on.
7. Can they show real before-and-after numbers?
Demos prove nothing about production. Ask for specific outcomes from shipped projects: not “improved efficiency” but “file prep went from eight hours to 25 minutes,” or “review team went from 14 people to three.” Real numbers, ideally with the context of what was built and how long it took. A partner with production experience will have these ready. A partner who only has demos will change the subject. Ours are on the case studies page, with the caps, the actuals and the timelines alongside the outcomes.
8. Do you own what they build?
Ask plainly: at the end, do we own the code, the models, the prompts, and the documentation? The answer should be an unqualified yes, in writing. If any of it stays locked to the vendor’s platform or license, you are renting your own system, and your switching cost is exactly what they are counting on.
9. How do they handle your data?
If your work touches personal, financial, or health data, this is non-negotiable. Ask whether your data will be used to train anyone’s model (it should not), whether they use enterprise APIs with data retention disabled, whether they can deploy inside your own cloud, and whether they will sign your data processing agreement. Vague answers here are a hard stop.
10. How fast can they start, and how do they scope?
Speed matters, but how they scope matters more. A good partner can usually start a strategy engagement within a week and a build a couple of weeks after scope sign-off. More telling is whether they phase the work: a cheap scoping step, then a prototype on real data, then the production build, with a stop point at each boundary so you can kill a bad idea early. A vendor who wants a big commitment before proving anything is asking you to carry all the risk.
Consultant, agency, or freelancer?
Before the ten questions, there is a prior one: what kind of partner do you need at all? A freelancer is cheapest for a narrow, contained task and carries a bus factor of one. A strategy consultant tells you what to do and leaves you without a team to do it. A large firm brings brand comfort, premium pricing and often junior delivery. An engineering-led firm that both advises and ships suits most companies with a real workflow to solve, which is what this checklist assumes you are vetting.
The point is not that one category is always right. It is that hiring a big firm for a two-week scoping job, or a solo freelancer for a system you need to run for five years, is a mismatch no amount of vetting fixes. We compare the options properly on AI consulting versus an automation agency and, if you are weighing a large firm specifically, on the big-firm alternative.
A quick scorecard
Score each vendor yes or no. Five or more no answers, or a no on pricing, ownership, or data handling, should end the conversation.
| Question | Green flag | Red flag |
|---|---|---|
| Will they say no to bad-fit AI? | Names things they would refuse | ”We can build anything” |
| Engineers or prompt people? | Years of production software | Only ever wired up model calls |
| Pricing model | Hard cap, sign-off on overruns | Open-ended hourly, no ceiling |
| Who does the work | Named seniors on the build | Vague, “our team” |
| Integration and data | Raised early, scoped | ”Phase two,” not mentioned |
| Hand-off | Docs and training in scope | Upsell or absent |
| Proof | Real before/after numbers | Demos only |
| Ownership | You own everything, in writing | Locked to their platform |
| Data handling | Retention off, your DPA, your cloud | Vague or evasive |
| Scoping | Phased with stop points | Big commitment up front |
What to ask on the first call
If you only have time for a few questions, ask these. What would you refuse to build for us, and why? Who specifically will do the work? What happens to the budget if the project runs over? What will we own at the end? Can you show me real numbers from something you shipped? The answers will tell you more than any proposal document, because they are hard to fake in real time.
Frequently asked questions
How do I know if an AI vendor is any good before I hire them? Look for production references with real numbers, a pricing model that caps your risk, named senior people on the work, and a willingness to tell you when AI is the wrong tool. Slides and demos are easy to produce. Shipped systems with measurable outcomes are not.
Is a big consulting firm a safer choice? Not necessarily. Large firms bring brand comfort but often staff junior teams, price at a premium, and move slowly. For a company of 20 to 500 people, a senior, engineering-led team that ships in weeks and hands the system back working is usually a better fit than a big-firm engagement built for enterprises ten times your size.
Should the same partner do strategy and the build? It helps. When one team both advises and ships, nothing gets lost in the hand-off between a strategist who wrote a deck and a builder who never sat in the room. If you split them, make sure they actually talk to each other, or you will pay to bridge the gap.
What is the single biggest red flag? Open-ended pricing with no cap, closely followed by a slick demo with no production references. Both mean the vendor is transferring risk to you, either financial or delivery risk, and hoping you do not notice until you have signed.
Take the scorecard above into your next vendor call and fill it in live. The questions are designed to be hard to fake in real time, which is why asking them on a call beats reading a proposal.
If you want to run it on us, we will answer all ten, including the ones about when not to hire us. Our pricing and case studies are published so you can check questions 3 and 7 before we even speak.
Last updated